Changelog-EXPERIMENTAL: Build: all experimental features are now runtime-enabled; no more ./configure --enable-experimental-features
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
By using fuzzer instrumentation for dependencies, we get more coverage
signal during fuzzing. This is useful when the fuzzer must figure out
how to take certain branches in a dependency.
In our case, the fuzz-bip32 target was failing to create a data buffer
that successfully passed fromwire_ext_key() parsing because the fuzzer
couldn't see what was happening inside libwally-core.
For example, if we use -fsanitize=undefined, we can't do unaligned
integer access, but since we didn't test with the sanitizer flags, we
didn't know this, and set `HAVE_UNALIGNED_ACCESS=1`.
Also, add -fno-sanitize-recover= in developer mode, so we actually
fail binaries if something is detected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The M1 Macs support both x86_64 and arm64 architectures, which forced
homebrew to use a different path for its storage (`/opt/homebrew/`
instead of `/usr/local`). If we don't adjust the path we'd mix x86_64
and arm64 libraries which can lead to weird compiler and linker
errors.
This patch just introduces `CPATH` and `LIBRARY_PATH` as suggested by
the homebrew team, and detects the current architecture automatically.
Changelog-Added: macos: Added m1 architecture support for macos
Mostly comments and docs: some places are actually paths, which
I have avoided changing. We may migrate them slowly, particularly
when they're user-visible.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We detect whether we have the rust tooling available (mainly `cargo`)
and enable or disable the rust libraries, plugins and examples when it
is enabled. Since the rest of the Makefiles assumes that executables
have an associated header and C source file, we also needed to add a
target that we can add non-C binaries to.
When looking for a pytest executable we should be looking for the ones
that `pip` installs in a virtualenv (`pytest` and `py.test`) before we
look for the ones that `apt` installs system-wide (`pytest3` and
`py.test3`) because these may not be part of the virtualenv that all
other packages are installed in.
This adds a new configuration, --enable-fuzzing (which is more than
welcome to be coupled with --enable-address-sanitizer), to pass the
fuzzer sanitizer argument when compiling objects. This allows libfuzzer
to actually be able "to fuzz" by detecting coverage and be smart when
mutating inputs.
As libfuzzer brings its own ~~fees~~ main(), we compile objects with
fsanitize=fuzzer-no-link, and special-case the linkage of the fuzz
targets.
A "lib" is added to abstract out the interface to the fuzzing tool used.
This allow us to use the same targets to fuzz using AFL, hongfuzz or w/e
by adding their entrypoints into libfuzz. (h/t to practicalswift who
introduced this for bitcoin-core, which i mimiced)
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
At least on my Ubuntu box, they're compatible. If they're not, we need
to disable regeneration altogether.
Fixes: #4075
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We alter the script to do the move to config.vars right at the end,
which is probably a good idea anyway.
Fixes: #4054
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also, allow the pg_config binary to be specified through the PG_CONFIG
environment variable, defaulting to 'pg_config' if unset. Explicitly
setting PG_CONFIG to an empty string will forcibly disable PostgreSQL
support, even if a PostgreSQL library is installed.
Changelog-Fixed: build: On systems with multiple installed versions of the PostgreSQL client library, C-Lightning might link against the wrong version or fail to find the library entirely. `./configure` now uses `pg_config` to locate the library.
Changelog-Fixed: build: On some operating systems the postgresql library would not get picked up. `./configure` now uses `pg_config` to locate the headers.
Since the probing binaries compiled by the configurator needs to run on
the host machine we provide a variable CONFIGURATOR_WRAPPER that can be
set to anything that you want to wrap the calls with.
One example is `qemu-aarch64-static`.
The description of
--enable/disable-experimental-features was a bogus "Developer mode, good
for testing" and
--enable/disable-valgrind was "Valgrind binary to use for tests" which
gave the false impression that it should be set to something like
/usr/local/bin/valgrind whereas it is a boolean option.
If we have the client library for postgres configure will define HAVE_POSTGRES
the same way it already handled libsqlite3 an we start linking against it.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is the counterpart of the annotations we did in the last few commits. It
extracts queries, passes them through a driver-specific query rewriter and
dumps them into a driver-specific query-list, along with some metadata to
facilitate processing later on. The generated query list is then registered as
a `db_config` and will be loaded by the driver upon instantiation.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We were just telling GCC not to treat them as errors: this suppresses them
entirely unless at -O3. People keep trying to "fix" them, when in fact
they're false positives, as revealed with "./configure COPTFLAGS=-O3".
Fixes: #2856
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that DEVELOPER was being reset to 0 by ./configure --reconfigure;
that's because we set the defaults first, then --reconfigure would only
override *unset* vars. We now set up defaults last.
We unify all our "find the default" functions, to neaten them; in
particular find_pytest and our open-coded valgrind testing function.
We can figure out whether valgrind works using /bin/true instead of waiting
until configurator is built.
We're also more careful with ${FOO-default} vs ${FOO:-default}; the former
does not replace if FOO is set to the empty string, which is possible for
some of our settings (COPTFLAGS, CDEBUGFLAGS in particular).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Drive-by fix of CONFIGURATOR_CC setting when CC is overridden on cmdline
(not via env var).
Note that clang defines __GNUC__ (to 4) :(
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`./configure CC=newcc --reconfigure` didn't set CC, because reconfigure
simply replaced all values: only make it replace undefined values.
Also, it didn't change COPTFLAGS or CWARNFLAGS, even if they were previously
the defaults (eg. --reconfigure --enable-developer).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unfortuntely we get spurious uninitialized variable warnings with
anything but -O3 or no optimization, so set default CWARNFLAGS
appropriately.
MCP bench results without optimization:
store_load_msec:28509-31001(29206.6+/-9.4e+02)
vsz_kb:580004-580016(580006+/-4.8)
store_rewrite_sec:11.640000-12.730000(11.908+/-0.41)
listnodes_sec:1.790000-1.880000(1.83+/-0.032)
listchannels_sec:21.180000-21.950000(21.476+/-0.27)
routing_sec:2.210000-11.160000(7.126+/-3.1)
peer_write_all_sec:36.270000-41.200000(38.168+/-1.9)
MCP bench with -Og: 22% speedup vs no optimization
store_load_msec:21963-23645(22841+/-6.6e+02)
vsz_kb:579916
store_rewrite_sec:10.080000-10.960000(10.456+/-0.3)
listnodes_sec:1.280000-1.390000(1.338+/-0.047)
listchannels_sec:14.770000-16.080000(15.518+/-0.46)
routing_sec:0.990000-6.660000(3.958+/-2.2)
peer_write_all_sec:29.950000-32.950000(31.138+/-1)
MCP bench with -O2: 31% speedup vs no optimization
store_load_msec:20713-22088(21505.6+/-4.8e+02)
vsz_kb:579928
store_rewrite_sec:9.570000-11.200000(10.192+/-0.54)
listnodes_sec:0.960000-1.090000(1.028+/-0.045)
listchannels_sec:10.400000-11.770000(11.012+/-0.48)
routing_sec:0.300000-3.140000(1.978+/-1.1)
peer_write_all_sec:28.980000-30.310000(29.572+/-0.44)
MCP bench with -O3 -flto: 36% speedup vs no optimization
store_load_msec:19616-20191(19862.6+/-1.9e+02)
vsz_kb:578452
store_rewrite_sec:8.980000-9.960000(9.55+/-0.32)
listnodes_sec:0.920000-1.910000(1.18+/-0.38)
listchannels_sec:8.960000-9.450000(9.206+/-0.16)
routing_sec:0.730000-1.850000(1.438+/-0.42)
peer_write_all_sec:28.090000-29.410000(28.772+/-0.42)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* remove libbase58, use base58 from libwally
This removes libbase58 and uses libwally instead.
It allocates and then frees some memory, we may want to
add a function in wally that doesn't or override
wally_operations to use tal.
Signed-off-by: Lawrence Nahum lawrence@greenaddress.it
