On some distributions (e.g. Gnu Guix) Python packages are not installed in
some standard directory, rather they are installed in different places and
the `PYTHONPATH` variable is modified to include the different places.
So, we must not use the name `PYTHONPATH` in our `Makefile` since `make`
will replace the `PYTHONPATH` environment variable, preventing e.g.
`tools/generate-wire.py` from finding `python-mako` installed on such
distributions.
WebSocket is a bit weird:
1. It starts like an HTTP connection, but they send special headers.
2. We reply with special headers, one of which involves SHA1 of one of theirs.
3. We are then in WebSocket mode, where each frame starts with a 2-20 byte
header.
We relay data in a simplistic way: if either side sends something, we
read it and relay it synchronously. That avoids any gratuitous
buffering.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, they are allowed to include .c files!
Here's `make check-units` on a maintainer-clean tree:
```
onchaind/test/run-onchainstress.c:4:10: fatal error: ../../hsmd/hsmd_wiregen.c: No such file or directory
4 | #include "../../hsmd/hsmd_wiregen.c"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:276: onchaind/test/run-onchainstress.o] Error 1
make: *** Waiting for unfinished jobs....
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It also gets rid of the requirement that close negotiation fee maximum
is the old commitment transaction. We still do that, however, to
avoid surprising old peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This touches a lot of text, mainly to change "if `option_anchor_outputs`"
to "if `option_anchors`"
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This includes the new bolt11 test vectors, and also removes the
requirement that HTLCs be less than 2^32 msat. We keep that for now
because Electrum enforced it on receive: in two releases we will stop
that too.
So no longer warn about needing mpp in that case either.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Protocol: No longer restrict HTLCs to
Enable non-dev builds to send custom messages.
Preserves 'dev-' for compat-enabled builds.
Changelog-Changed: JSON-RPC: moved dev-sendcustommsg to sendcustommsg
For markdown, there's no simple comment prefix: we need a postfix too.
We also need to use "" since we want to use ' in some of the Makefiles
in future when V=1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This includes anysegwit and the updated HTLC tiebreak test vector. It
also adds explicit wording for invalid per_commitment_secret (which
nicely matches our code already!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you do update-mocks in a dirty tree, the recursive make that it
uses will try to rebuild things! Suppress that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They are currently not installable due to circular dependencies and
stuff, so we just add their source to the path and lnprototest will
pick them up from there.
Instead of "only check suffix quotes when EXPERIMENTAL_FEATURES", make
it so we only check suffix quotes if you override BOLTVERSION on the
cmdline.
Before this, "make check-source-bolt" was effectively a NOOP with
--enable-experimental-features.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The main change which affects us is that 2016 blocks to forget a channel
is a fixed number in the spec; we make this clear by renaming the
(developer-only) max_funding_unconfirmed to dev_max_funding_unconfirmed
and making it compile DEVELOPER only.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You can now activate dual-funded channels using the
`--experimental-dual-fund` flag
Changelog-Changed: Config: `--experimental-dual-fund` runtime flag will enable dual-funded protocol on this node
We assume if they set this to 0 (which nobody did previously), they're
using it as a modern flag and use it to indicate when they're
finished. Otherwise, we count how many blocks they've sent and use
that to determine whether they've finished.
See: https://github.com/lightningnetwork/lightning-rfc/pull/826
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we use `sync_complete` for gossip range query replies, with detection for older spec nodes.
A new target `check-gen-updated` to verify that all derived/generated
files that were modified were also checked in. This is used on CI to
check, and is not added to `check` since it'll complain on dirty
trees, i.e., before the devs check in their changes.
Suggested-by: Rusty Russell <@rustyrussell>
There's a 60 second delay in one of the contrib tests, and I just want
to run flake8 on my alterations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
== is a bash extension; in shell it's a single =:
```
/bin/sh: 1: [: unexpected operator
sql-rewrite wallet/db_postgres_sqlgen.c
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
$(BIN_PROGRAMS) is defined afterwards, leading to the problem that
lightningd doens't get rebuilt (in fact, it was running my installed
lightningd instead!).
It also needs $(PLUGINS) and the subdaemons, so hoist them all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a new configuration, --enable-fuzzing (which is more than
welcome to be coupled with --enable-address-sanitizer), to pass the
fuzzer sanitizer argument when compiling objects. This allows libfuzzer
to actually be able "to fuzz" by detecting coverage and be smart when
mutating inputs.
As libfuzzer brings its own ~~fees~~ main(), we compile objects with
fsanitize=fuzzer-no-link, and special-case the linkage of the fuzz
targets.
A "lib" is added to abstract out the interface to the fuzzing tool used.
This allow us to use the same targets to fuzz using AFL, hongfuzz or w/e
by adding their entrypoints into libfuzz. (h/t to practicalswift who
introduced this for bitcoin-core, which i mimiced)
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This adds an environment variable $NO_PYTHON check to Makefiles so that:
- It checks and runs into an defined error instead of some python hickup:
`ModuleNotFoundError: No module named 'mako'`
- makes it possible to manually export this environment variable NO_PYTHON=1
to run the same testcase that travis is running on job 1 which causes
so much pain ;)
Changelog-None
Kinda uses an antipattern of `cd`ing into the directory and calling `make`
there, but this keeps the contrib dirs self-contained so we can split them out
eventually.
I got a corrupt file, which looked like multiple concurrent attempts
to build it. So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).
Also move into plugins/ where it logically belongs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
At least on my Ubuntu box, they're compatible. If they're not, we need
to disable regeneration altogether.
Fixes: #4075
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make's 'wildcard' function returns matching file paths in an
unpredictable order, thereby causing false mismatches in the produced
SHA256STAMPs from system to system. This commit sorts the file paths
given to 'cat' to make the stamps deterministic.
Changelog-Fixed: The build system no longer spuriously regenerates generated sources due to differences in `readdir`(3) sort order.
This avoids overwriting the ones in git, and generally makes things neater.
We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to make experimental versions of these completely separate files.
Also remove the dependency on the Makefile itself: it simply causes
unnecessary churn. We can always force-rebuild when we change a rule.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now that SHA256STAMP protects us, we can avoid timestamps altogether
so we don't get missing builds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS. Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This should be more robust in future: we SHA256 all of the deps.
For wiregen we prefix with EXPERIMENTAL_FEATURES, since it can effect them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also, allow the pg_config binary to be specified through the PG_CONFIG
environment variable, defaulting to 'pg_config' if unset. Explicitly
setting PG_CONFIG to an empty string will forcibly disable PostgreSQL
support, even if a PostgreSQL library is installed.
Changelog-Fixed: build: On systems with multiple installed versions of the PostgreSQL client library, C-Lightning might link against the wrong version or fail to find the library entirely. `./configure` now uses `pg_config` to locate the library.
Changelog-Fixed: build: On some operating systems the postgresql library would not get picked up. `./configure` now uses `pg_config` to locate the headers.
This means some files get renamed, and I took the opportunity to clarify
our naming (the *d* is important!)
1. channeld/channel_wire.csv -> channeld/channeld_wire.csv
2. channeld/gen_channel_wire.h -> channeld/channeld_wiregen.h
3. enum channel_wire_type -> enum channeld_wire
4. WIRE_CHANNEL_FUNDING_DEPTH -> WIRE_CHANNELD_FUNDING_DEPTH.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's hard to see the error:
Before:
$ make
File config.vars not found: you must run ./configure before running make.
CC: cc -DBINTOPKGLIBEXECDIR="../libexec/c-lightning" -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/x86_64-linux-gnu/libbacktrace-build -I external/libsodium/src/libsodium/include -I external/libsodium/src/libsodium/include/sodium -I external/x86_64-linux-gnu/libsodium-build/src/libsodium/include -I . -I/usr/local/include -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS -DBUILD_ELEMENTS=1 -c -o
LD: cc -fsanitize=address -Lexternal/x86_64-linux-gnu -lwallycore -lsecp256k1 -ljsmn -lbacktrace -lsodium -L/usr/local/lib -lm -lgmp -lsqlite3 -lz -o
After:
$ make
File config.vars not found: you must run ./configure before running make.
make: *** No rule to make target 'config.vars', needed by 'show-flags'. Stop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
See https://github.com/lightningnetwork/lightning-rfc/pull/767
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: channels now pruned after two weeks unless both peers refresh it (see lightning-rfc#767)
As per https://github.com/lightningnetwork/lightning-rfc/pull/785
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: config: the default CLTV expiry is now 34 blocks, and final expiry 18 blocks as per new BOLT recommendations.
This prevents recompiling everything when you are changing just a doc, or
touching only one file among hundreds of sources, just because the
`gen_version.h` is changed, especially since only one source actually
depends on that header.
Since we end up consolidating some of the return values for `pay` and
`paystatus` and change the public interface we need to add the compatibility
flag and guard the switchover behind it.
Changelog-Added: We now install `lightning-hsmtool` for your `hsm_secret` needs.
See: https://github.com/ElementsProject/lightning/issues/3717#issuecomment-644844594
It seems reasonable to add this to the standard install, and to document it properly as well, hopefully we can fill in the documentation better later on.
They're almost entirely autogenerated, and we use symlinks into the
top directory to reduce replication.
They can't be under pyln.spec.message, because a package can't also
be a namespace.
We also add fulltext and desc fields, and exclude our "gen" files from
flake8, since the spec quotes contain weird whitespace.
Changelog-Added: Python: pyln.spec.bolt{1,2,4,7} packages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The main change here is that the previously-optional open/accept
fields and reestablish fields are now compulsory (everyone was
including them anyway). In fact, the open/accept is a TLV
because it was actually the same format.
For more details, see lightning-rfc/f068dd0d8dfa5ae75feedd99f269e23be4777381
Changelog-Removed: protocol: support for optioned form of reestablish messages now compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
- we've moved tmpctx management to setup.c from daemon.c, so we update
the `check-tmpctx`
- `common_setup(char *)` is now a valid analog for `setup_locale`, so we
check for either in check-setup_locale
Hide CFLAGS and LDFLAGS line noise each time an object file is compiled
or linked.
Also add a `make show-flags` command for displaying CC, LD, CFLAGS and
LDFLAG information. This is shown at the start of each build.
Use `V=1 make` to restore original output
make
CC: gcc -DBINTOPKGLIBEXECDIR="../libexec/c-lightning" -Wall [..]
LD: gcc -Og -Lexternal -lwallycore -lsecp256k1 -ljsmn [..]
...
cc wallet/test/run-db.c
cc lightningd/test/run-jsonrpc.c
cc lightningd/test/run-invoice-select-inchan.c
cc lightningd/test/run-log-pruning.c
cc lightningd/test/run-find_my_abspath.c
cc cli/test/run-large-input.c
cc cli/test/run-remove-hint.c
ld lightningd/lightning_hsmd
ld lightningd/lightning_gossipd
ld lightningd/lightning_openingd
ld lightningd/lightning_channeld
ld lightningd/lightning_closingd
...
Signed-off-by: William Casarin <jb55@jb55.com>
Changelog-Changed: build: default compile output is prettier and much less verbose
We had them split according the separate use-cases:
- testing
- doc-gen
- wire-gen
But that was causing new contributors to miss some dependencies when they
first got hacking. So this consolidates all of our own dependencies in a root
requirements.txt, with the notable exception of `pyln-client`, `pyln-testing`
and `pyln-proto` which are distributed as PyPI modules and therefore have
their own dependencies that need to be tracked in the module root.
Closes#3518
Don't let make pollute subprojects' environment with our own `CFLAGS`,
which are quite strict because that breaks at least libwally-core:
```sh
$ ./configure ...
$ CFLAGS=whatever_this_is_irrelevant make
...
cd external/libwally-core-build && ../libwally-core/configure ...
...
CFLAGS = -DBINTOPKGLIBEXECDIR="\"../libexec/c-lightning\"" -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror -std=gnu11 -g -fstack-protector -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/libbacktrace-build -I . -I/usr/local/include -DCCAN_TAKE_DEBUG=1 -DCCAN_TAL_DEBUG=1 -DCCAN_JSON_OUT_DEBUG=1 -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS -DBUILD_ELEMENTS=1 -W -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings -fvisibility=hidden -O3
...
In file included from ../../libwally-core/src/base58.c:4:
../../libwally-core/src/ccan/ccan/endian/endian.h:71:24: error: unused function 'bswap_16'
[-Werror,-Wunused-function]
static inline uint16_t bswap_16(uint16_t val)
^
```
If `CFLAGS` is set in its environment, then `make` would export our own
`CFLAGS` to any subprocesses it starts, which means subprojects would
inherit our `CFLAGS="-Wall -Werror"` in their environments.
GNU Make's documentation:
https://www.gnu.org/software/make/manual/html_node/Variables_002fRecursion.html#Variables_002fRecursion
> make exports a variable only if it is either defined in the environment initially...
Example:
```make
A = x
default:
echo $$A
```
then:
```sh
$ make # prints nothing, A is not exported to the subprocess
$ A=y make # prints "x", our A=x is exported to the subprocess
```
Changelog-None
Added in d901304120, this column is null in old dbs like mine:
2020-02-15T00:08:41.444Z **BROKEN** database: Accessing a null column 12 in query SELECT id, channel_htlc_id, msatoshi, cltv_expiry, hstate, payment_hash, payment_key, routing_onion, failuremsg, malformed_onion, origin_htlc, shared_secret, received_time FROM channel_htlcs WHERE direction= ? AND channel_id= ? AND hstate != ?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Most is taken from lightningd/bitcoind and adapted. This currently
exposes 5 commands:
- `getchaininfo`, currently called at startup to check the network and
whether we are on IBD.
- `getrawblockbyheight`, which basically does the `getblockhash` +
`getblock` trick.
- `getfeerate`
- `sendrawtransaction`
- `getutxout`, used to gather infos about an output and currently used by
`getfilteredblock` in `lightningd/bitcoind`.
On CI it is nice to cache the external dependencies. However if we
always compile them in the same folder we cannot cache for multiple
different architectures. After this commit native compile targets will
still live in `external` but cross compiled versions will live in
`external/<arch>`.
Put `LC_ALL=C sort` in a variable and use it everywhere. It was
forgotten in one place.
Without `LC_ALL=C` it would order `a.b` before `a-c` even though
`. (0x2E)` > `- (0x2D)`.
Changelog-None
Before this change `make` would build everything but the docs.
They would be built during `make install` which is unexpected.
Add a dependency to `doc-all` to the `default` target so that docs are
build together with the rest of the programs. Leave the `install-data`
as is, to still build the docs if they are not built yet during install.
Changelog-None
This should not affect any consumer of the API since we just shift the actual
implementation from one side to the other, and keep aliases in place so
scripts don't break.
We also bump the version number from 0.0.7.3 to 0.7.4 which allows us to be in
sync with c-lightning itself, and remove the superfluous `0` in front.
If we have the client library for postgres configure will define HAVE_POSTGRES
the same way it already handled libsqlite3 an we start linking against it.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
/bin/sh: 1: ccan/ccan/cdump/tools/cdump-enumstr: Text file busy
make[1]: *** [common/Makefile:81: common/gen_htlc_state_names.h] Error 2
make[1]: *** Waiting for unfinished jobs....
The fix is to make sure all generated headers are made first, and
thus cdump-enumstr.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
installcheck calls $(MAKE) under the covers, which can race with the
current builds, and we can try installing something which is still
being built:
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.
/usr/bin/ld: can not read symbols: File truncated
/usr/bin/ld: .eh_frame/.stab edit: File truncated
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.interp' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.note.ABI-tag' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.note.gnu.build-id' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.hash' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynsym' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynstr' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.version' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.version_r' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.rela.dyn' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.rela.plt' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.init' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.plt' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.plt.got' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.text' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.fini' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.rodata' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.eh_frame' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.init_array' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.fini_array' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.data.rel.ro' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynamic' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.got' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `.data' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_json_command' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_hooks' not in segment
/usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_type_to_string' not in segment
Makefile:390: recipe for target 'lightningd/lightningd' failed
make[1]: *** [lightningd/lightningd] Error 1
make[1]: Leaving directory '/home/rusty/lightning'
Makefile:553: recipe for target 'installcheck' failed
make: *** [installcheck] Error 2
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our reproducibile builds use the dirname to get version, but they have
a v in them (the tools/repro-build.sh script gets this right, so I
copied that).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies the dependencies:
1. Objs depend on headers, not other objs.
2. Programs depend on objs.
3. A .o file will generally implicitly depend on the .c file it's built from.
4. If a file has a build line, it's often better to list all deps there.
5. I spotted some missing 'make clean' files.
The particular problem in this case seems to be that make would use
tools/test/gen_test.c before it was ready. It's probably confused by
the use of recursive make via update-mocks, so explicitly split that
into two stages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Dumb programs which have a --daemon option call fork() early. This is
terrible UX since startup errors get lost: the program exits with
"success" immediately then you discover via the logs that it didn't
start at all.
However, forking late introduced a heap of problems with changing
pids. Instead, fork early but keep stderr and the parent around: if
we fail early on, the parent fails with us. We release our parent
with an explicit action just before the main loop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
My test machine started failing on dynamic plugin tests, unable to
find the lightning module:
ModuleNotFoundError: No module named 'lightning'
This is because plugins at startup are run from whatever directory
you're in, but on refresh are run from the lightning-dir.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We always rebuild headerversions to examine critical system headers,
however that stomps on parallel builds with:
make[1]: execvp: tools/headerversions: Text file busy
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes it build for me on FreeBSD 11:
1. $(MAKE) has to passed through into update-mocks.
2. FreeBSD sed doesn't turn \n into a newline on RHS.
3. Bash and mako dependencies were missing from INSTALL.md
Fixes: #2850
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Add a test for checking that the bolt-gens do the right thing
for a fairly exhaustive test case set (and that it compiles).
Note that this doesn't check that we've got the memory assignment
pieces worked out.
It's got a kind of exotic reliance on the update-mocks in that in
order to depend on as little of the wire/ code as possible (we
only import wire/wire.h), we include an AUTOGENERATE comment
in the test_cases CSV file, and then run update-mocks as part of
the build for that file.
updates the bolt version to 6639cef095a2ecc7b8f0c48c6e7f2f906fbfbc58.
this requires us to use the new bolt parser at generate-bolt.py
and updates to all of the type specifications (ie. from u8 -> byte)
These are generalized from our internal implementations.
The main difference is that 'struct json_escaped' is now 'struct
json_escape', so we replace that immediately.
The difference between lightningd's json-writing ringbuffer and the
more generic ccan/json_out is that the latter has a better API and
handles escaping transparently if something slips through (though
it does offer direct accessors so you can mess things up yourself!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I decided to try a faster implementation, only to find our crc32c was
not correct! Ouch.
I removed the crc32c functions from ccan/crc, and added a new crc32c
module which has the Mark Adler x86-64-optimized variants.
We bump gossip_store version again, since csums have changed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unfortuntely we get spurious uninitialized variable warnings with
anything but -O3 or no optimization, so set default CWARNFLAGS
appropriately.
MCP bench results without optimization:
store_load_msec:28509-31001(29206.6+/-9.4e+02)
vsz_kb:580004-580016(580006+/-4.8)
store_rewrite_sec:11.640000-12.730000(11.908+/-0.41)
listnodes_sec:1.790000-1.880000(1.83+/-0.032)
listchannels_sec:21.180000-21.950000(21.476+/-0.27)
routing_sec:2.210000-11.160000(7.126+/-3.1)
peer_write_all_sec:36.270000-41.200000(38.168+/-1.9)
MCP bench with -Og: 22% speedup vs no optimization
store_load_msec:21963-23645(22841+/-6.6e+02)
vsz_kb:579916
store_rewrite_sec:10.080000-10.960000(10.456+/-0.3)
listnodes_sec:1.280000-1.390000(1.338+/-0.047)
listchannels_sec:14.770000-16.080000(15.518+/-0.46)
routing_sec:0.990000-6.660000(3.958+/-2.2)
peer_write_all_sec:29.950000-32.950000(31.138+/-1)
MCP bench with -O2: 31% speedup vs no optimization
store_load_msec:20713-22088(21505.6+/-4.8e+02)
vsz_kb:579928
store_rewrite_sec:9.570000-11.200000(10.192+/-0.54)
listnodes_sec:0.960000-1.090000(1.028+/-0.045)
listchannels_sec:10.400000-11.770000(11.012+/-0.48)
routing_sec:0.300000-3.140000(1.978+/-1.1)
peer_write_all_sec:28.980000-30.310000(29.572+/-0.44)
MCP bench with -O3 -flto: 36% speedup vs no optimization
store_load_msec:19616-20191(19862.6+/-1.9e+02)
vsz_kb:578452
store_rewrite_sec:8.980000-9.960000(9.55+/-0.32)
listnodes_sec:0.920000-1.910000(1.18+/-0.38)
listchannels_sec:8.960000-9.450000(9.206+/-0.16)
routing_sec:0.730000-1.850000(1.438+/-0.42)
peer_write_all_sec:28.090000-29.410000(28.772+/-0.42)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, `make unittest/bitcoin/test/run-secret_eq_consttime`
didn't set VALGRIND if it was set via config.vars, so runs for a *long*
time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These weren't checked by CI yet, and they are really short so I just added
them to the check-python target.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We need to do it in various places, but we shouldn't do it lightly:
the primitives are there to help us get overflow handling correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise a straight "make install" gives:
install: cannot stat 'plugins/pay': No such file or directory
make: *** [Makefile:482: install-program] Error 1
Fixes: #2288
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is mainly just copying over the copy-editing from the
lightning-rfc repository.
[ Split to just perform changes after the UNKNOWN_PAYMENT_HASH change --RR ]
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Rusty Russell <@rustyrussell>
This is based on Christian's change, but removes all trace of the old codes.
I've proposed another spec change which removes this code altogether:
https://github.com/lightningnetwork/lightning-rfc/pull/544
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Rusty Russell <@rustyrussell>
This is mainly just copying over the copy-editing from the
lightning-rfc repository.
[ Split to just perform changes prior to the UNKNOWN_PAYMENT_HASH change --RR ]
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Rusty Russell <@rustyrussell>
git is needed to generate version information
Add a sanity check so that the build don't continue with an empty VERSION. This
is useful for sandboxed build where we might have forgot to include git.
Signed-off-by: William Casarin <jb55@jb55.com>
There were a few reports that upgrading Ubuntu recently caused issues
because we assert that the sqlite3 library version matches the one we
were built with. 'make' doesn't fix this, because it doesn't know the
external libraries have changed.
Fix this harder, with a helper which updates a file every binary depends
on, which gets relinked every time so we detect link changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that this changes the order of arguments to pipecmd to match the
documentation, so we fix all the callers!
Also make configure re-run when configurator changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will kill a test that was running for 550 seconds, so that we
get a traceback before the travis inactivity timeout of 600 seconds
kicks in. The traceback should show us which test got stuck and where
it got stuck.
Signed-off-by: Christian Decker <@cdecker>
The only change is that the final_incorrect_htlc_amount field is now 64
bit. Since no implementation yet parses that field, we just updated it
quietly in the spec.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After Ubuntu 18.10 upgrade, lots of new flake8 warnings.
$ flake8 --version:
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 1.6.0) CPython 3.6.7rc1 on Linux
Note it seems that W503 warned about line breaks before binary
operators, and W504 complains about them after. I prefer W504, so
disable W503.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was from a different series, so I just cherry-picked it.
It adds ccan/membuf as a depenency of ccan/rbuf, though we don't use
it directly yet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If another channel has set the optional `htlc_maximum_msat` field,
we should correctly parse that field and respect it when drawing up
routes for payments.
The code to regenerate the local BOLT copy was causing eternal rebuild.
So only build if it's wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
BOLT 7's been updated to split the flags field in `channel_update`
into two: `channel_flags` and `message_flags`. This changeset does the
minimal necessary to get to building with the new flags.
