Commit Graph

6343 Commits

Author SHA1 Message Date
Rusty Russell
049529542a lightningd: delay reprocessing of incoming htlcs at startup until plugins ready.
Fixes: #2923
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-01 16:55:32 +02:00
Rusty Russell
f0ab4d72da pytest: add test that htlc_accepted hook only gets called after init.
We fail this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-01 16:55:32 +02:00
nicolas.dorier
49496ab3ba [Docker] Use stretch-slim instead of Alpine 2019-08-30 16:17:51 +02:00
darosior
8f8e955421 Update installation instructions and Dockerfile
Get rid of the 'asciidoc' dependency, make developers also install 'doc/requirements.txt'
2019-08-30 00:34:11 +00:00
darosior
092f20aeeb README: Replace broken links to manpages 2019-08-30 00:34:11 +00:00
Rusty Russell
3ef5160fa7 travis: add --enable-experimental-features (build-only) test.
Also, add a 32-bit build-only test, and move all the build-only tests
to the top.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Rusty Russell
02b9b7f6e6 tests: update mocks for --enable-experimental-features builds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Christian Decker
b77270fa46 docs: Add the title of the manpage to TOC and sort alphabetically
Fixes #2987
2019-08-29 14:56:24 +02:00
Rusty Russell
5e3e2f4e17 run-secret_eq_consttime: try harder.
This will never be reliable under high load, without making it unable
to detect real errors.  But the test is useful because if we don't
have this test we'll never notice if we break the const-timedness of
our implementation.

So, move the calloc out of the test loop (which seems to make it more
reliable), and then after we've run it, check the 1-minute load
average.  Too high, we don't complain about results.  It's not
perfect, but it's better.

Running 100 times (-O3) serially gave 100 successes with the following results:
	Constant: Within 5% 562-926(832.89+/-73)/1000 times
        Non-constant: More than 5% slower 860-990(956.35+/-26)/1000 times

More importantly, if we swap the const and non-const tests, we get
the expected 100 failures:
	Non-constant: Within 5% 14-79(41.17+/-14)/1000 times
	Constant: More than 5% slower 44-231(111.89+/-33)/1000 times

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
6901732ee0 lightningd: create --list-features-only which lists what features we support.
This allows the lightning-rfc protocol tests to automatically query what
features we support.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
17edcfa24d features: don't assume we'll always only advertize optional features.
Generalize things a bit so OPTIONAL_FEATURE() and COMPULSORY_FEATURE()
work with either odd or even features, then explicitly use OPTIONAL_FEATURE
in our internal feature array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
a38131f349 EXPERIMENTAL: advertize the extended_queries feature.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
d943d8abbc lightningd: expose full onion error when we have it.
Mainly useful for testing.  In particular, we don't save it to the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
189b2f1313 BOLT: update CSV to latest bolt version.
This removes the WIRE_FINAL_EXPIRY_TOO_SOON which leaked too much info,
and adds the blockheight to WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
2600a6ed2e channeld: get current block height when an HTLC fails.
We need it to put in the error code for
WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
6349222ea2 Spec: Update to latest BOLT, include our first global feature definition.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
6310a183af pytest: ignore occasional bad gossip in reorg tests.
It can happen, as expected when we reorg and update for a now-invalid
channel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
20a2bf9547 pytest: make test_forward_event_notification more explicit.
We seem to be getting intermittant failures, but it's hard
to disgnose.  Simplify it by moving all the test logic into
the test itself, and making the plugin dumber.  This means we'll
see exactly what the differences are if it fails again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
dbc1ebe06a Makefile: fix race when checking manpages.
By not depending on plugins, we can call lightningd before plugins are
built, and not get all options:

    diff of command names vs manpage names:
    --- /dev/fd/63  2019-08-22 05:06:55.265659216 +0000
    +++ /dev/fd/62  2019-08-22 05:06:55.265659216 +0000
    @@ -3,6 +3,8 @@
     allow-deprecated-apis=
     always-use-proxy=
     announce-addr=
    +autocleaninvoice-cycle=
    +autocleaninvoice-expired-by=
     autolisten=
     bind-addr=
     bitcoin-cli=

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
394aef2391 Makefile: fix race with 'make check'
installcheck calls $(MAKE) under the covers, which can race with the
current builds, and we can try installing something which is still
being built:

    collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
    compilation terminated.
    /usr/bin/ld: can not read symbols: File truncated
    /usr/bin/ld: .eh_frame/.stab edit: File truncated
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.interp' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.note.ABI-tag' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.note.gnu.build-id' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.hash' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynsym' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynstr' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.version' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.gnu.version_r' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.rela.dyn' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.rela.plt' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.init' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.plt' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.plt.got' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.text' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.fini' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.rodata' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.eh_frame' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.init_array' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.fini_array' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.data.rel.ro' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.dynamic' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.got' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `.data' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_json_command' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_hooks' not in segment
    /usr/bin/ld: lightningd/lightningd: warning: allocated section `xautodata_type_to_string' not in segment
    Makefile:390: recipe for target 'lightningd/lightningd' failed
    make[1]: *** [lightningd/lightningd] Error 1
    make[1]: Leaving directory '/home/rusty/lightning'
    Makefile:553: recipe for target 'installcheck' failed
    make: *** [installcheck] Error 2

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
3f1ca18f2f pytest: optimize test_closing_torture
VALGRIND=1, SLOW_MACHINE=0:
Before: 197.74 seconds
After: 135.43 seconds

Note that we now spend about 13 seconds in teardown, could probably
be optimized.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
686df235a2 pytest: fix race in test_block_backfill.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
11833192a1 tools: fix Makefile warning
Makefile:424: warning: overriding recipe for target 'clean'
    tools/Makefile:12: warning: ignoring old recipe for target 'clean'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-28 04:04:28 +00:00
Rusty Russell
28185c397c gossipd: fix gossip send in case query_flags cause no output.
Fortunately, again, only happens with EXPERIMENTAL_FEATURES.

If the query causes us not to actually send anything, we won't
get called again.  This can validly happen if they only asked for
the node_announcements, for example.

(Found by protocol tests).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Rusty Russell
855dff704c gossipd: test crc32 routines using test vectors from PR.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Rusty Russell
d1a1592cc8 gossipd: fix calculation of crc32 of update.
Currently EXPERIMENTAL_FEATURES only, fortunately.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-27 12:35:25 +02:00
Simon Vrouwe
5b5d70d640 lightningd/plugin: extend plugins->startup window to include plugins_config
Fixes incorrect configuration[`startup`] in plugin `init`, modified tests
to test this.
2019-08-27 00:02:20 +00:00
Rusty Russell
c2f58aec9d contrib/plugins: cowsay
___________________________________________________________________________
/ The most important plugin. Ever. Much thanks to @practicalswift and @jb55 \
\ for their improvements!                                                   /
 ---------------------------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 23:10:36 +00:00
Rusty Russell
0ec8304901 gossipd: fix premature towire_reply_short_channel_ids_end if no node_announcement.
Our "are we finished?" logic was wrong: it tested if there are no more
node_announcements, but it's possible that there were no node_announcements
for either end of the channel whose information we sent.

This is actually quite unusual on the real network: looking at mainnet
statis from last May, 4301 of 4337 nodes have node_announcements.

However, with query flags it's much more likely, since they might not
ask for node announcements at all.

(Found by gossip protocol tests)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 23:09:00 +00:00
Rusty Russell
af645b38c8 common: fix missing HOME case, when we override --lightning-dir.
It fails because option.c tal_free's the old value, which must be
a tal ptr.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 23:09:00 +00:00
Simon Vrouwe
2f5f364e17 pylightning/plugin: add lightning_version member to plugin 2019-08-26 15:39:03 -07:00
Simon Vrouwe
da74f0118d lightningd/plugin: at plugin start, set env LIGHTNINGD_VERSION
So plugins can know what version they are dealing with.
2019-08-26 15:39:03 -07:00
Rusty Russell
9dd314226d devtools/mkclose: make a mutual close transaction.
Guess who's been writing gossip protocol tests?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell
c9817b225b bitcoin/tx: pass struct amount_sat by copy.
This is the normal convention for this type; it makes using converters
a little easier.  See next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell
2f1e116510 gossipd: use htable_count() rather than reaching into htable struct.
Now ccan/htable provides the helper, let's use it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Rusty Russell
39b34a35c8 bitcoin/tx.c: don't free witness implicitly.
This causes a crash in mkfunding, which didn't expect it:

    $ devtools/mkfunding 16835ac8c154b616baac524163f41fb0c4f82c7b972ad35d4d6f18d854f6856b 1 0.01btc 253 76edf0c303b9e692da9cb491abedef46ca5b81d32f102eb4648461b239cb0f99 0000000000000000000000000000000000000000000000000000000000000010 0000000000000000000000000000000000000000000000000000000000000020
    # funding sig: 798d96d5a057b5b7797988a855217f41af05ece3ba8278366e2f69763c72e78565d5dd7eeddc0766ddf65557c92b9c52c301f23f94d2cf681860d32153e6ae1e
    # funding witnesses: [
    Aborted (core dumped)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-26 08:44:22 +00:00
Damian Mee
e4d0ef6e11 doc: fix wording for fundchannel_complete 2019-08-23 08:23:42 -07:00
Rusty Russell
997af6f3f4 contrib: make stricter shellcheck from Ubuntu 18.04 happy
In contrib/bootstrap-node.sh line 7:
if type lightning-cli >/dev/null 2>&1; then
   ^-- SC2039: In POSIX sh, 'type' is undefined.

In contrib/startup_regtest.sh line 41:
	type lightning-cli || return
        ^-- SC2039: In POSIX sh, 'type' is undefined.

In contrib/startup_regtest.sh line 42:
	type lightningd || return
        ^-- SC2039: In POSIX sh, 'type' is undefined.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-23 08:22:28 -07:00
darosior
3d851541f4 Travis: install doc/ requirements 2019-08-22 01:35:01 +00:00
darosior
529630a453 doc/Makefile: update readthedocs pages on doc-all
Stolen from @rustyrussell's #2928
2019-08-22 01:35:01 +00:00
darosior
77f34fad2b doc/Makefile: adapt 'check-manpages' to markdown 2019-08-22 01:35:01 +00:00
darosior
49be9b5545 doc/manpages: Add manpages to readthedocs 2019-08-22 01:35:01 +00:00
darosior
d37a2d7c3e doc/manpages: regenerate manpages using markdown templates 2019-08-22 01:35:01 +00:00
darosior
76a4c5f287 doc/Makefile: Use mrkd to generate manpages 2019-08-22 01:35:01 +00:00
darosior
80927039e0 doc/manpages: replace asciidoc by markdown 2019-08-22 01:35:01 +00:00
darosior
3f29216f7b doc/Makefile: Add lightning-plugin manpage 2019-08-22 01:35:01 +00:00
Rusty Russell
0954feddc7 json: speed up shutdown.
We currently end up sleeping for 1 second for channeld and gossipd:
better to use a normal blocking waitpid and an alarm to wake us in
case they don't exit.

This speeds up `lightning-cli stop` on my machine from 2.008s to 0.008s:
a 286 times speedup!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-22 01:34:03 +00:00
Rusty Russell
9dfc854d94 tools/Makefile: put all tools/ stuff here.
Generally a Makefile should control make within that directory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell
278b12982d tools/test/Makefile: disable unless we're in DEVELOPER mode.
Because it required update-mocks, which is a hack which relies on the
format of linker errors (!) I'd prefer to make this --enable-developer
only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00
Rusty Russell
c74c1e56cc tools/test/Makefile: fix always-on generation of tools/test/gen_test.c
We need to tell it that the tmp file is an intermediate, so doesn't need
remaking if it doesn't exist.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-21 05:07:15 +00:00