Commit Graph

6310 Commits

Author SHA1 Message Date
Rusty Russell
bf3b77a947 Travis: skip testing VALGRIND=1 DEVELOPER=0, remove the slowest non-developer tests.
I don't remember ever seeing a bug which only showed up in VALGRIND=1 with developer
mode disabled, so don't test that, and spread out the other test more evenly.

In addition, disable the worst-performing tests in DEVELOPER=0 mode.

Here timings from my build machine: the worst 6 (- DEVELOPER=0 VALGRIND=0)
with the same tests (+ DEVELOPER=1 VALGRIND=1)

-452.42s call     tests/test_pay.py::test_channel_spendable
+87.69s call     tests/test_pay.py::test_channel_spendable
-335.66s call     tests/test_gossip.py::test_gossip_store_compact_on_load
+47.41s call     tests/test_gossip.py::test_gossip_store_compact_on_load
-332.07s call     tests/test_connection.py::test_opening_tiny_channel
+89.71s call     tests/test_connection.py::test_opening_tiny_channel
-331.97s call     tests/test_pay.py::test_channel_spendable_large
+56.23s call     tests/test_pay.py::test_channel_spendable_large
-305.28s call     tests/test_invoices.py::test_invoice_routeboost
+37.57s call     tests/test_invoices.py::test_invoice_routeboost
-284.28s call     tests/test_plugin.py::test_htlc_accepted_hook_forward_restart
+49.12s call     tests/test_plugin.py::test_htlc_accepted_hook_forward_restart

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 11:14:38 +00:00
Rusty Russell
63103db3f3 pytest: disable some v. slow tests when !DEVELOPER.
We've gone over 50 minutes with valgrind, and my measurements show
these are the slowest.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 05:53:36 +00:00
Rusty Russell
a7a9f1c9d3 onchaind: fix bug when grinding feerates with multiple possibilities.
Fixes: #2820
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 05:53:36 +00:00
Rusty Russell
9424477cb5 onchaind/test: new test, for grinding feerate when multiple possibilities.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-14 05:53:36 +00:00
Rusty Russell
0f5036d866 cli: remove formatting hint correclty when it's not the last element.
I noticed that the 'lightning-cli summary' output was wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-13 20:45:45 +00:00
Rusty Russell
69b02e287a cli: avoid use-after-realloc when we delete format hint.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-13 20:45:45 +00:00
Rusty Russell
f1e84b3d99 devtools: make clean should remove devtools/gen_print_onion_wire.[c,h,o]
Reported-by: @JavierRSobrino
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-13 05:12:10 +00:00
nicolas.dorier
6cf42fe6c1 Add python-mako to the arm32 build 2019-08-12 05:44:11 +00:00
Rusty Russell
3fec154413
CHANGELOG.md: shuffle and format into order, label rc1
(Squashed in fixup by Darosior <darosior@protonmail.com>)
2019-08-11 22:54:25 +00:00
Cryptcoin Junkey
bdf1ffcb4f Enable to build the docker image (ElementsProject#2941).
Signed-off-by: Cryptcoin Junkey <cryptcoin.junkey@gmail.com>
2019-08-11 18:21:03 +02:00
nicolas.dorier
19cf199f0e Add docker ARM64 support 2019-08-11 18:19:22 +02:00
Rusty Russell
854e3c5303 pytest: wait for sync by default when starting lightningd.
Otherwise we get some spurious test failures.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
ca28c30eff funding: don't allow funding new channels until we're synced.
This is probably worth preventing.

1. Our depth estimate would be inaccurate possibly leading to us
   timing out too early.
2. If we're not up-to-date our onchain funds are unknown.
3. We wouldn't be able to send or receive HTLCs until we're synced anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
c3a35416da lightningd: don't allow channeld to accept HTLCs if we're not synced.
We want to still allow incoming connections, and reestablishment of
channels, but if one tries to give us an HTLC, stall until we're
synced.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
6195a878f7 lightningd: don't allow sending of HTLCs while still syncing.
If we don't know block height, we shouldn't be sending HTLCs.  This
stops us forwarding HTLCs as well as new payments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
3eebd0cc20 lightningd: add flag for whether we're synced, and callback infrastructure.
We consider ourselves synced when bitcoind is synced and we're synced
with that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
faded9a9cf bitcoind: detect when it's still syncing, add field to getinfo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
be8ebf2667 pytest: fix btcproxy mock logic.
You're supposed to be able to hand mock_rpc either a function to call,
or a dict canned response.  We never did the latter, and the logic
was broken.

It was testing the key, not the value for whether it was a dict.  And
it could never have given a valid response anyway, since it wouldn't
know the id to use.  So assume it's a successful result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
7a592a2b5c contrib/startup_regtest.sh: avoid getting stuck in initialblockdownload.
And give a hint as to what cmds are available, since I forget!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00
Rusty Russell
2a09124574 contrib/startup_regtest.sh: set sane defaults for lightning & bitcoin dirs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00
Rusty Russell
97b39d84d5 README.md: more rewords.
0. Add an index
1. Mention regtest early on.
2. Link to doc/INSTALL.md for build-it-yourself.
   Since they're reading this doc, they're probably already on GH so don't discourage it.
3. Talk about contrib/startup_regtest.sh.
4. Use mainnet in the examples, since that's what people probably want.
5. Refer to default .lightning dir, and manpage.
6. Trim the JSON-API section down to simple commands.
7. Add section on getting started: bootstrap script and plugins.
8. Less details on config files, just refer to the man page.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:00:49 +02:00
Rusty Russell
125f14a8d5 pytest: fix flaky "peer reconnected" message in test_option_upfront_shutdown_script
If l2 doesn't think we're onchain yet, it treats the new connection from l1
as a reconnection, triggering 'ValueError: 1 nodes had unexpected reconnections'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 18:56:35 +02:00
Rusty Russell
4274b9f0af lightingd: increase listen queue on rpc socket.
I suspect multiple plugins trying to connect at the same
time are overrunning the 1-deep listen queue:

From man listen(2):

       The backlog argument defines the maximum length to which the  queue  of
       pending  connections  for sockfd may grow.  If a connection request ar‐
       rives when the queue is full, the client may receive an error  with  an
       indication  of ECONNREFUSED

Fixes: #2922
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 10:03:07 +00:00
lisa neigut
0c96c89d67 db-fix: resolve crash on fundchannel
Fixes error introduced by 1dbdc74bc where a new fundchannel
can cause a crash after start if the max dbid is for a closed
channel.
2019-08-10 02:52:13 +00:00
Rusty Russell
5e78960be0 CHANGELOG: note the support for #557.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
51541f53d8 gossipd: test vectors for https://github.com/lightningnetwork/lightning-rfc/pull/557
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
4de47f6db5 gossipd: use default zlib compression, hack for zlib expansion.
These both allow us to reproduce the test vectors in the next patch.  But
using Z_DEFAULT_COMPRESSION is a reasonable idea anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
f6cf4bf62a spec: remove encoding byte from checksums.
Make the TLV element a simple array.  This is a bit neater, in fact, and
makes the test vectors in that 557 PR work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
8abd850d3c gossipd: append timestamps & checksums to reply_channel_range if asked (EXPERIMENTAL)
In fact, we always generate them, we only send them if asked.  And we set
the flags to 0 if not --enable-experimental-features, so we never send in
that case.

Generating checksums involves pulling the channel_update from the
gossip_store, which is suboptimal: there's a FIXME to store the
checksum in memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
c7853197ae gossipd: generalize encoding functions
We're about to use the for gossip extended info too, which *don't* put
the encoding byte at the beginning of the data stream.  So this removes
some "scids" from function names and separates out the "prepend a byte"
case from the "external encoding_type" case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
0de11da5e4 gossipd: decode and obey query_short_channel_ids's TLV query_flags (EXPERIMENTAL)
These indicate what fields we are to return.  If there's now TLV, or we
haven't got --enable-experimental-features, it's set to all 1s so behaviour
is unchanged.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
e7728c0eea common: implement decoding of query-flags for query_short_channel_ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
d2030539e1 EXPERIMENTAL: pull in PR 557 (with minor fixes): range query support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
darosior
b966e5e783 Add a test for 'fundchannel_start' crash on deconnection 2019-08-10 00:04:01 +08:00
Christian Decker
cc5f2dcae3 readme: First pass at homogenizing the readme a bit
We haven't touched the readme for quite some time, just randomly added to it,
and it's starting to show. This is my attempt at cleaning it up a bit (more to
come):

 - No longer discourage users from running on mainnet, we're way beyond that
   point.
 - No longer instruct users to build from source, when we have real binary
   releases, on the PPA, the releases page and the docker images.
 - Cut down on the docker specific instructions, they are taking a lot of room
   when only a minority will likely run them that way
 - Generally make the README more of a dispatch for more in-depth
   documentation rather than trying to address everything right on the
   front-page.
 - Add a bit of context about running on top of a pruned node

Signed-off-by: Christian Decker <decker.christian@gmail.com>


Header from folded patch 'fixup!_readme__first_pass_at_homogenizing_the_readme_a_bit.patch':

fixup! readme: First pass at homogenizing the readme a bit
2019-08-09 10:29:37 +00:00
Rusty Russell
710d015e5b lightningd: fix crash when peer disconnects after fundchannel_start, before cancel/complete
Fixes: #2831
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 10:57:42 +02:00
Rusty Russell
f9ecc76d99 gossipd: check that we don't try to access a deleted gossip entry.
We ignored this before, which meant that the DEVELOPER-mode check that we
delete the correct record didn't check that it wasn't already deleted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 08:58:05 +02:00
Rusty Russell
0edc0ae5e9 pytest: don't use deprecated options for close() in tests.
Only downside is you have to wait 1 second at least before
unilaterally closing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell
83e654a106 close: change to a unilateraltimeout argument.
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call.  `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.

Timing out JSON calls is generally deprecated: that's the job of the
client.  And the semantics of this are confusing, even to me!  A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.

The transition code is awkward, but we'll manage for the three
releases until we can remove it.

The new defaults are to unilaterally close after 48 hours.

Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell
b35dc4689b pytest: enable deprecated APIs for multi-arg closes.
We're about to change the API, so this makes the tests still work
across the transition (and, as a bonus, tests our backwards compat
shim).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rene Pickhardt
dfac1d15a2 included feedback by Rusty to check the max_concurrent_htlc value for both peers of a channel 2019-08-09 05:45:06 +00:00
Rene Pickhardt
1dd890825a doc: Document max-concurrent-htlcs option. 2019-08-09 05:45:06 +00:00
Rene Pickhardt
8e7428da53 Added possibility to configure max_concurrent_htlcs value for our channels. Eclaire has a default of 30 and I thought why not going with their value and while doing so make it configureable. 2019-08-09 05:45:06 +00:00
lisa neigut
0bd95fc068 contrib: add txindex to bitcoind for regtest 2019-08-09 05:07:18 +00:00
lisa neigut
0acdeeec06 fixup mkcommit's reverse ordered chainparams 2019-08-09 05:07:18 +00:00
darosior
f98ae2d1d3 Correct version regex in Makefile, add git as dependency 2019-08-09 02:49:45 +00:00
ZmnSCPxj
5dbff077c2 Makefile: Also install manpage for lightningd. 2019-08-09 02:36:59 +00:00
ZmnSCPxj
821832032d doc/lightningd.8.txt: Initial manpage for lightningd itself.
Closes: #1360
2019-08-09 02:36:59 +00:00
Rusty Russell
69b7ef1508 lightningd: fix up typesafe-cb bitcoind_getfilteredblock
`const struct filteredblock *` everywhere, as the typesafe_cb_preargs
macro required.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 02:31:51 +00:00
Christian Decker
5e36257375 pytest: Test that we delete channels opened below our birth height
This was the initial issue that was addressed by #2756 and now we just test
that all is working as expected.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00