Commit Graph

6343 Commits

Author SHA1 Message Date
Rusty Russell
be8ebf2667 pytest: fix btcproxy mock logic.
You're supposed to be able to hand mock_rpc either a function to call,
or a dict canned response.  We never did the latter, and the logic
was broken.

It was testing the key, not the value for whether it was a dict.  And
it could never have given a valid response anyway, since it wouldn't
know the id to use.  So assume it's a successful result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 22:09:09 +02:00
Rusty Russell
7a592a2b5c contrib/startup_regtest.sh: avoid getting stuck in initialblockdownload.
And give a hint as to what cmds are available, since I forget!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00
Rusty Russell
2a09124574 contrib/startup_regtest.sh: set sane defaults for lightning & bitcoin dirs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:07:13 +02:00
Rusty Russell
97b39d84d5 README.md: more rewords.
0. Add an index
1. Mention regtest early on.
2. Link to doc/INSTALL.md for build-it-yourself.
   Since they're reading this doc, they're probably already on GH so don't discourage it.
3. Talk about contrib/startup_regtest.sh.
4. Use mainnet in the examples, since that's what people probably want.
5. Refer to default .lightning dir, and manpage.
6. Trim the JSON-API section down to simple commands.
7. Add section on getting started: bootstrap script and plugins.
8. Less details on config files, just refer to the man page.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 19:00:49 +02:00
Rusty Russell
125f14a8d5 pytest: fix flaky "peer reconnected" message in test_option_upfront_shutdown_script
If l2 doesn't think we're onchain yet, it treats the new connection from l1
as a reconnection, triggering 'ValueError: 1 nodes had unexpected reconnections'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 18:56:35 +02:00
Rusty Russell
4274b9f0af lightingd: increase listen queue on rpc socket.
I suspect multiple plugins trying to connect at the same
time are overrunning the 1-deep listen queue:

From man listen(2):

       The backlog argument defines the maximum length to which the  queue  of
       pending  connections  for sockfd may grow.  If a connection request ar‐
       rives when the queue is full, the client may receive an error  with  an
       indication  of ECONNREFUSED

Fixes: #2922
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 10:03:07 +00:00
lisa neigut
0c96c89d67 db-fix: resolve crash on fundchannel
Fixes error introduced by 1dbdc74bc where a new fundchannel
can cause a crash after start if the max dbid is for a closed
channel.
2019-08-10 02:52:13 +00:00
Rusty Russell
5e78960be0 CHANGELOG: note the support for #557.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
51541f53d8 gossipd: test vectors for https://github.com/lightningnetwork/lightning-rfc/pull/557
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
4de47f6db5 gossipd: use default zlib compression, hack for zlib expansion.
These both allow us to reproduce the test vectors in the next patch.  But
using Z_DEFAULT_COMPRESSION is a reasonable idea anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
f6cf4bf62a spec: remove encoding byte from checksums.
Make the TLV element a simple array.  This is a bit neater, in fact, and
makes the test vectors in that 557 PR work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
8abd850d3c gossipd: append timestamps & checksums to reply_channel_range if asked (EXPERIMENTAL)
In fact, we always generate them, we only send them if asked.  And we set
the flags to 0 if not --enable-experimental-features, so we never send in
that case.

Generating checksums involves pulling the channel_update from the
gossip_store, which is suboptimal: there's a FIXME to store the
checksum in memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
c7853197ae gossipd: generalize encoding functions
We're about to use the for gossip extended info too, which *don't* put
the encoding byte at the beginning of the data stream.  So this removes
some "scids" from function names and separates out the "prepend a byte"
case from the "external encoding_type" case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
0de11da5e4 gossipd: decode and obey query_short_channel_ids's TLV query_flags (EXPERIMENTAL)
These indicate what fields we are to return.  If there's now TLV, or we
haven't got --enable-experimental-features, it's set to all 1s so behaviour
is unchanged.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
e7728c0eea common: implement decoding of query-flags for query_short_channel_ids.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
Rusty Russell
d2030539e1 EXPERIMENTAL: pull in PR 557 (with minor fixes): range query support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-10 02:48:34 +00:00
darosior
b966e5e783 Add a test for 'fundchannel_start' crash on deconnection 2019-08-10 00:04:01 +08:00
Christian Decker
cc5f2dcae3 readme: First pass at homogenizing the readme a bit
We haven't touched the readme for quite some time, just randomly added to it,
and it's starting to show. This is my attempt at cleaning it up a bit (more to
come):

 - No longer discourage users from running on mainnet, we're way beyond that
   point.
 - No longer instruct users to build from source, when we have real binary
   releases, on the PPA, the releases page and the docker images.
 - Cut down on the docker specific instructions, they are taking a lot of room
   when only a minority will likely run them that way
 - Generally make the README more of a dispatch for more in-depth
   documentation rather than trying to address everything right on the
   front-page.
 - Add a bit of context about running on top of a pruned node

Signed-off-by: Christian Decker <decker.christian@gmail.com>


Header from folded patch 'fixup!_readme__first_pass_at_homogenizing_the_readme_a_bit.patch':

fixup! readme: First pass at homogenizing the readme a bit
2019-08-09 10:29:37 +00:00
Rusty Russell
710d015e5b lightningd: fix crash when peer disconnects after fundchannel_start, before cancel/complete
Fixes: #2831
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 10:57:42 +02:00
Rusty Russell
f9ecc76d99 gossipd: check that we don't try to access a deleted gossip entry.
We ignored this before, which meant that the DEVELOPER-mode check that we
delete the correct record didn't check that it wasn't already deleted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 08:58:05 +02:00
Rusty Russell
0edc0ae5e9 pytest: don't use deprecated options for close() in tests.
Only downside is you have to wait 1 second at least before
unilaterally closing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell
83e654a106 close: change to a unilateraltimeout argument.
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call.  `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.

Timing out JSON calls is generally deprecated: that's the job of the
client.  And the semantics of this are confusing, even to me!  A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.

The transition code is awkward, but we'll manage for the three
releases until we can remove it.

The new defaults are to unilaterally close after 48 hours.

Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rusty Russell
b35dc4689b pytest: enable deprecated APIs for multi-arg closes.
We're about to change the API, so this makes the tests still work
across the transition (and, as a bonus, tests our backwards compat
shim).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 05:47:16 +00:00
Rene Pickhardt
dfac1d15a2 included feedback by Rusty to check the max_concurrent_htlc value for both peers of a channel 2019-08-09 05:45:06 +00:00
Rene Pickhardt
1dd890825a doc: Document max-concurrent-htlcs option. 2019-08-09 05:45:06 +00:00
Rene Pickhardt
8e7428da53 Added possibility to configure max_concurrent_htlcs value for our channels. Eclaire has a default of 30 and I thought why not going with their value and while doing so make it configureable. 2019-08-09 05:45:06 +00:00
lisa neigut
0bd95fc068 contrib: add txindex to bitcoind for regtest 2019-08-09 05:07:18 +00:00
lisa neigut
0acdeeec06 fixup mkcommit's reverse ordered chainparams 2019-08-09 05:07:18 +00:00
darosior
f98ae2d1d3 Correct version regex in Makefile, add git as dependency 2019-08-09 02:49:45 +00:00
ZmnSCPxj
5dbff077c2 Makefile: Also install manpage for lightningd. 2019-08-09 02:36:59 +00:00
ZmnSCPxj
821832032d doc/lightningd.8.txt: Initial manpage for lightningd itself.
Closes: #1360
2019-08-09 02:36:59 +00:00
Rusty Russell
69b7ef1508 lightningd: fix up typesafe-cb bitcoind_getfilteredblock
`const struct filteredblock *` everywhere, as the typesafe_cb_preargs
macro required.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 02:31:51 +00:00
Christian Decker
5e36257375 pytest: Test that we delete channels opened below our birth height
This was the initial issue that was addressed by #2756 and now we just test
that all is working as expected.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
674017fa1e changelog: Add note on the block backfilling and cleanup fix
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Rusty Russell
99236f86f9 lightningd: rename 'satoshis' to 'amount' to avoid confusing check-source.
The type is enough (it's a struct amount_sat) to avoid confusion with
btc or msats.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-09 02:31:51 +00:00
Christian Decker
379079c5f3 gossip: Only backfill blocks that are below our birth height
If we were to just insert filtered blocks in the range that we will scan later
we'd be hitting the uniqueness constraints later.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
95d891ebf1 bitcoind: Queue up calls to getfilteredblock and dispatch results
Instead of allowing all calls to `getfilteredblock` to be scheduled on the
`bitcoind` queue right away we instead add them in a separate queue, and
process a single call at a time. This limits the concurrency and avoids
thrashing `bitcoind`. At the  same time we dispatch incoming results back to
all calls that were queued for that particular blockheight, reducing the
overall number of calls and an increase in overall speed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
040cda2875 bitcoind: Initialize filteredblock->outpoints with filteredblock
We will be calling the callback out of order once we fan out the results of a
single lookip to multiple calls, so being sure that everything is allocated
ahead of time is necessary.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
43b3eea783 bitcoind: Don't log when a transaction output is detected as spent
Since we now check all P2WSH outputs in a block, this is getting quite a
common occurence, so logging just produces lots of noise.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
98447e454e gossip: Use the getfilteredblock method to look up scid outputs
Just a tiny shim to reconcile the `get_output` with `getfileteredblock`.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
3dbaae38e3 wallet: Add function to add filteredblocks from backfilling
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
f4e434d8e1 bitcoind: Add a multi-step getfilteredblock method
This will eventually replace the multi-step `getblockhash` + `getblock` +
`gettxout` mechanism, and return entire filtered blocks which can be added to
the DB, and represent the full set of P2WSH UTXOs.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Christian Decker
5bb411b572 pytest: Add test for the block backfilling trigger by gossip
This is just the test that we use to verify block backfilling below the wallet
birth height is working correctly.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-08-09 02:31:51 +00:00
Rusty Russell
af0200f9d0 fixup! doc: fix up documentation about when we move into lightning-dir. 2019-08-08 18:17:12 +08:00
Rusty Russell
bcff30bb3f fixup! doc: fix up documentation about when we move into lightning-dir. 2019-08-08 18:17:12 +08:00
Rusty Russell
202ab91234 doc: fix up documentation about when we move into lightning-dir.
And make sure that plugins know that they should not touch things
until their init call.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-08 18:17:12 +08:00
Rusty Russell
118150227e cli: restore 0.7.0-style whitespace printing.
@renepickhardt has a shell script we broke.  While we still produce
perfectly valid JSON, we should not gratuitously change tool output.

Plus, I prefer the missing space before the ':'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-08 18:16:48 +08:00
Rusty Russell
2f30cdc731 contrib: simple script to bootstrap node.
The helpme plugin is more comprensive, but this at least connects to a
few random nodes, and doesn't require python libraries in path or anything.

I selected the nodes from helpme.py, eliminating ones I couldn't reach.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-07 21:13:17 +08:00
Rusty Russell
b73a85a75e lightningd: don't say 'killing channel' when HTLC times out.
We're actually only killing the connection.  I saw this in my logs,
but it was all OK.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-07 21:12:52 +08:00
darosior
cd91c06ce9 lightningd/notification: Add missing includes for 'forward_event'
And update test mocks
2019-08-07 01:55:38 +00:00