Commit Graph

830 Commits

Author SHA1 Message Date
ZmnSCPxj
48df6c8566 lightningd/io_loop_with_timers.c: Move mainloop to its own source file, have chaintopology use it.
Fixes: #2687
2019-05-31 17:57:10 +02:00
Christian Decker
883d4b4e6e pytest: Add a test that reproduces #2687
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-05-31 17:57:10 +02:00
Simon Vrouwe
52c84b2387 pytest: add test_funding_reorg_remote_lags
Nodes may disagree about short_channel_id before channel
announcement.
2019-05-27 13:29:32 +02:00
Simon Vrouwe
5b0c174251 pytest: add test_funding_reorg_private, reorg changes funding height
Reorg changes short_channel_id after lockin of private channel, while
one node restarts.

test that:
- peer->depth_togo in billboard decrements
- reorg and scid change is detected by running node and restarting node
- both `old` and `new` scids are in rtable

Also added a comment to test_blockchaintrack to clarify.
2019-05-27 13:29:32 +02:00
Simon Vrouwe
7726681aa6 pytest: added BitcoinD.simple_reorg() method, which can change tx height and/or txindex
Now without bitcoind restart.
bitcoin-cli `prioritisetransaction` came to the rescue!
Its argument `fee_delta` (apparently) lowers the txs _effective_ feerate
soo low that bitcoind wont mine it ... untill we raise it when we want
it to be mined.
2019-05-27 13:29:32 +02:00
Simon Vrouwe
24f43e5910 pytest: fundwallet tighten up, make sure we see CONFIRMED in log line 2019-05-27 13:29:32 +02:00
Simon Vrouwe
245807a085 pytest: tighten a log test Owning output... with address type and confirmation
Because the call (wallet_extract_owned outputs) that prints that line can happen
_before_ or _after_ confirmation in block, adding `CONFIRMED` in the later.
2019-05-27 13:29:32 +02:00
Christian Decker
8feb05aef4 subd: Remove overly verbose log when receiving a message
These tend to spam the logs, so I removed them :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-05-26 16:19:10 +02:00
Christian Decker
ced2ff8aa7 pytest: Mark repeat offenders as flaky
It's always the same few tests that fail, so I'm marking them as flaky and
will do a cleanup round later to individually address them.

I've been tracking the failures on the master branch for a few months now and
these are the worst offenders:

|-----------------------------------------|--------|--------|-------|
| test_name                               | state  | branch | count |
|-----------------------------------------|--------|--------|-------|
| test_pay_direct                         | FAILED | master | 31    |
| test_reconnect_gossiping                | FAILED | master | 20    |
| test_htlc_send_timeout                  | FAILED | master | 15    |
| test_pay_limits                         | FAILED | master | 13    |
| test_permfail                           | FAILED | master | 13    |
| test_funding_reorg_private              | FAILED | master | 12    |
| test_invoice                            | FAILED | master | 12    |
| test_invoice_preimage                   | FAILED | master | 12    |
| test_shutdown                           | FAILED | master | 12    |
| test_withdraw                           | FAILED | master | 12    |
| test_gossip_store_load_v3               | FAILED | master | 11    |
| test_onchain_multihtlc_their_unilateral | FAILED | master | 10    |
| test_opening_tiny_channel               | FAILED | master | 10    |
| test_channel_reenable                   | FAILED | master | 9     |
| test_crashlog                           | FAILED | master | 9     |
| test_gossip_weirdalias                  | FAILED | master | 9     |
| test_invoice_expiry                     | FAILED | master | 9     |
| test_onchain_multihtlc_our_unilateral   | FAILED | master | 9     |
| test_peerinfo                           | FAILED | master | 9     |
| test_private_channel                    | FAILED | master | 9     |
| test_shutdown_reconnect                 | FAILED | master | 9     |
| test_closing                            | FAILED | master | 8     |
| test_closing_different_fees             | FAILED | master | 8     |
| test_closing_while_disconnected         | FAILED | master | 8     |
| test_gossip_timestamp_filter            | FAILED | master | 8     |
| test_gossipwith                         | FAILED | master | 8     |
| test_invoice_routeboost                 | FAILED | master | 8     |
| test_onchain_middleman                  | FAILED | master | 8     |
| test_plugin_notifications               | FAILED | master | 8     |
| test_reconnect_channel_peers            | FAILED | master | 8     |
| test_bad_onion                          | FAILED | master | 7     |
| test_closing_torture                    | FAILED | master | 7     |
| test_fulfill_incoming_first             | FAILED | master | 7     |
| test_funding_reorg_disagree_scid_route  | FAILED | master | 7     |
| test_pay_disconnect                     | FAILED | master | 7     |
| test_balance                            | FAILED | master | 6     |
| test_check_command                      | FAILED | master | 6     |
| test_closing_id                         | FAILED | master | 6     |
| test_gossip_addresses                   | FAILED | master | 6     |
| test_gossip_notices_close               | FAILED | master | 6     |
| test_setchannelfee_usage                | FAILED | master | 6     |
| test_announce_address                   | FAILED | master | 5     |
| test_connect_by_gossip                  | FAILED | master | 5     |
| test_gossip_badsig                      | FAILED | master | 5     |
| test_gossip_disable_channels            | FAILED | master | 5     |
| test_gossip_jsonrpc                     | FAILED | master | 5     |
| test_gossip_persistence                 | FAILED | master | 5     |
| test_gossip_pruning                     | FAILED | master | 5     |
| test_onchain_different_fees             | FAILED | master | 5     |
| test_onchain_dust_out                   | FAILED | master | 5     |
|-----------------------------------------|--------|--------|-------|
2019-05-26 16:19:10 +02:00
Christian Decker
56f7efad46 pytest: Stabilize test_reconnect_gossiping
We weren't waiting for l2 to register the peer before asking it to ping it.
2019-05-26 16:19:10 +02:00
Christian Decker
a0fc915674 pytest: Start the Bitcoin RPC proxy as soon as it is created
Avoids attempting to rebind on restart.
2019-05-26 16:19:10 +02:00
Jan Sarenik
180ea5bb38 test_closing.py: isAlive -> is_alive
isAlive() is deprecated, use is_alive() instead
2019-05-26 15:07:33 +02:00
Rusty Russell
3d2af2237c pytest: fix test_htlc_send_timeout now pay doesn't return ROUTE_NOT_FOUND.
This brings up an interesting quirk though, in that we report "3
attempts", where we really should have done one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell
15dc0a5c18 pay: fix transient status 'failed' in listpays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell
18d2506958 pytest: test for transient "failed" status during ongoing payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell
71460ac073 pay: don't say "Could not find a route" unless we never tried a payment.
It's deeply confusing: we say this after exhausting all other routes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-22 04:32:34 +00:00
Rusty Russell
e5b5f1d7e5 openingd: add openchannel hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-20 20:31:07 -04:00
Rusty Russell
6ee2cd8ce3 openingd: fix hangup when gossipd compacts.
My raspberry pi node hung up on my other node:
   lightning_openingd-... chan #1: Got bad message from gossipd: 0db1

This is because we didn't handle that message in one path.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-16 15:54:17 -04:00
Rusty Russell
d054fef0f0 pytest: fix test_bech32_funding
We try to look up the funding tx, but it's already spent that to fund
the channel, so we need txindex if this test is to work reliably.

It's not clear to me why this *ever* worked, but if fails on my new
ThreadRipper build machine with valgrind:

  >	wallettx = l1.bitcoin.rpc.getrawtransaction(wallettxid, True)
...
  E           bitcoin.rpc.InvalidAddressOrKeyError: {'code': -5, 'message': 'No such mempool transaction. Use -txindex to enable blockchain transaction queries. Use gettransaction for wallet transactions.'}

  /usr/lib/python3/dist-packages/bitcoin/rpc.py:231: InvalidAddressOrKeyError
 
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-14 04:58:52 +02:00
Rusty Russell
7ede5aac31 gossip_store: change format so we store raw messages.
Save some overhead, plus gets us ready for giving subdaemons direct
store access.  This is the first time we *upgrade* the gossip_store,
rather than just discarding.

The downside is that we need to add an extra message after each
channel_announcement, containing the channel capacity.

After:
  store_load_msec:28337-30288(28975+/-7.4e+02)
  vsz_kb:582304-582316(582306+/-4.8)
  store_rewrite_sec:11.240000-11.800000(11.55+/-0.21)
  listnodes_sec:1.800000-1.880000(1.84+/-0.028)
  listchannels_sec:22.690000-26.260000(23.878+/-1.3)
  routing_sec:2.280000-9.570000(6.842+/-2.8)
  peer_write_all_sec:48.160000-51.480000(49.608+/-1.1)

Differences:
  -vsz_kb:582320
  +vsz_kb:582316
  -listnodes_sec:2.100000-2.170000(2.118+/-0.026)
  +listnodes_sec:1.800000-1.880000(1.84+/-0.028)
  -peer_write_all_sec:51.600000-52.550000(52.188+/-0.34)
  +peer_write_all_sec:48.160000-51.480000(49.608+/-1.1)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-13 05:16:18 +00:00
Rusty Russell
2a2341c56c bolt11: fix decoding and encoding of unknown fields.
Fixes: #2527
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-10 11:25:36 +00:00
Christian Decker
bc5dbb6a80 pytest: Ensure lightningd instances have the correct numeric ids
For performance reasons we start the lightningd instances in
parallel. However, if we only assign the numeric ids (used for log-prefixes
and home directories) when we are already running in parallel, we are not
guaranteed to get the numeric ids matching the return value of `get_nodes` or
`line_graph`. With this patch we now select numeric ids before parallelizing
the start.

Signed-off-by: Christian Decker <@cdecker>
2019-05-04 04:47:06 +00:00
trueptolemy
77f98f8a5c test_pay: Add test_forward_local_failed_stats() in 5 cases
Here I add the test for this 5 local_failed case in this commit.
There 5 cases for FORWARD_LOCAL_FAILED status:
    1. When Msater resolves the reply about the next peer infor(sent by Gossipd), and need handle unknown next peer failure in channel_resolve_reply();
    2. When Master handle the forward process with the htlc_in and the id of next hop, it tries to drive a new htlc_out but fails in forward_htlc();
    3. When we send htlc_out, Master asks Channeld to add a new htlc into the outgoing channel but Channeld fails. Master need handle and store this failure in rcvd_htlc_reply();
    4. When Channeld receives a new revoke message, if the state of corresponding htlc is RCVD_ADD_ACK_REVOCATION, Master will tries to resolve onionpacket and handle the failure before resolving the next hop in peer_got_revoke();
    5. When Onchaind finds the htlc time out or missing htlc, Master need handle these failure as FORWARD_LOCAL_FAILED in if it's forward payment case.
2019-05-03 11:45:57 +02:00
Rusty Russell
dddfdc7f10 pytest: add dev hack for testing option_upfront_shutdown_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:18:33 +00:00
Rusty Russell
cfebe66762 openingd: support receipt of upfront_shutdown_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:18:33 +00:00
Christian Decker
c96cc8a444 pytest: Fix the flaky test_plugin_connected_hook test
It was a bit eager :-)

Signed-off-by: Christian Decker <@cdecker>
2019-04-30 17:56:09 -05:00
Christian Decker
88f425fcc5 pytest: Make directory cleanup robust against setup failures
We were triggering a second exception in the directory cleanup step by
attempting to access a field that'd only be set upon entering the test code
itself. That error did not contribute to the problem resolution, so now we
check whether that field is set before accessing it.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-30 17:56:09 -05:00
Christian Decker
be853f563a wallet: Clamp maxheight to positive number for large minconf
Fixes #2518

Signed-off-by: Christian Decker <decker.christian@gmail.com>
Changelog-fixed: `minconf` no longer gets wrapped around for large values, which was causing funds with insufficient confirmations to be selected.
2019-04-30 17:52:52 -05:00
Christian Decker
e40b7c5584 pytest: Add test_minconf_withdraw to reproduce issue #2518
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-30 17:52:52 -05:00
Rusty Russell
ca7864f2f3 invoice_hook: remove nested result.
I misunderstood the API, this ended up nesting a result inside the JSON-RPC
result.

No concerns about backwards compatibility since this is so new.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-23 15:26:42 +02:00
Rusty Russell
7f7ad4f89f connected_hook: allow hook to specify an error message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-23 15:26:42 +02:00
Rusty Russell
bd655678a6 pytest: fix old code in test, which can cause spurious failures.
We generated blocks to announce the channel, but it can also expire
the HTLC if the timing is wrong.  We don't need to anyway, since we
fixed the FIXME; we store local unannounced channels for restoration

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-17 09:25:55 +02:00
Michael Schmoock
4985693bea feat: subtract dust reserves on the fly from min-capacity-sat 2019-04-16 15:01:28 -07:00
Michael Schmoock
3fa539fc1b chore: increase min-capacity-sat to 10k
The old value of 1000 sat was too small to cover the dust reserves.
This lead to the situation when trying to open a channel with minimal
amount, the channels got refused because they were not able cover the
commitment fees.

For this reason the minimal capacity should be increased to i.e. 10k
satoshi, as the technical minimum that also accounts for fees and
reserves is somewhere around 6k sat.
2019-04-16 15:01:28 -07:00
Michael Schmoock
7ed9bbd3dd test: make routeboost_private use higher capacity
This change is required, so we can increase the minimum channel capacity
(min-channel-sat) from 1k to 10k sat within the next commits.
2019-04-16 15:01:28 -07:00
Simon Vrouwe
c053dc9a6a lightningd: fix/refactor select_inchan for invoice route-hint, use fractional excess as weight
Refactored the weighted-reservoir-sampling algo to make it more straightforward.
It now uses the excess as fraction of capacity as weight. This favors channels that
are more _relatively_ unbalanced to be used for incoming payment.

Now passes test_invoice_routeboost_private() when using max fundamount=16777215.
2019-04-16 21:22:13 +00:00
lisa neigut
a8cc933351 closing: add message to billboard when closing txn is broadcast
make it a bit easier to track mutual channel closures by
adding broadcast txid to the listpeers billboard.

since lightningd manages the 'identity' of the closing tx we need
to send it back to closingd so it can update the billboard
appropriately.
2019-04-12 03:35:16 +00:00
Rusty Russell
9c373fecb6 pytest: test the invoice hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-12 03:32:48 +00:00
Rusty Russell
6630b99cf7 lightningd: move local invoice resolution into invoice.c function.
We're going to make it async, so start by moving the core code into
invoice.c and having that directly call fail/success functions for the
htlc.

We add an extra check in fulfill_htlc() that the HTLC state is correct:
that can't happen now, but may once we're async.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-12 03:32:48 +00:00
Rusty Russell
cccce75e56 patch refine-test_gossip_persistence.patch 2019-04-11 18:31:34 -07:00
Rusty Russell
da884751e8 gossipd: make routing_add_channel_update discard old timestamps.
This is currently done higher up, in handle_channel_update(), but
that's one reason why handle_channel_update() has to do a channel
lookup.  Moving the check down means handle_channel_update() can do a
minimal "get node id for this channel" so it can check the signature.

This helps, because the chan lookup semantics are changing in the next
few patches.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Rusty Russell
ec50ec6a71 gossipd: make gossip loading stats accurate.
They didn't count the header sizes when reporting bytes, which is
misleading.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-11 18:31:34 -07:00
Rusty Russell
ba41238df9 invoice: allow suffixes.
Makes it much easier to set it to 6 hours, for example.

Fixes: #2551
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-10 20:03:39 -07:00
Rusty Russell
77b859eaec lightning-cli: don't produce bad JSON if fields contain ".
The user can explicitly create such things (within [] or ") as we paste
those cases literally, but not for the simple cases.

Fixes: #2550
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-10 20:03:39 -07:00
Christian Decker
27afc804d5 json-rpc: Include received and resolved time to listforward result
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-10 23:48:39 +00:00
Rusty Russell
5009d628a3 lightning-cli: do pretty-printing.
Plugins don't do it right anyway, and we're about to remove it from
lightningd.  Produces same format as json_pp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-09 12:37:16 -07:00
Michael Schmoock
b0b86c9eb8 test: opening_tiny_channel with min_capacity_sat 2019-04-09 13:20:52 +02:00
Rusty Russell
ede7d0518e connectd: unlink any old socket when we use --bind-addr.
They don't clean up after themselves, so best we do it here (by this
point we've already done the pid check to make sure we're the only
lightningd here anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 22:00:00 +02:00
Rusty Russell
32215177fe pytest: more testing for local socket addresses.
In particular, the assert when `--addr=/sockname` is used, and that it
doesn't clean up on restart, requiring manual deletion of the socket.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 22:00:00 +02:00
Rusty Russell
2bd7df93c6 gossipd: preserve unannounced channels across store compaction.
Otherwise we'd forget them on restart, again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-08 04:41:43 +00:00