Commit Graph

1979 Commits

Author SHA1 Message Date
niftynei
4506f639fa coin_moves: remove 'index' for moves; bump version
Get rid of the 'movement_idx', since we replay events now.

Since we're removing a field from the 'coin_movement' event emission, we
bump the version type.

Changelog-Updated: `coin_movements` events have been revamped and are now on version 2.
2021-12-28 04:42:42 +10:30
niftynei
27ff87c045 coin_moves: de-dupe moves as they come in
test_onchain_dust_out restarts a node, which produces duplicate events.
this is expected, but we need to de-duplicate the events stream to get
accurate results
2021-12-28 04:42:42 +10:30
niftynei
d2c4d4aec2 coin_mvts: rewrite how onchain events are recorded, update tests
The old model of coin movements attempted to compute fees etc and log
amounts, not utxos. This is not as robust, as multi-party opens and dual
funded channels make it hard to account for fees etc correctly.

Instead, we move towards a 'utxo' view of the onchain events. Every
event is either the creation or 'destruction' of a utxo. For cases where
the value of the utxo is not (fully) debited/credited to our account, we
also record the output_value. E.g. channel closings spend a utxo who's
entire value we may not own.

Since we're now tracking UTXOs onchain, we can now do more complex
assertions about the onchain footprint of them. The integration tests
have been updated to now use more 'chain aware' assertions about the
ending state.
2021-12-28 04:42:42 +10:30
Vincenzo Palazzo
43ff949ea7 lightningd: support hsm error code
Suggested-by: Rusty Russell
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Changed: Support hsm specific error error code in lightning-cli
2021-12-15 12:24:54 +10:30
Rusty Russell
1ac5a1b7e2 pytest: restore gossip advertizement tests for default port.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-14 13:41:31 +10:30
Simon Vrouwe
28816c387c testing: remove test_htlc_accepted_hook_shutdown, move relevant parts
into test_plugin_shutdown

The fact that plugins are kept alive untill the very end also ensures
their hooks are not silently unregistered.
2021-12-14 09:33:10 +10:30
Simon Vrouwe
9b70c9d63b testing: fix flake in test_fetchinvoice_3hop 2021-12-14 09:33:10 +10:30
Michael Schmoock
c2d2cc1274 connectd: fix empty error message
1. Adds the missing DNS error massages so they can be handled by
   connect_control.
2. Prepends a 'All addresses failed' to code 401 message, so we
   always have at least some error message to the user.

Changelog-None
2021-12-08 13:52:24 +01:00
Michael Schmoock
bad09887e0 pytest: fix test_pluginconnected_hook_chaining
The last line of the testcase was checking on the wrong node l3
instead of l1. l3 didn't had the plugins configured that would
produce the log entry we were looking for not to be present.

Changelog-None
2021-12-06 14:53:37 +01:00
Jan Sarenik
96f28323bd Set default port according to network
The idea is to have different default ports for different networks.

Current default port is `9735` for everything. Let's use it for
the mainnet and reuse the difference added to the default port
from `rpc_port` values in `bitcoin/chainstate.c`.

Testnet would be `19735` (adding rpc_port - 8332 = `10000`).

Signet would be `39735` (adding rpc_port - 8332 = `30000`).

Regtest would be `19846` (adding rpc_port - 8332 = `10111`).

With Vincenzo's kind pair-programming help over tmate.

Two other commits were squashed into this one so that bisecting
never ends up in half-baked state:

1. chainparams: Fix regtest default rpc_port

   bitcoind -help says this:

    -rpcport=<port>
         Listen for JSON-RPC connections on <port> (default: 8332, testnet:
         18332, signet: 38332, regtest: 18443)

2. test_gossip: Default port for regtest

   hex: 2607 is now .... (could be 4d86 but Elements uses another port)
   dec: 9735 is now any port (could be 19846 ^^ but now is for any port)

   The lines which were binding to default port were removed as the
   default port is different on each network.

NOTE: Remember not to modify gossip_store tests which loads everything raw
      including the checksums.

Changelog-Changed: If the port is unspecified, the default port is chosen according to used network similarly to Bitcoin Core.
2021-12-06 17:10:08 +10:30
Rusty Russell
4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell
bdabef9a9b pytest: fix occasional flake in test_announce_address
We make sure the gossip msg was sent, but the other node might not
have digested it yet:

```
        # Check other node can parse these
>       addresses = l2.rpc.listnodes(l1.info['id'])['nodes'][0]['addresses']
E       KeyError: 'addresses'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:18 +10:30
Rusty Russell
b45544c659 options: fix handling of wildcard (allproto) address.
We treated ':' as an empty DNS name in EXPERIMENTAL, which is wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-05 16:43:09 +01:00
Rusty Russell
cf40e585c3 pytest: test to show conflict between websocket and wildcard addresses.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-05 16:43:09 +01:00
Andrew Toth
69bc1191cb tests: add pay test for exclude arg 2021-12-04 21:37:42 +10:30
Rusty Russell
5a5cf8c696 pytest: fix flake in testing.
As noted in 0a406230d0 (diff-5871d4c569454db5e625383975462132da0bd03d32df145d8d72d8fafd86d952R3544-R3546)

Turns out we sometimes hang up before l2 sees the previous tx revoked,
so we get a normal unilateral close, not a cheat.

Reported-by: Simon Vrouwe
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 09:48:00 +10:30
Rusty Russell
605b3bf985 pytest: re-enable modern/obsolete fetchonion tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell
b74848f6f6 common: remove support for pre v0.10.2 onionmessages.
Temporarily disable sendpay_blinding test which uses obsolete onionmsg;
there's still some debate on the PR about how blinded HTLCs will work.

Changelog-EXPERIMENTAL: onionmessage: removed support for v0.10.1 onion messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Simon Vrouwe
aae5e7822f testing: fix test_upgrade_statickey_onchaind
Seems a timing issue that should be figured out, his makes the test pass.
2021-11-30 13:34:44 +10:30
Simon Vrouwe
2240f09f8b testing: fix test_closing_higherfee
shutdown_subdaemons frees the channel and calls destroy_close_command_on_channel_destroy, see gdb:

    0  destroy_close_command_on_channel_destroy (_=0x55db6ca38e18, cc=0x55db6ca43338) at lightningd/closing_control.c:94
    1  0x000055db6a8181b5 in notify (ctx=0x55db6ca38df0, type=TAL_NOTIFY_FREE, info=0x55db6ca38e18, saved_errno=0) at ccan/ccan/tal/tal.c:237
    2  0x000055db6a8186bb in del_tree (t=0x55db6ca38df0, orig=0x55db6ca38e18, saved_errno=0) at ccan/ccan/tal/tal.c:402
    3  0x000055db6a818a47 in tal_free (ctx=0x55db6ca38e18) at ccan/ccan/tal/tal.c:486
    4  0x000055db6a73fffa in shutdown_subdaemons (ld=0x55db6c8b4ca8) at lightningd/lightningd.c:543
    5  0x000055db6a741098 in main (argc=21, argv=0x7ffffa3e8048) at lightningd/lightningd.c:1192

Before this PR, there was no io_loop after shutdown_subdaemons and client side raised a
general `Connection to RPC server lost.`
Now we test the more specific `Channel forgotten before proper close.`, which is good!

BTW, this test was added recently in PR #4599.
2021-11-30 13:34:44 +10:30
Simon Vrouwe
89cbdf4dce testing: test that RPC calls made in shutdown fail and receive error code -5
and the two conditions in which plugins can receive shutdown notification
2021-11-30 13:34:44 +10:30
Simon Vrouwe
ef503f2fea testing: remove test_stop_pending_fundchannel
Not needed anymore, see previous commit
2021-11-30 13:34:44 +10:30
Simon Vrouwe
5fb3674233 testing: test hook semantics is preserved in shutdown
plugins expect their hooks to work also in shutdown, see issue #4883
2021-11-30 13:34:44 +10:30
Michael Schmoock
a3ea9fdc87 chore: use EXPERIMENTAL for BOLT7 DNS #911
Changelog-EXPERIMENTAL: Ability to announce DNS addresses
2021-11-30 09:38:17 +10:30
Michael Schmoock
2b8896a0b5 pytest: test connecting to a DNS only announced node 2021-11-30 09:38:17 +10:30
Michael Schmoock
01e8a523e9 bolt7: allow announcement of ADDR_TYPE_DNS 2021-11-30 09:38:17 +10:30
Rusty Russell
7a6fd70078 pytest: Reduce memory consumption by test_plugin_disable
It runs 6 nodes: under valgrind this ends up consuming 5.3 GB RSS.  By
stopping nodes between, we peak about 1G RSS.

Measured using:
	(while true; do echo $(for i in 4 5 6; do ps uh | tr -s ' ' | cut -d\  -f$i | tally; done); sleep 5; done)&

(Which measures my other processes as well, but that's only about 100M).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-27 12:41:37 +00:00
ZmnSCPxj jxPCSnmZ
a294683675 wallet/db_sqlite3.c: Support direct replication of SQLITE3 backends.
ChangeLog-Added: With the `sqlite3://` scheme for `--wallet` option, you can now specify a second file path for real-time database backup by separating it from the main file path with a `:` character.
2021-11-17 12:10:07 +10:30
Rusty Russell
68043c2e8c common: clean up autodata in common_shutdown().
valgrind locally complains about the allocations in autodata leaking:

```
==138200== 16 bytes in 1 blocks are still reachable in loss record 1 of 2
==138200==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==138200==    by 0x10D41A: autodata_register_ (autodata.c:20)
==138200==    by 0x10E7B8: register_autotype_type_to_string (type_to_string.h:79)
==138200==    by 0x10F5CA: register_one_type_to_string0 (block.c:259)
==138200==    by 0x19734C: __libc_csu_init (in /home/rusty/devel/cvs/lightning/common/test/run-route-specific)
==138200==    by 0x4A3D03F: (below main) (libc-start.c:264)
==138200== 
==138200== 176 bytes in 1 blocks are still reachable in loss record 2 of 2
==138200==    at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==138200==    by 0x10D472: autodata_register_ (autodata.c:26)
==138200==    by 0x122D37: register_autotype_type_to_string (type_to_string.h:79)
==138200==    by 0x122F1F: register_one_type_to_string0 (node_id.c:50)
==138200==    by 0x19734C: __libc_csu_init (in /home/rusty/devel/cvs/lightning/common/test/run-route-specific)
==138200==    by 0x4A3D03F: (below main) (libc-start.c:264)
==138200== 
make: *** [Makefile:638: unittest/common/test/run-route-specific] Error 7
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
82c66b10df pytest: allow bad gossip in pruning test.
If we forget a channel, we can get upset when we get an update about it:

```
2021-11-04T00:35:43.8242370Z lightningd-3: 2021-11-04T00:29:22.073Z DEBUG   gossipd: Pruning channel 103x1x1 from network view (ages 61 and 22s)
...
2021-11-04T00:35:43.8263502Z lightningd-3: 2021-11-04T00:29:22.509Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x1/0
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
65bb989cf1 pytest: don't checksum plugins on startup in VALGRIND developer mode.
This loads up 20MB of plugins temporarily; we seem to be getting OOM
killed under CI and I wonder if this is contributing.

Doesn't significantly reduce runtime here, but I have lots of memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell
09bbda4bca pytest: don't create 5 nodes in test_fetchinvoice.
CI seems to be OOM killing us; 5 may be too many under valgrind.

VALGRIND=1 pytest tests/test_pay.py::test_fetchinvoice 
Before:
	1 passed in 199.33s (0:03:19)

After:
	1 passed in 177.91s (0:02:57)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell
9d18180172 lightningd: really do allow two Torv3 addresses.
This surprised me, since the CHANGELOG for [0.8.2] said:

	We now announce multiple addresses of the same type, if given. ([3609](https://github.com/ElementsProject/lightning/pull/3609))

But it lied!

Changelog-Fixed: We really do allow providing multiple addresses of the same type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell
2f247c7bfb torv2: remove support for advertizing and connecting.
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.

I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Rusty Russell
c00202a0da pytest: remove test_shutdown_alternate_txid.
We're about to require that fundchannel_complete() take a PSBT, where it
does sanity checks to avoid this error, making this a difficult mistake
to make.

Is it time to remove this functionality anyway?  @cdecker?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Christian Decker
012c2ecf17 pytest: Create less dust in test_delpay_payment_split
We were triggering the dust panic.
2021-10-23 12:59:13 +02:00
niftynei
303fbce479 tests: raise dust limit on mpp test
Fails liquid-regtest otherwise; liquid tends to hit the dust limit
earlier than non-liquid tx, and MPP exacerbates this by divvying up
payments into dusty bits then attempting to shove them through the same
channel, hitting the dust max. The MPP then fails as not all the parts
were able to arrive at their destination.
2021-10-23 12:59:13 +02:00
niftynei
b57fed047a dusty htlcs: don't fail the channel, make it error a whole bunch
Let's make this a softer launch by just warning on the channel til the
feerates go back down.

You can also 'fix' this by upping your dust limit with
the `max-dust-htlc-exposure-msat` config.
2021-10-23 12:59:13 +02:00
niftynei
150d065088 tests: check for incoming + outgoing dust limits 2021-10-23 12:59:13 +02:00
niftynei
208b161226 tests: bump the ceiling for dust, fixes failures for liquid
Liquid's threshold for dust is a bit higher, so we bump up the max limit
here so we can actually get the whole MPP'd payment sent over the wire
2021-10-23 12:59:13 +02:00
Rusty Russell
ed6eaf9171 experimental-websocket-port: option to create a WebSocket port.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-22 11:56:30 +02:00
Rusty Russell
c503232cde common: use bitcoin_outpoint.
I started pulling this thread, and the entire codebase got unravelled.

Oh well, it's done now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
012dedc3d8 fuzz: make it build again.
How did this pass CI?  I saw this break in my PR, but it's (long)
broken in master.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Vincenzo Palazzo
67b790e3dc tests: Change currency from msat to sat in the regex string
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-10-14 10:09:20 +10:30
Christian Decker
cd7d87f98e pay: listpays groups by payment_hash and groupid
Fixes #4482
Fixes #4481

Changelog-Added: pay: Payment attempts are now grouped by the pay command that initiated them
Changelog-Fixed: pay: `listpays` returns payments orderd by their creation date
Changelog-Fixed: pay: `listpays` no longer groups attempts from multiple attempts to pay an invoice
2021-10-13 13:41:18 +10:30
Christian Decker
98f74762a5 pytest: Adjust test_sendpay to the new semantics 2021-10-13 13:41:18 +10:30
Christian Decker
ce3d3d8e54 pytest: Fix test_onchain_timeout to use groupid 2021-10-13 13:41:18 +10:30
Christian Decker
428982a9e7 pytest: Add groupid to test_partial_payment_{timeout,restart} 2021-10-13 13:41:18 +10:30
Christian Decker
bc089d3350 pytest: Fix up test_partial_payment to use a single groupid 2021-10-13 13:41:18 +10:30
Christian Decker
324257b2bf pytest: Reproduce #4482 2021-10-13 13:41:18 +10:30