Commit graph

12795 commits

Author SHA1 Message Date
Alex Myers
2f050621b0 pytest: add blackbox tests for reckless
A canned lightningd/plugins is used to test against. This allows faster
and more deterministic outcomes.

Changelog-None
2023-04-09 12:41:11 +09:30
Alex Myers
cf203369bc reckless: use environment variable redirects
This will be used during CI testing in the following commit.
2023-04-09 12:41:11 +09:30
Alex Myers
55cddcd350 reckless: add support for additional networks
This should have been added earlier as @cdecker suggested, but is needed
to enable CI testing.

Changelog-Changed: Reckless - added support for networks beyond bitocoin and regtest
2023-04-09 12:41:11 +09:30
Alex Myers
e61401aab9 reckless: don't crash on subprocess calls
They prefer Paths to be explicitly cast as strings
2023-04-09 12:41:11 +09:30
Rusty Russell
88905e8372 tests: split fetchinvoice recurrence tests into separate test.
This is for VLS, which doesn't implement signing for the non-standard
recurrence fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:23:14 +09:30
Rusty Russell
d4ffc75691 Makefile: update to latest BOLT text.
In particular:
	- Bolt 4: add route blinding construction
	- Bolt 4: add blinded payments

And this means it's not experimental, so we can turn it on
by default!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: blinded payments are now supported by default (not just with `--experimental-onion-messages`)
2023-04-07 21:22:56 +09:30
Rusty Russell
458a85042b Makefile: update to BOLT 20066dc2aba906f37f3be5a810ae67040f265377
"BOLT 03: fix static-remote same amt and pre-image test vector"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
f26b1166b7 Makefile: update bolts a0bbe47b0278b4f152dbaa4f5fab2562413a217c
"BOLT 04: remove associated data from test vector"

(We actually use merge point).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
dfa6c0ca52 Makefile: bolt version b38156b9510c0562cf50f8758a64602cc0315c19
"Allow nodes to overshoot final htlc amount and expiry (#1032)"

Note that this also renamed `min_final_cltv_expiry` to the more-correct
`min_final_cltv_expiry_delta`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
fdf9b13bdb Makefile: update bolts fc40879995ebc61cc50dfd729512f17afb15b355.
"Allow nodes to overshoot the MPP `total_msat` when paying (#1031)"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Allow slight overpaying, even with MPP, as spec now recommends.
2023-04-07 21:22:56 +09:30
Rusty Russell
15f8e1e63c Makefile: update bolts to 60cfb5972ad4bec4c49ee0f9e729fb3352fcdc6a.
"BOLT 4: Remove legacy format, make var_onion_optin compulsory."

This also renamed the redundant "tlv_payload" to "payload", so we
replace "tlv_tlv_payload" with "tlv_payload" everyhere!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
06d42694d5 wire: fix extracted patch.
Too much context, it didn't apply if you try to rebuild from source!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
a3ebc1bac4 lightningd: update comments now channel-type is merged.
It's in the main bolt, so remove qualifies from the bolt quote so they
are checked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Vincenzo Palazzo
bf9c4df0de test: add the timeout to the waitpay command
Inside our integration testing we get another timeout,
so this commit adds a timeout to the waitpay command to avoid waiting forever.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-04-07 11:55:31 +02:00
Anmol Agrawal
dae92c5830 Update INSTALL.md 2023-04-07 16:01:47 +09:30
Rusty Russell
a3b81ba17f onchaind: no longer need information about current feerates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
c1bc4d0ead onchaind: remove now-unused direct tx creation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
9496e9fbef onchaind: propose_ignore specifically to ignore if output reaches depth.
We do this for HTLCs which will timeout to them: we watch them in case we
want to fulfill them as a preimage comes in, but once they reach depth we
can forget about them.

We change the message, which causes some more test churn.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
c5b7dbcd98 pytest: clean up wait_for_onchaind_tx interface, remove wait_for_onchaind_broadcast
Using single tuples in Python is ugly, so:
1. Rename wait_for_onchaind_tx to wait_for_onchaind_txs.
2. Make it take tuples explicitly.
3. Make wait_for_onchaind_tx a simpler wrapper/unwrapper.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
0c27acc705 onchaind: use lightningd to sign and broadcast htlc expired txs.
This is when they closed the channel, we can simply make our own tx to
expire the HTLC.  (The other case is where we closed the channel, and
we have a special htlc_timeout tx which we have their signature for).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
5bdd532e70 onchaind: use lightningd to sign and broadcast htlc_timeout transactions.
This breaks tests/test_closing.py::test_onchain_all_dust's accouting
checks.

That test doesn't really test what it claims to test; sure, onchaind
*says* it's going to ignore the output due to high fees, but the tx
still gets mined.

I cannot figure out what the test is supposed to look like, so I
simply disabled the accounting checks :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
868fa8ae81 onchaind: use lightningd to sign and broadcast htlc spending txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
a9dfec0e71 onchaind: use lightningd to sign and broadcast htlc_success transactions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
2f6be4e6bb common: expose low-level htlc_tx function.
We'll want this, as lightningd will want to produce htlc txs based on
what it's told from onchaind, so we need a lower-level accessor.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
3e53c6e359 onchaind: have lightningd create our penalty txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
36dd70e677 onchaind, pytest: disable RBF logic.
We'll reimplement it once lightningd makes all the onchain txs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
80cd6f0afe lightningd: remember depth of closing transaction.
We'll use this later to calculate deadlines for spending txs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
9d5dfa7bdd onchaind: use lightningd for spending our unilateral "to us" output.
This follows the same pattern as the "spend htlc tx" in fact.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
07413c20b9 onchaind: use lightningd to send "delayed_output_to_us" from HTLC txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
3e83bed460 pyln-testing: adapt wait_for_onchaind_broadcast function for when onchaind uses lightningd for broadcast.
We can no longer grab the tx in one line as we did with
wait_for_onchaind_broadcast, we need to track the broadcast from
lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
86e044a9a8 onchaind: infrastructure to offload tx creation to lightningd.
Since we do both our own internal handling and handing it to
lightningd, we add to `proposed_resolution` to handle the lightningd
case.

Note, in particular, that we fix the blockheight calculation: it's out
by one, in that if we see a tx and our CSV lock is 5, we only need to
wait 4 more blocks, not 5.  This will matter as we start using it, and
convert the tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
956e6c4055 lightningd: handle first case of onchaind handing a tx to us to create.
We add code for the case of spending a (timelocked) to-us output of an
HTLC output, so lightningd can do it (rather than onchaind doing all
the work itself).

onchaind still needs to know whether we bothered to create the tx
(fees might have caused it to evaporate, so it should consider it
immediately resolved rather than waiting for it), and what the
witnesses were, and which parts of the witnesses were signatures (as
these parts might change, with RBF or in future, combining other txs).

The inputs (known to onchaind) and the witnesses (told by lightningd)
uniquely identify the spend for the purposes of onchaind.  In
particular, they definitely distinguish HTLC-timeout and HTLC-success
cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
87264540c3 hsmd: add support for lightningd signing onchain txs.
We previously used WIRE_HSMD_SIGN_DELAYED_PAYMENT_TO_US,
WIRE_HSMD_SIGN_REMOTE_HTLC_TO_US, WIRE_HSMD_SIGN_PENALTY_TO_US and
WIRE_HSMD_SIGN_LOCAL_HTLC_TX which allow onchaind to sign txs,
but only for its specific channel.

We now want lightningd to sign these, but it's not bound to a specific
channel.  So let's add variants that don't require that.

We are also now explicit about *what input* to sign.  It's always zero
for now, but future combinations may change that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
45193db7ea lightningd: add initializing state.
Importantly, the code in jsonrpc.c which actually does the io_break:

```
	/* Once the stop_conn conn is drained, we can shut down. */
	if (jcon->ld->stop_conn == conn && jcon->ld->state == LD_STATE_RUNNING) {
		/* Return us to toplevel lightningd.c */
		log_debug(jcon->ld->log, "io_break: %s", __func__);
		io_break(jcon->ld);
```

By having the state not set until later, we avoid running this.  Of course,
we need to avoid calling the main loop when we get there, if we've already
been told to shutdown.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 14:43:48 +02:00
Rusty Russell
30335e1dc3 tests: test for stopping node while it's starting.
In CI we see crashes in this case:

```
lightningd: lightningd/connect_control.c:734: void connectd_activate(struct lightningd *): Assertion `ret == ld->connectd' failed.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 14:43:48 +02:00
Rusty Russell
efeb030eef common: fix build of run-channel_type.c
Broken in master, perhaps due to rebase?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 14:43:48 +02:00
Christian Decker
168bc54700 msggen: Add VersioningCheck
This is a visitor that ensures every new field has at least an `added`
field, and that we don't change the `added` or `deprecated` annotation
after the fact.
2023-04-06 10:16:19 +09:30
Christian Decker
60b12ec096 msggen: Use the inferred optional field
Changelog-Changed: msggen: The generated interfaces `cln-rpc` anc `cln-grpc` can now work with a range of versions rather than having to match the CLN version
2023-04-06 10:16:19 +09:30
Christian Decker
392cacac81 msggen: Add an optional patch
This patch annotates the fields with a new `optional` attribute which
determines whether the field should be considered an inferred optional
due to being added or deprecated.
2023-04-06 10:16:19 +09:30
Christian Decker
5df469cfca msggen: Add patching system, add added and deprecated to Field
The patching system allows us to enrich the raw schema with some
additional information. In this specific case we want to backfill the
`added` and `deprecated` fields for the multiversion support.
2023-04-06 10:16:19 +09:30
Rusty Russell
eee3965d02 db: db_set_intvar/db_get_var should take a const char *.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:50:32 +09:30
Rusty Russell
df9552bcc1 db: make db_exec_prepared_v2 return void.
It calls db_fatal() if it fails anyway, so don't expect anyone to check.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:50:32 +09:30
Rusty Russell
9bcf28afb3 db: catch SQL errors unless we're expecting them.
I couldn't figure out why my new SQL query was returning 0 rows,
and it was because we were ignoring errors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:50:32 +09:30
niftynei
d2176e3385 postgres: add missing 'update_count' to stmt
Reported-By: @rustyrussell
Changelog-Fixed: Plugins: `bookkeeper` onchain fees calculation was incorrect with PostgresQL.
2023-04-06 09:50:32 +09:30
Rusty Russell
13ae1a5168 pyln.testing: remove Throttler.
CI seems to block; Christian suggests the throttler may be to blame somehow?

Since trying to fix it made it worse, let's just remove it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:10:53 +09:30
Rusty Russell
1e24d4a0a0 Makefile: fix check-gen-update to diff *all* files.
This would have caught the missing man page change!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:10:53 +09:30
Rusty Russell
e6db0eafc2 plugins/bcli: use getmempoolinfo to determine minimum possible fee.
Fixes: #4473
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: wallet: we no longer make txs below minrelaytxfee or mempoolminfee.
2023-04-06 09:01:48 +09:30
Rusty Russell
38bc04907b onchaind: two minor tidyups.
Firstly, amount should not be `static`, so use a separate line to
declare those (fee is static, as it's cached across calls).

Secondly, new_tracked_output doesn't take(), it copies.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:01:48 +09:30
Rusty Russell
3a61f3a350 onchaind: helper to read and queue unwanted messages.
We only do this in one place now, but we're going to add another.

Also, make queued messages const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:01:48 +09:30
Rusty Russell
e51f629e34 bitcoind: fix clone_bitcoin_tx() when tx is take().
We need to actually steal it onto requested context in this case!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-06 09:01:48 +09:30