Commit Graph

12812 Commits

Author SHA1 Message Date
Rusty Russell
3a3370f4c1 feerates: add floor field for the current minimum feerate bitcoind will accept
Changelog-Added: JSON-RPC: `feerates`: added `floor` field for current minimum feerate bitcoind will accept
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
812a5a14c0 plugins/bcli: use the new feerate levels, and the floor.
Fixes: #4473
Changelog-Deprecated: Plugins: `estimatefees` returning feerates by name (e.g. "opening"); use `fee_floor` and `feerates`.
Changelog-Fixed: Plugins: `bcli` now tells us the minimal possible feerate, such as with mempool congestion, rather than assuming 1 sat/vbyte.
2023-04-10 07:31:12 +09:30
Rusty Russell
9e2d4240b1 lightningd: handle bcli plugins returning fee_floor and feerates parameters.
Changelog-Added: Plugins: `estimatefees` can return explicit `fee_floor` and `feerates` by block number.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
c46473e615 lightningd: allow "NNblocks" and "minimum" as feerates.
And consolidate descriptions into lightning-feerates().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` now allow "minimum" and NN"blocks" as `feerate` (`feerange` for `close`).
2023-04-10 07:31:12 +09:30
Rusty Russell
64b1ddd761 lightningd: clean up feerate handling, deprecate old terms.
Drop try_get_feerate() in favor of explicit feerate_for_deadline() and
smoothed_feerate_for_deadline().

This shows us everywhere we deal with old-style feerates by names.

`delayed_to_us` and `htlc_resolution` will be moving to dynamic fees,
so deprecate those.

Note that "penalty" is still used for generating penalty txs for
watchtowers, and "unilateral_close" still used until we get zero-fee
anchors.

Changelog-Added: JSON-RPC: `feerates` `estimates` array shows fee estimates by blockcount from underlying plugin (usually *bcli*).
Changelog-Changed: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) value *slow* is now 100 block-estimate, not half of 100-block estimate.
Changelog-Deprecated: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) expressed as, "delayed_to_us", "htlc_resolution", "max_acceptable" or "min_acceptable".  Use explicit block counts or *slow*/*normal*/*urgent*/*minimum*.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
cdb85d5618 lightningd: handle fees as blockcount + range.
Rather than have specific-purpose levels, have an array of
[blockcount, feerate], and rebuild the specific-purpose levels
for now on top.

We also keep a *separate* smoothed feerate, so you can ask for that
explicitly.

Since all the plugins used the same formula to derive the different
named fee levels, we apply the reverse to return to the underlying
estimates: updating the interface comes next.

This is ugly for now, but various specific-purpose levels will be
going away, as we shift to deadline-driven fees.

This temporarily breaks the floor calculation, so that test is
disabled.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
faae44713b lightningd: clarify uses of dynamic (mempool) feerate floor, and static.
We have the FEERATE_FLOOR constant if you don't care, but usually you want
to use the current bitcoind lower limit, so call get_feerate_floor()
(which is currently the same, but coming!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
7aa8c76002 pytest: test parsefeerate explicitly.
Since we're messing with feerates, it's good to test this directly upfront.

Also, fix documentation!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
a2ca34ccf5 common: add tal_arr_insert helper to utils.h
We have tal_arr_remove and tal_arr_append already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
6799cd5d0b plugins/bcli: move commit-fee (dev-max-fee-multiplier) and into core.
Turns out the two bcli replacements I checked (`sauron` and
`trustedcoin`) don't even implement this, and the multiplier makes
more sense in lightningd, especially as we move to bcli just providing
raw feerate estimates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Rusty Russell
b53cc69cfd msggen: fix incorrect assertion.
Adding a new field with `added` fails:

```
AssertionError: Field Feerates.perkb.estimates[] does not have an `added` annotation
```

Looks like this assertion is wrong: we should get an added from the field itself or
from the .msggen.json file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-10 07:31:12 +09:30
Matt Morehouse
ca80dee514 doc: add section about improving fuzzing corpora
We want to encourage contributions to the seed corpora that improve
coverage.
2023-04-09 12:41:27 +09:30
Matt Morehouse
6e11a2e416 doc: document "make check-fuzz" 2023-04-09 12:41:27 +09:30
Matt Morehouse
14afa6efe7 make: add check-fuzz target
The target builds and runs each fuzz target on its seed corpus.
2023-04-09 12:41:27 +09:30
Matt Morehouse
fa988d2942 fuzz: add check-fuzz.sh
The script runs each fuzz target on its seed corpus and prints any
failures.
2023-04-09 12:41:27 +09:30
Matt Morehouse
a1a1373090 fuzz: improve corpus merging
The following arguments were copied from Bitcoin Core's corpus merging
script https://github.com/bitcoin/bitcoin/blob/master/test/fuzz/test_runner.py:

-shuffle=0
-prefer_small=1
-use_value_profile=1
2023-04-09 12:41:27 +09:30
Matt Morehouse
9384692e2a fuzz: add initial seed corpora
These corpora were generated with default libFuzzer flags with 30+ hours
of CPU time, and then minimized with:
  ./fuzz-TARGET -merge=1 -shuffle=0 -prefer_small=1 -use_value_profile=1 corpora/fuzz-TARGET UNMINIMIZED_CORPUS
2023-04-09 12:41:27 +09:30
Alex Myers
2f050621b0 pytest: add blackbox tests for reckless
A canned lightningd/plugins is used to test against. This allows faster
and more deterministic outcomes.

Changelog-None
2023-04-09 12:41:11 +09:30
Alex Myers
cf203369bc reckless: use environment variable redirects
This will be used during CI testing in the following commit.
2023-04-09 12:41:11 +09:30
Alex Myers
55cddcd350 reckless: add support for additional networks
This should have been added earlier as @cdecker suggested, but is needed
to enable CI testing.

Changelog-Changed: Reckless - added support for networks beyond bitocoin and regtest
2023-04-09 12:41:11 +09:30
Alex Myers
e61401aab9 reckless: don't crash on subprocess calls
They prefer Paths to be explicitly cast as strings
2023-04-09 12:41:11 +09:30
Rusty Russell
88905e8372 tests: split fetchinvoice recurrence tests into separate test.
This is for VLS, which doesn't implement signing for the non-standard
recurrence fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:23:14 +09:30
Rusty Russell
d4ffc75691 Makefile: update to latest BOLT text.
In particular:
	- Bolt 4: add route blinding construction
	- Bolt 4: add blinded payments

And this means it's not experimental, so we can turn it on
by default!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: blinded payments are now supported by default (not just with `--experimental-onion-messages`)
2023-04-07 21:22:56 +09:30
Rusty Russell
458a85042b Makefile: update to BOLT 20066dc2aba906f37f3be5a810ae67040f265377
"BOLT 03: fix static-remote same amt and pre-image test vector"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
f26b1166b7 Makefile: update bolts a0bbe47b0278b4f152dbaa4f5fab2562413a217c
"BOLT 04: remove associated data from test vector"

(We actually use merge point).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
dfa6c0ca52 Makefile: bolt version b38156b9510c0562cf50f8758a64602cc0315c19
"Allow nodes to overshoot final htlc amount and expiry (#1032)"

Note that this also renamed `min_final_cltv_expiry` to the more-correct
`min_final_cltv_expiry_delta`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
fdf9b13bdb Makefile: update bolts fc40879995ebc61cc50dfd729512f17afb15b355.
"Allow nodes to overshoot the MPP `total_msat` when paying (#1031)"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: Allow slight overpaying, even with MPP, as spec now recommends.
2023-04-07 21:22:56 +09:30
Rusty Russell
15f8e1e63c Makefile: update bolts to 60cfb5972ad4bec4c49ee0f9e729fb3352fcdc6a.
"BOLT 4: Remove legacy format, make var_onion_optin compulsory."

This also renamed the redundant "tlv_payload" to "payload", so we
replace "tlv_tlv_payload" with "tlv_payload" everyhere!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
06d42694d5 wire: fix extracted patch.
Too much context, it didn't apply if you try to rebuild from source!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Rusty Russell
a3ebc1bac4 lightningd: update comments now channel-type is merged.
It's in the main bolt, so remove qualifies from the bolt quote so they
are checked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 21:22:56 +09:30
Vincenzo Palazzo
bf9c4df0de test: add the timeout to the waitpay command
Inside our integration testing we get another timeout,
so this commit adds a timeout to the waitpay command to avoid waiting forever.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-04-07 11:55:31 +02:00
Anmol Agrawal
dae92c5830 Update INSTALL.md 2023-04-07 16:01:47 +09:30
Rusty Russell
a3b81ba17f onchaind: no longer need information about current feerates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
c1bc4d0ead onchaind: remove now-unused direct tx creation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
9496e9fbef onchaind: propose_ignore specifically to ignore if output reaches depth.
We do this for HTLCs which will timeout to them: we watch them in case we
want to fulfill them as a preimage comes in, but once they reach depth we
can forget about them.

We change the message, which causes some more test churn.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
c5b7dbcd98 pytest: clean up wait_for_onchaind_tx interface, remove wait_for_onchaind_broadcast
Using single tuples in Python is ugly, so:
1. Rename wait_for_onchaind_tx to wait_for_onchaind_txs.
2. Make it take tuples explicitly.
3. Make wait_for_onchaind_tx a simpler wrapper/unwrapper.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
0c27acc705 onchaind: use lightningd to sign and broadcast htlc expired txs.
This is when they closed the channel, we can simply make our own tx to
expire the HTLC.  (The other case is where we closed the channel, and
we have a special htlc_timeout tx which we have their signature for).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
5bdd532e70 onchaind: use lightningd to sign and broadcast htlc_timeout transactions.
This breaks tests/test_closing.py::test_onchain_all_dust's accouting
checks.

That test doesn't really test what it claims to test; sure, onchaind
*says* it's going to ignore the output due to high fees, but the tx
still gets mined.

I cannot figure out what the test is supposed to look like, so I
simply disabled the accounting checks :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
868fa8ae81 onchaind: use lightningd to sign and broadcast htlc spending txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
a9dfec0e71 onchaind: use lightningd to sign and broadcast htlc_success transactions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
2f6be4e6bb common: expose low-level htlc_tx function.
We'll want this, as lightningd will want to produce htlc txs based on
what it's told from onchaind, so we need a lower-level accessor.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
3e53c6e359 onchaind: have lightningd create our penalty txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
36dd70e677 onchaind, pytest: disable RBF logic.
We'll reimplement it once lightningd makes all the onchain txs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
80cd6f0afe lightningd: remember depth of closing transaction.
We'll use this later to calculate deadlines for spending txs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
9d5dfa7bdd onchaind: use lightningd for spending our unilateral "to us" output.
This follows the same pattern as the "spend htlc tx" in fact.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
07413c20b9 onchaind: use lightningd to send "delayed_output_to_us" from HTLC txs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
3e83bed460 pyln-testing: adapt wait_for_onchaind_broadcast function for when onchaind uses lightningd for broadcast.
We can no longer grab the tx in one line as we did with
wait_for_onchaind_broadcast, we need to track the broadcast from
lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
86e044a9a8 onchaind: infrastructure to offload tx creation to lightningd.
Since we do both our own internal handling and handing it to
lightningd, we add to `proposed_resolution` to handle the lightningd
case.

Note, in particular, that we fix the blockheight calculation: it's out
by one, in that if we see a tx and our CSV lock is 5, we only need to
wait 4 more blocks, not 5.  This will matter as we start using it, and
convert the tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
956e6c4055 lightningd: handle first case of onchaind handing a tx to us to create.
We add code for the case of spending a (timelocked) to-us output of an
HTLC output, so lightningd can do it (rather than onchaind doing all
the work itself).

onchaind still needs to know whether we bothered to create the tx
(fees might have caused it to evaporate, so it should consider it
immediately resolved rather than waiting for it), and what the
witnesses were, and which parts of the witnesses were signatures (as
these parts might change, with RBF or in future, combining other txs).

The inputs (known to onchaind) and the witnesses (told by lightningd)
uniquely identify the spend for the purposes of onchaind.  In
particular, they definitely distinguish HTLC-timeout and HTLC-success
cases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30
Rusty Russell
87264540c3 hsmd: add support for lightningd signing onchain txs.
We previously used WIRE_HSMD_SIGN_DELAYED_PAYMENT_TO_US,
WIRE_HSMD_SIGN_REMOTE_HTLC_TO_US, WIRE_HSMD_SIGN_PENALTY_TO_US and
WIRE_HSMD_SIGN_LOCAL_HTLC_TX which allow onchaind to sign txs,
but only for its specific channel.

We now want lightningd to sign these, but it's not bound to a specific
channel.  So let's add variants that don't require that.

We are also now explicit about *what input* to sign.  It's always zero
for now, but future combinations may change that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-07 11:49:09 +09:30