Commit Graph

14134 Commits

Author SHA1 Message Date
Christian Decker
23ce9a947d pytest: Mark test_exclude_adjacent_routehint as skip
It has about a 7% flakyness rate.
2023-10-26 15:51:07 +02:00
Christian Decker
8f4c997248 pytest: Mark test_splice as falky 2023-10-26 15:51:07 +02:00
Christian Decker
1621c6216f rs: Fix the manifests for publication 2023-10-26 15:51:07 +02:00
Christian Decker
b6e2624970 gci: Fix up the crates.io publishing job 2023-10-26 15:51:07 +02:00
Christian Decker
7deeff8404 pytest: Fix a falke in test_gossip_pruning
Fixes cause of https://github.com/ElementsProject/lightning/actions/runs/6518612194/job/17704479411
2023-10-26 15:51:07 +02:00
Christian Decker
5c12d14e11 pytest: Skip test_commit_crash_splice as it is too flaky 2023-10-26 15:51:07 +02:00
Christian Decker
ee8fe11fbd cln-plugin: Make logging optional
Under some circumstances we may want to not log to `lightningd`
directly, but rather configure the logging ourselves. This is useful
for example if we want to use `tracing` and `tracing-subscriber` to
add custom handling, or add opentelemetry span tracing.

Changelog-Changed: cln-plugin: Suppress internal logging handler via `with_logging(false)`
2023-10-26 12:26:01 +02:00
Chris Guida
88ac2dcb7e add test, update schemas, fix warning_custom_fallbacks 2023-10-26 19:11:17 +10:30
Chris Guida
54ef84e891 display paid_outpoint on invoices when present
Changelog-Added: JSON-RPC: `listinvoices` new field `paid_outpoint` if an invoice is paid onchain.
2023-10-26 19:11:17 +10:30
Chris Guida
8c03dd8161 mark invoice as paid when we detect an on-chain payment to a fallback 2023-10-26 19:11:17 +10:30
Chris Guida
8bbf2342f7 lightningd: adapt invoice_try_pay declaration for onchain payment.
i.e. `set` may be NULL, so we need to explicitly hand the amount.
We also add an outpoint.
2023-10-26 19:11:17 +10:30
Chris Guida
e3e0abc73b add autogenerated fallbacks (not custom fallbacks) to invoice_fallbacks table 2023-10-26 19:11:17 +10:30
Chris Guida
cdc0bd47ff track fallbacks for invoices, pt 1 2023-10-26 19:11:17 +10:30
Chris Guida
6b764c0e7c wallet: add invoice_fallbacks table and paid_outpoint to invoices 2023-10-26 19:11:17 +10:30
Chris Guida
fc4b2f864e add invoices-onchain-fallback config option and warnings
Changelog-Added: Config: `invoices-onchain-fallback` to automatically add an onchain p2tr address to invoices, and allow that for payment.
2023-10-26 19:11:17 +10:30
Chris Guida
305ecb1832 bitcoin: expose fmt_bitcoin_txid 2023-10-26 19:11:17 +10:30
Chris Guida
78a3b97463 wallet: add newaddr_inner() helper 2023-10-26 19:11:17 +10:30
niftynei
411bb2884a fallback: json add fallback cleanup
Expand to handle P2TR addresses, use the common script encoding utility
function instead of reimplementing it.
2023-10-26 19:11:17 +10:30
Ken Sedgwick
44798e298c hsmd: implement the hsmd outpoint check
Tihis commit is implementing a 2-phase commit between
the signer the node and the peer.

The main reason for this is that everybody must agree on the lock,
otherwise one of them will want N signatures (on the splice candidates),
and another will produce only 1 signature.

check_outpoint is the "prepare" for the signer, and lock_outpoint is the
"commit". if check_outpoint returns true, lock_outpoint must not fail.

Link: https://github.com/ElementsProject/lightning/issues/6722
Suggested-by: @devrandom
Co-Developed-by: Ken Sedgwick <ken@bonsai.com>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-10-26 16:42:47 +10:30
Ken Sedgwick
485cabb25e splice: Add hsmd_check_outpoint and hsmd_lock_outpoint
In general, a validating signer may be under a different operational
environment than the node, and therefore may have a different
source of on-chain data. The signer may therefore temporarily disagree
on whether a funding or splice transaction is locked (buried).

We would like to ensure agreement between the signer and the
node on how to progress a channel's state.

The following message are added to provide a solution:

- `check_outpoint(outpoint) -> bool` - check if the signer agrees that a funding candidate outpoint is buried
- `lock_outpoint(outpoint)` - change the funding/splice state to locked

Link: https://github.com/ElementsProject/lightning/issues/6722
Suggested-by: @devrandom
Co-Developed-by: Ken Sedgwick <ken@bonsai.com>
Changelog-Added: hsmd protocol: Added hsmd_check_outpoint and hsmd_lock_outpoint
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-10-26 16:42:47 +10:30
Dusty Daemon
dc4e0a400f Splice: Better balance checking
* Regression test added for Issue #6572 (issuecomment-1730808863) w/stuck HTLC
* `check_balance` adjusted to calculate pending HTLCs explicitly
* Test confirmed to fail prior to PR #6713

ChangeLog-Fixed: Issue splicing with pending / stuck HTLCs fixed.
2023-10-26 13:24:04 +10:30
Rusty Russell
13fec8d657 doc: recover.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell
a50b795bc2 lightningd: implement recover command.
Changelog-Added: JSON-RPC: `recover` command to force (unused) lightningd node to restart with `--recover` flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell
7f18ed7743 lightningd: allow --recover to take a 64-char hex string.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `--recover` can take a 32-byte hex string, as well as codex32.
2023-10-26 12:59:55 +10:30
Rusty Russell
3b622f06c3 lightningd: use param_check on all commands which do extra checks.
This makes `check` much more thorough, and useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` now does much more checking on every command (not just basic parameter types).
2023-10-26 12:59:55 +10:30
Rusty Russell
c490be62f0 lightningd: preapproveinvoice doesn't need to string lightning: prefix.
We already do this in param_invstring now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell
95f20a3978 lightningd, pyln-testing: do extra checks to make sure check *cannot* write to db.
Put an assertion inside db.c, and run every command we do (in testing) through
a `check` variant.

I inserted a deliberate bug (made addpsbtoutput call wallet_get_newindex()
before returning when running `check`, and indeed, backtrace as expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell
a01216505f common: add new param_check() which doesn't abort if we're simply checking.
We often want to do more parameter checks after param(), so allow a
new param_check(), with the proviso that the caller needs to also return
command_check_done() after other checks if command_check_only(cmd) is true.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell
25110ff2cc connectd: fix fd leak for --offline.
```
**BROKEN** connectd: dev_report_fds: 5 open but unowned?
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:59:55 +10:30
Rusty Russell
35a47c57e3 lightningd: add direct close outputs to listfunds (mutual close).
We had a complaint that you can't CPFP a mutual close, which you
should be able to do.

Fixes: #6692
Changelog-Fixed: wallet: close change outputs show up immediately in `listfunds` so you can CPFP.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:58:34 +10:30
Rusty Russell
bddff79bc5 pytest: test that we can CPFP a mutual close.
You can't (yet): it doesn't even show in listfunds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-26 12:58:34 +10:30
Peter Neuroth
747725dabd test: Add test for datastoreusage
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Peter Neuroth
d032a17f75 pyln: Add datastoreusage command
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Peter Neuroth
833b7b4baa grpc: Add datastoreusage command
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Peter Neuroth
510984b366 rpc: Add datastoreusage command
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Peter Neuroth
1b5dc2e055 msggen: Add datastoreusage command
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Peter Neuroth
aab948e538 datastore: Add datastoreusage command
datastoreusage returns the total_bytes that are stored under a given
{Key} or from root. {Key} is the entry point from which we begin to
traverse the datastore.

Changelog-Added: JSON-RPC: `datastoreusage`: returns the total bytes that are stored under a given key.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-10-26 12:58:04 +10:30
Dusty Daemon
5aea5fff2f splice[FLAKE]: stale channel announcement fix
Under certain conditions, when splicing a new channel quickly enough, an old channel announcement would emit *after* `mutual_splice_lock` and *before* announcement signature exchange.

Since the original channeld wouldn’t start the announcement timer until signatures were exchagned, this wasn’t an issue before.

Now splicing enables us to go from having announcement sigs to losing them, so we have to be prepared for this case.

Changelog-None
2023-10-26 12:55:14 +10:30
Rusty Russell
985d0db143 lightningd: fix crash when blocks come it fast.
This is from the recent rework, as revealed by CI:

```
2023-10-25T00:57:29.1394358Z lightningd-2 2023-10-25T00:27:22.912Z **BROKEN** lightningd: FATAL SIGNAL 6 (version 4ba8a31-modded)
2023-10-25T00:57:29.1395035Z lightningd-2 2023-10-25T00:27:22.912Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x5574aa3c29dd
2023-10-25T00:57:29.1395670Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x5574aa3c2b6f
2023-10-25T00:57:29.1396230Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f7cce44251f
2023-10-25T00:57:29.1396768Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f7cce4969fc
2023-10-25T00:57:29.1397297Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f7cce442475
2023-10-25T00:57:29.1397945Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f7cce4287f2
2023-10-25T00:57:29.1398616Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:95 (call_error) 0x5574aa56d36a
2023-10-25T00:57:29.1399308Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:169 (check_bounds) 0x5574aa56d565
2023-10-25T00:57:29.1399968Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:179 (to_tal_hdr) 0x5574aa56d5b7
2023-10-25T00:57:29.1400613Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: ccan/ccan/tal/tal.c:514 (tal_free) 0x5574aa56e058
2023-10-25T00:57:29.1401364Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:204 (rebroadcast_txs) 0x5574aa313ccc
2023-10-25T00:57:29.1402128Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:843 (updates_complete) 0x5574aa315a6b
2023-10-25T00:57:29.1402884Z lightningd-2 2023-10-25T00:27:22.913Z **BROKEN** lightningd: backtrace: lightningd/chaintopology.c:1055 (get_new_block) 0x5574aa31659f
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 23:11:37 +02:00
Rusty Russell
0550bb0a7d pytest: fix flake in test_remote_addr_port
```
FAILED tests/test_connection.py::test_remote_addr_port - TimeoutError: Unable to find "[re.compile('Update our node_announcement for discovered address')]" in logs.
```

Because it can happen before the "Already have funding locked in" message:

```
lightningd-2 2023-10-24T22:07:02.018Z DEBUG   gossipd: Update our node_announcement for discovered address: 127.0.0.1:1234
lightningd-2 2023-10-24T22:07:02.019Z DEBUG   lightningd: Plugin chanbackup returned from peer_connected hook call
lightningd-2 2023-10-24T22:07:02.019Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#2: Peer has reconnected, state CHANNELD_NORMAL: connecting subd
lightningd-2 2023-10-24T22:07:02.036Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-channeld-chan#2: pid 63142, msgfd 67
lightningd-2 2023-10-24T22:07:02.037Z DEBUG   035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-chan#2: Already have funding locked in (and ready to announce)
```

Also, wait_for_log() asserts itself, no need to assert on result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 23:11:37 +02:00
Ken Sedgwick
39e17ff83b Restore any missing psbt metadata that resource constrained signers strip
Fixes ([#6764])
Changelog-Fixed: Restore any missing metadata that resource constrained signers stripped
2023-10-25 11:33:07 +02:00
Rusty Russell
9c5e364f16 lightningd: don't re-enter transaction if we have to call plugin_exclusive_loop.
```
Already in transaction from lightningd/plugin.c:727
```

There are two callers, and one didn't disable transactions, so do it in plugin_exclusive_loop.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 10:53:48 +02:00
Rusty Russell
eeee5d6249 lightningd: really fill in our own details when channeld says to make channel_update.
Now we've asserted that channeld would tell lightningd the same thing it
would do anyway, we can simply have channeld say "enable=True|False" and
lightningd fill in the other fields.

This means there's a pile of things channeld doesn't need to know any more!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 07:00:05 +10:30
Rusty Russell
5dbe218bd4 channeld/gossipd/lightningd: reverse polarity of channel_update 'disable_flag'
Rename it to `enable` and invert it everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 07:00:05 +10:30
Rusty Russell
9c04d0b02d lightningd: make sure that channeld_local_channel_update fields are redundant.
channeld used to talk directly to gossipd, so it made sense for it to
tell gossipd directly when it wanted it to make a new channel_update.

When that changed with v0.11, we simply directed the message via
lightningd.

But much of the information is actually told to channeld by lightningd!

So I applied this assertion and ran the test suite, before the next patch makes it redundant.

We got one assertion: test_setchannel_zero deliberately drives the
advertized htlc_max over the real htlc max in test_setchannel_zero for
testing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 07:00:05 +10:30
Rusty Russell
222da7f185 channeld: don't attach channel_update to errors, let lightningd do it.
This is far simpler: lightningd no longer needs to tell channeld when updates change, etc.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 07:00:05 +10:30
Rusty Russell
075f79033a pyln-testing: fix LihgtningNode.openchannel()
`[False]` is considered True in Python, so this flake fix didn't work.

```
>>> if [False]:
...     print('x')
... 
x
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-25 07:00:05 +10:30
daywalker90
eda6a4b44a switch from ToString to Display for ShortChannelId 2023-10-24 11:35:14 +02:00
Rusty Russell
5f664dac77 pytest: fix flake in test_excluded_adjacent_routehint
Wait to be sure l1->l2 is ready.  And use the same pattern for l2->l3.

```
    def test_excluded_adjacent_routehint(node_factory, bitcoind):
        """Test case where we try have a routehint which leads to an adjacent
        node, but the result exceeds our maxfee; we crashed trying to find
        what part of the path was most expensive in that case
    
        """
        l1, l2, l3 = node_factory.line_graph(3)
    
        # We'll be forced to use routehint, since we don't know about l3.
        wait_for(lambda: len(l3.rpc.listchannels(source=l2.info['id'])['channels']) == 1)
        inv = l3.rpc.invoice(10**3, "lbl", "desc", exposeprivatechannels=l2.get_channel_scid(l3))
    
        # This will make it reject the routehint.
        err = r'Fee exceeds our fee budget: 1msat > 0msat, discarding route'
        with pytest.raises(RpcError, match=err):
>           l1.rpc.pay(bolt11=inv['bolt11'], maxfeepercent=0, exemptfee=0)

tests/test_pay.py:3420: 
...
>           raise RpcError(method, payload, resp['error'])
E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt10n1pjntczasp59x0weqkg4u9amd364yaeyw6rmgnmf9qtra6epylcntvt65yalpzspp5x8wgtmjhq33qruk6mmhutyr7w74xxjhct7v9tppel0t9p4rtautsdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqgqqqqqqqqpqqqqqzsqqc9qxpqysgq4euy2qyzl2nufpxv6ahf0s5zry0h5dgrpa5adwu4swrdvjw7qe48cj8kp5fl7k20ex0x3dnk6e8xk5jp82snrdcr6he7eyqd0wrmvlgqwe5nma', 'maxfeepercent': 0, 'exemptfee': 0}, error: {'code': 210, 'message': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 1000msat}]}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 10:02:52 +02:00
Rusty Russell
b0ab350927 pytest: fix flake in test_listforwards_and_listhtlcs
Make sure l1 sees chanell to l4.

```
   def test_listforwards_and_listhtlcs(node_factory, bitcoind):
        """Test listforwards and listhtlcs commands."""
        l1, l2, l3, l4 = node_factory.get_nodes(4, opts=[{}, {}, {}, {}])
    
        l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
        l2.rpc.connect(l3.info['id'], 'localhost', l3.port)
        l2.rpc.connect(l4.info['id'], 'localhost', l4.port)
    
        c12, c12res = l1.fundchannel(l2, 10**5)
        c23, _ = l2.fundchannel(l3, 10**5)
        c24, _ = l2.fundchannel(l4, 10**5)
    
        # Wait until channels are active
        mine_funding_to_announce(bitcoind, [l1, l2, l3, l4])
        l1.wait_channel_active(c23)
    
        # successful payments
        i31 = l3.rpc.invoice(1000, 'i31', 'desc')
        l1.rpc.pay(i31['bolt11'])
    
        i41 = l4.rpc.invoice(2000, 'i41', 'desc')
>       l1.rpc.pay(i41['bolt11'])

tests/test_misc.py:2871: 
...
>           raise RpcError(method, payload, resp['error'])
E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnert20n1pjnd7xlsp5lcwwuxw90nanl7cpfzp4jf7w5tats5aalkksswpps6cqrwpu0l3spp5pzng94efstxjhdc2ya8hhgrch33qavzgv4gahga3y6dxqshezuqsdq8v3jhxccxqyjw5qcqp99qxpqysgqys70jhf8aj4zpwrvfrse54c5kfaqxj98xl4ddxp7jkm75e4p59gy6l2ryvz4fejg5reqkxw44k90rp0l7c8vxut424m4ek743k0lpwcqzg2a60'}, error: {'code': 210, 'message': 'Destination 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199 is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199 is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 2000msat}]}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-24 10:02:52 +02:00