Commit Graph

15338 Commits

Author SHA1 Message Date
ShahanaFarooqui
81e30eab52 docs: Updating release checklist with Hotfix release instructions
Changelog-None.
2024-10-03 22:24:56 -07:00
Se7enZ
fea30491c3 doc: Include documentation for test targeting via PYTEST_TESTS.
Also update the table of optional environment variables to set when
running the integration tests.
2024-10-03 21:51:26 -07:00
Se7enZ
537d613035 pytest: Support targeting specific tests using make pytest.
Select tests by setting the `PYTEST_TESTS` environment variable.

Changelog-None
2024-10-03 21:51:26 -07:00
Se7enZ
cbe94bcb42 pyln-testing: Fix file descriptor leak in bitcoind fixture. ([#7130])
Changelog-Fixed: pyln-testing: Fix file descriptor leak in bitcoind fixture. ([#7130])
2024-10-03 19:04:35 -07:00
Aditya Sharma
27b4a03419 tests: Add test_getemergencyrecoverdata to test the RPC getemergencyrecoverdata. 2024-10-03 18:59:10 -07:00
Aditya Sharma
2be7433ae6 doc: Add documentation for the RPC getemergencyrecoverdata. 2024-10-03 18:59:10 -07:00
Aditya Sharma
4c775f16aa plugins/chanbackup: Add RPC to fetch data from emergency.recover file. 2024-10-03 18:59:10 -07:00
Rusty Russell
b8acd3b37c askrene: more code tweaks on feedback from Lagrang3.
1. describe_disabled should point out if node itself is disabled.
2. Hoist constraint check for neater if branching.
3. Use amount_msat_max/min for greater clarity.
4. Simply disable channels, don't zero htlc_min/max when node disabled.

I also fixed the diagnostic of htlc_max correctly, which removes a FIXME.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
e3c9bc6d3a askrene: trivial changes to avoid -O3 compiler warnings.
The code is a bit too complex for gcc to track it:

```
In file included from ccan/ccan/tal/str/str.h:7,
                 from plugins/askrene/askrene.c:11:
plugins/askrene/askrene.c: In function ‘do_getroutes’:
ccan/ccan/tal/tal.h:324:23: error: ‘routes’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  324 | #define tal_count(p) (tal_bytelen(p) / sizeof(*p))
      |                       ^~~~~~~~~~~
plugins/askrene/askrene.c:476:24: note: ‘routes’ was declared here
  476 |         struct route **routes;
      |                        ^~~~~~
plugins/askrene/askrene.c:475:29: error: ‘amounts’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  475 |         struct amount_msat *amounts;
      |                             ^~~~~~~
plugins/askrene/askrene.c:488:69: error: ‘probability’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  488 |         json_add_u64(response, "probability_ppm", (u64)(probability * 1000000));
      |                                                        ~~~~~~~~~~~~~^~~~~~~~~~
cc plugins/askrene/dijkstra.c
cc1: all warnings being treated as errors
```

On my local machine, it also warns in param_dev_channel, so I fixed that too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
630ec6a566 askrene: give better feedback when we can't find a suitable route.
This turns out to be critical for users: also stops them from
bothering us when their node is offline or has insufficient capacity!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
e54c0f8ded askrene: don't replace constraints, simply accumulate.
Lagrang3 points out it's less useful (when we time them out), and probably
a premature optimization anyway.

Suggested-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
c307b77d2f askrene: split askrene-create-channel into create-channel and update-channel.
This allows for explicit partial updates to channels (e.g. just change
fees, or just disable) without haveing to set the other fields.

This generalizes askrene-disable-channel, which is removed.

We also take the chance to use the proper BOLT 7 terms in the API:

- htlc_minimum_msat
- htlc_maximum_msat
- cltv_expiry_delta
- fee_base_msat
- fee_proportional_millionths

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
5052f0763f gossmap: keep capacity for locally-generated channels as well.
It was weird not to have a capacity associated with localmods channels, and
fixing it has some very nice side effects.

Now the gossmap_chan_get_capacity() call never fails (we prevented reading
of channels from gossmap in the partially-written case already), so we
make it return the capacity.  We do this in msat, because that's what
all the callers want.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
a60063e763 common/gossmods_listpeerchannels: include channel capacity in callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
a65e325b13 gossmap: implement partial updates.
This is actually what we want in several places: to only override one or
two fields in a channel_update.

We add a gossmap_local_setchan() with a similar API to the old
gossmap_local_updatechan(), for the case where we want to set every
field.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
bb3663c4a0 askrene: ignore disabled channels for min-cost-flow.
We also set htlc_max to 0 when disabling, so the tests worked, but
this is correct.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
154ad585ec pytest: enhance test_getroutes_auto_sourcefree with same case *without* auto.sourcefree.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
3253623785 gossmods_from_listpeerchannels: use correct type for cltv_delta.
Doesn't matter now, but will with the next change where we want to
pass a pointer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
fcc0d2bad8 askrene: remove unused parameter in layer_add_localmods.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
321ec0875f askrene: rework constraints to exist in pairs.
This is a bit more efficient, but moreover the JSONRPC API is more
logical this way.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
3c5c22b17a askrene: change inform interface, take into account reserve.
Lagrang3 points out that if we hit a maximum, we should take into account
the reserve.  This is true, but it's hard for the caller to do, so change
the API to be slightly higher level.

Tell "inform" what happened, and it adjust the constraints appropriately.
This makes the least assumptions possible (a reserve does *not* mean that
the capacity was actually used at that time).

We also add a mode to say "this succeeded": for now this does nothing,
but it could reduce both min/max capacities, and add capacity in the
other direction.  This is useful for future payments, but not as useful
for the current one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
d50838b60f askrene: implement listreservations
And actually write tests!

Suggested-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
0398d2ff73 askrene: remember individual reservations, for better debugging.
Suggested-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
10dc40a895 askrene: clean up reserve array handling.
I got confused, as we had a struct containing two arrays.  Simply expose the
reserve_hop struct and use arrays directly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
2df53c32c2 askrene: make route_query contain pointer to the command.
This is important for errors and feedback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
b88f4cb854 askrene: askrene-create-layer and askrene-remove-layer.
It's generally better to be explicit with these things: currently typos
would be ignored.  But it's also much easier to clean up entire layers
as we use them for temporary (per-payment) effects.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
29cc227a53 askrene: use short_channel_id_dir in API.
It's generally much more convenient, and it's already present in
other APIs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Lagrang3
33404b03a0 add askrene-disable-channel
Changelog-EXPERIMENTAL: askrene: add askrene-disable-channel RPC
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
2024-10-04 11:27:53 +09:30
Rusty Russell
f46219b505 common: round out the short_channel_id_dir JSON routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
d8ca9baa3b askrene: add support for disabled channels in layers.
Based-on-the-patch-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
bc1aabb014 gossmap: don't crash on localmods on non-existant channels.
We allow adding them, but crash when we remove the localmods.  Yet
this could theoretically happen if a channel we modified was removed
from the gossmap, anyway.

Reported-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
0a23c63d37 askrene: optimize, by calling tal_count less.
I like the clarity, but this is a hot path.  Fortunately these arrays
have very well defined lengths.

Before: 5.81 seconds
After: 1.06 seconds

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
9f0c0e1cca askrene: use a simple array as our queue.
We only ever visit each node once, so we can just use an array.  This
avoids calling tal() all the time, which is *especially* slow when we're
memory tracking.

I had an old canned gossmap which I benchmarked for these (and in
particular one node was unreachable, and that was slow):

Before: 17.27 seconds
After: 5.80 seconds

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
79fa3d3d7a tests/data: add compressd canned gossip from mainnet.
We can run a lot of tests on this.

Look at me, I'm number 3301!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
d1a6649c73 gossmap-compress: warn about updates which we don't include in map.
There are some, they're unusable high-fee.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
01b65d1a15 gossmap: print out nodeids and aliases, so you can map them back after compression.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
e55a4850cc devtools/gossmap-compress: fix channels.
We can only use the loop to iterate once.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
d143c22976 gossmap-compress: sort nodes into most to least as intended.
Interestingly, compressed sizes don't change, but uncompressed does:

```
$ file /tmp/gs.out-wrongorder
/tmp/gs.out-wrongorder: gzip compressed data, max compression, from Unix, original size modulo 2^32 1594822
$ ls -l /tmp/gs.out-wrongorder 
-rw-rw-r-- 1 rusty rusty 494337 Sep 20 13:34 /tmp/gs.out-wrongorder
$ file /tmp/gs.out
/tmp/gs.out: gzip compressed data, max compression, from Unix, original size modulo 2^32 1498634
$ ls -l /tmp/gs.out
-rw-rw-r-- 1 rusty rusty 494353 Sep 20 13:33 /tmp/gs.out
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
ShahanaFarooqui
7ac32ebbf8 Update pyln-client for clnrest 2024-10-03 11:20:52 -07:00
Rusty Russell
00e0b99d3a hsmtool: provide nodeid from hsm secret.
This allows tools to validate that it is accessing the correct hsm_secret for this node!

This is extremely important for backups: if they are using VLS, they need to back *that*
up instead, for example.

Changelog-Added: `hsmtool`: `getnodeid` command derives the node id from the hsm_secret, to verify it's the correct secret.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-25 12:07:40 -07:00
ShahanaFarooqui
e36b68ab48 gpg: New public key
Changelog-None.
2024-09-24 21:00:25 -07:00
Rusty Russell
21fcd4594b cln-rpc/src/model.rs: fix difference between master and my machine.
Not sure why this differs from master, so trying pushing my version
to master?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-24 10:24:51 +09:00
niftynei
5ae7212e37 anchors: dont include utxos which are csv locked
When building a tx to spend *now* we shouldn't include CSV locked utxos
as we want them to be immediately spendable.
2024-09-23 11:39:33 -07:00
Peter Neuroth
0a6870f13d rust-libs: Bump versions
We bump the version to minor version 0.2 to allow ourselves to bump on
patch level when we need to fix a bug or update a dependency.

Changelog-Changed: Plugins: `cln-grpc` Upgrade tonic version and
introduce new versioning scheme.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2024-09-22 00:12:39 -07:00
Peter Neuroth
a6366c2814 rust-libs: Upgrade tonic to v0.12
Tonic had some breaking changes since 0.8. We want to be closer to newer
versions for downstream consumers to be able to benefit from changes in tonic.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2024-09-22 00:12:39 -07:00
Jesse de Wit
2798b4b7be cln-grpc: add anchors/even to primitives
The `anchors/even` channel type was missing from the grpc bindings.
It is now the default channel type, so `fundchannel` over grpc fails
with:

```
Failed to deserialize response : unknown variant `anchors/even`,
expected one of `static_remotekey/even`, `anchor_outputs/even`,
`anchors_zero_fee_htlc_tx/even`, `scid_alias/even`, `zeroconf/even`
```

Changelog-Changed: Channel type `anchors/even` was added
to the grpc bindings.
2024-09-20 17:01:04 -07:00
Rusty Russell
aba67a757c lightningd: rename state_change_entry to channel_state_change, and use pointers.
This name is clearer than the old one.

And since the struct contains a string, it's more natural for the
struct to be the tal parent of the string so it's a real object.  This means
we need an array of pointers, so each struct can be its own tal object.

wallet_state_change_get is hoisted higher in the code and made static.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30
Rusty Russell
0440700132 lightningd: move the state changes into struct channel.
And instead of loading them in listpeerchannels, use them.  This means
listpeerchannels no longer touches the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSONRPC: `listpeerchannels` (and thus, pay) sped up on very large nodes.
2024-09-21 06:20:18 +09:30
Rusty Russell
eb979980a9 db: clean up channel_stats handling.
Indirection via a string and an enum is just adding confusion here.

And move the `struct channel_stats` into channel.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30
Rusty Russell
3aace10828 listpeerchannels: use struct channel stats, don't fetch from db.
This avoids a db lookup on every iteration of listpeerchannels, which
can be slow on large nodes (Postgres, I assume).

We can now simply add the fields we want to channel load, and remove
wallet_channel_stats_load entirely.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30