Commit Graph

15508 Commits

Author SHA1 Message Date
Rusty Russell
bc1aabb014 gossmap: don't crash on localmods on non-existant channels.
We allow adding them, but crash when we remove the localmods.  Yet
this could theoretically happen if a channel we modified was removed
from the gossmap, anyway.

Reported-by: Lagrang3 <lagrang3@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 11:27:53 +09:30
Rusty Russell
0a23c63d37 askrene: optimize, by calling tal_count less.
I like the clarity, but this is a hot path.  Fortunately these arrays
have very well defined lengths.

Before: 5.81 seconds
After: 1.06 seconds

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
9f0c0e1cca askrene: use a simple array as our queue.
We only ever visit each node once, so we can just use an array.  This
avoids calling tal() all the time, which is *especially* slow when we're
memory tracking.

I had an old canned gossmap which I benchmarked for these (and in
particular one node was unreachable, and that was slow):

Before: 17.27 seconds
After: 5.80 seconds

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
79fa3d3d7a tests/data: add compressd canned gossip from mainnet.
We can run a lot of tests on this.

Look at me, I'm number 3301!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
d1a6649c73 gossmap-compress: warn about updates which we don't include in map.
There are some, they're unusable high-fee.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
01b65d1a15 gossmap: print out nodeids and aliases, so you can map them back after compression.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
e55a4850cc devtools/gossmap-compress: fix channels.
We can only use the loop to iterate once.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
Rusty Russell
d143c22976 gossmap-compress: sort nodes into most to least as intended.
Interestingly, compressed sizes don't change, but uncompressed does:

```
$ file /tmp/gs.out-wrongorder
/tmp/gs.out-wrongorder: gzip compressed data, max compression, from Unix, original size modulo 2^32 1594822
$ ls -l /tmp/gs.out-wrongorder 
-rw-rw-r-- 1 rusty rusty 494337 Sep 20 13:34 /tmp/gs.out-wrongorder
$ file /tmp/gs.out
/tmp/gs.out: gzip compressed data, max compression, from Unix, original size modulo 2^32 1498634
$ ls -l /tmp/gs.out
-rw-rw-r-- 1 rusty rusty 494353 Sep 20 13:33 /tmp/gs.out
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-10-04 08:47:53 +09:30
ShahanaFarooqui
7ac32ebbf8 Update pyln-client for clnrest 2024-10-03 11:20:52 -07:00
Rusty Russell
00e0b99d3a hsmtool: provide nodeid from hsm secret.
This allows tools to validate that it is accessing the correct hsm_secret for this node!

This is extremely important for backups: if they are using VLS, they need to back *that*
up instead, for example.

Changelog-Added: `hsmtool`: `getnodeid` command derives the node id from the hsm_secret, to verify it's the correct secret.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-25 12:07:40 -07:00
ShahanaFarooqui
e36b68ab48 gpg: New public key
Changelog-None.
2024-09-24 21:00:25 -07:00
Rusty Russell
21fcd4594b cln-rpc/src/model.rs: fix difference between master and my machine.
Not sure why this differs from master, so trying pushing my version
to master?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-24 10:24:51 +09:00
niftynei
5ae7212e37 anchors: dont include utxos which are csv locked
When building a tx to spend *now* we shouldn't include CSV locked utxos
as we want them to be immediately spendable.
2024-09-23 11:39:33 -07:00
Peter Neuroth
0a6870f13d rust-libs: Bump versions
We bump the version to minor version 0.2 to allow ourselves to bump on
patch level when we need to fix a bug or update a dependency.

Changelog-Changed: Plugins: `cln-grpc` Upgrade tonic version and
introduce new versioning scheme.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2024-09-22 00:12:39 -07:00
Peter Neuroth
a6366c2814 rust-libs: Upgrade tonic to v0.12
Tonic had some breaking changes since 0.8. We want to be closer to newer
versions for downstream consumers to be able to benefit from changes in tonic.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2024-09-22 00:12:39 -07:00
Jesse de Wit
2798b4b7be cln-grpc: add anchors/even to primitives
The `anchors/even` channel type was missing from the grpc bindings.
It is now the default channel type, so `fundchannel` over grpc fails
with:

```
Failed to deserialize response : unknown variant `anchors/even`,
expected one of `static_remotekey/even`, `anchor_outputs/even`,
`anchors_zero_fee_htlc_tx/even`, `scid_alias/even`, `zeroconf/even`
```

Changelog-Changed: Channel type `anchors/even` was added
to the grpc bindings.
2024-09-20 17:01:04 -07:00
Rusty Russell
aba67a757c lightningd: rename state_change_entry to channel_state_change, and use pointers.
This name is clearer than the old one.

And since the struct contains a string, it's more natural for the
struct to be the tal parent of the string so it's a real object.  This means
we need an array of pointers, so each struct can be its own tal object.

wallet_state_change_get is hoisted higher in the code and made static.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30
Rusty Russell
0440700132 lightningd: move the state changes into struct channel.
And instead of loading them in listpeerchannels, use them.  This means
listpeerchannels no longer touches the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSONRPC: `listpeerchannels` (and thus, pay) sped up on very large nodes.
2024-09-21 06:20:18 +09:30
Rusty Russell
eb979980a9 db: clean up channel_stats handling.
Indirection via a string and an enum is just adding confusion here.

And move the `struct channel_stats` into channel.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30
Rusty Russell
3aace10828 listpeerchannels: use struct channel stats, don't fetch from db.
This avoids a db lookup on every iteration of listpeerchannels, which
can be slow on large nodes (Postgres, I assume).

We can now simply add the fields we want to channel load, and remove
wallet_channel_stats_load entirely.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30
Rusty Russell
762d624810 lightningd: move stats into struct channel.
Like other fields, mirror them into the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-21 06:20:18 +09:30
Lagrang3
0aa52b7fdd askrene: remove unused function
Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
2024-09-19 12:16:53 +09:30
Rusty Russell
9b60f6cc6d askrene: re-check min_htlc violations after correcting for MCF rounding.
Thanks to @Lagrang3 for spotting this!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
8c551ae48a askrene: move flow refining code to its own file.
askrene.c was getting quite long, and this is self-contained.

The only code change is a convenience accessor for the per-htlc-cost
hash table.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
9d966b46a7 askrene: take into account the reduction in "spendable" with additional HTLCs.
"spendable" is for a single HTLC: if we own the channel, this amount
decreases with every HTLC, as we have to pay fees.  We have access to this since
we call listpeerchannels anyway, so we can calculate the additional costs and
use it in the refining phase.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
3e96955c72 pytest: test askrene treats htlc_maximum_msat and htlc_minimum_msat restrictions properly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
db29a2d6b5 askrene: don't have get_flow_paths() handle htlc_max, htlc_min and extra millisats.
We don't actually hit the htlc_max cases, since the flow code already
constrains us to that.

And handling htlc_min is better done in the caller, where diagnostics
are better (basically, we should eliminate them, and if that means no
route, give a clear error message).

And the refinement step can handle any extra millisats from rounding.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
f0331cd82e askrene: add a "refining" step to add fees and handle corner cases.
This is the root cause of the problem worked around in 50949b7b9c
"askrene: hack in some padding so we don't overflow capacities."

When adding fees to flows, we didn't recheck the boundary conditions: in
renepay this is done by routebuilder.

Fortunately, we can use our "reservations" infrastructure to temporarily
use capacity as we process flows, so we handle the cases where they are
not independent correclty.

My assumption is that the resulting errors are small, so we divide
them between the remaining flows based on highest-to-least
probability.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
5883aa85ca askrene: rename struct flow amount to delivers.
This is clearer: it's the final amount, not the amount we send!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
829954ac71 askrene: remove struct flow probability member.
Simply calculate it when we need it, which means we don't have to keep it
up-to-date as we tweak the flow.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
923e37a545 askrene: remove unused flow routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
17191259ea patch libplugin-batch.patch 2024-09-19 12:16:53 +09:30
Rusty Russell
1362448352 common/bolt12: do more required checks in invoice_decode.
Rather than making the callers do this, make the invoice decoder perform
the various sanity checks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
1b8a64f5e6 askrene: apply "auto.sourcefree" to channels created in layers, too.
We had a workaround for channels added by "auto.local", but instead we
should make it work properly.

I didn't do this before because we can't manipulate the localmods while
they're applied, but it's simple to do it in two stages.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
06957dc832 pytest: separate out routine which checks only some fields of getroutes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
e90be8d957 pyln-testing: add gossip_store_file arg to get_node()
This makes it much easier to use generated gossip_store files.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
5ea049ea12 pytest: duplicate test_live_spendable as a canned gossmap test.
Not quite the same, as it doesn't have the "auto.local" layer, but it exhibits
the same problem if we revert the fix for test_live_spendable.

And it's much faster!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
b11642538e pytest: ensure there are no duplicate paths in test_live_spendable, and check total.
There aren't, but make sure.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
679f46f733 common/amount: rename amount_sat_zero/amount_msat_zerp -> amount_sat_is_zero/amount_msat_is_zero.
I used `amount_msat_eq(x, AMOUNT_MSAT(0))` because I forgot this
function existed.  I probably missed it because the name is surprising,
so add "is" in there to make it clear it's a boolean function.

You'll note almost all the places which did use it are Eduardo's and
Lisa's code, so maybe it's just me.

Fix up a few places which I could use it, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
f05f871c92 common/amount: add amount_msat_accumulate()
Saves some typing, and is clearer than checking if both args really
are the same!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
6150d1b3fa patch general-node-id-assign.patch 2024-09-19 12:16:53 +09:30
Rusty Russell
379a4ee16a common/amount: add routine to calculate fees backwards.
If I put in X, how much can I get out after fees are subtracted?

This was inspired by Eduardo's channel_maximum_forward in renepay, which
is basically the same thing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
Rusty Russell
db74ca7cbe lightningd: remove incorrect old-payment-htlc-delete logic.
This was incorrect once we stopped removing old payments on failure,
which was the reason we had to remove the HTLCs.

It also removed by partid, which is wrong since it should have done
old_payment->partid and old_payment->groupid!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-19 12:16:53 +09:30
ShahanaFarooqui
75c616c7b3 Makefile: Next version updated to 24.11
Changelog-None.
2024-09-18 16:59:27 +09:30
ShahanaFarooqui
5ba1824fd5 doc: Release checklist update 2024-09-18 16:59:27 +09:30
ShahanaFarooqui
8558993f77 doc: Pull request template update 2024-09-18 16:59:27 +09:30
ShahanaFarooqui
3d3e86e22c test: CI error fixes
- Removes CI value error for Broken logs
- Fixes CI errors due to deprecated listconfigs 'important-plugins'
- Removed listchannels deprecated local test

Changelog-None.
2024-09-18 16:59:27 +09:30
Michael Schmoock
02176f7004 pytests: extend the offline mode testcase 2024-09-17 19:53:25 +02:00
ShahanaFarooqui
7725ca1dbc meta: Add changelog for hotfix release v24.08.1
Changelog-None.
2024-09-17 10:24:24 -07:00
Rusty Russell
f0cf088489 lightningd: don't crash if plugin dies during initial plugins_init.
Thanks to Michael Schmook for the excellent bug report.

Fixes: https://github.com/ElementsProject/lightning/issues/7671
Changelog-Fixed: lightningd: no longer crash if a plugin dies during lightningd startup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-09-17 10:24:01 -07:00