Commit graph

15115 commits

Author SHA1 Message Date
Rusty Russell
f8b259d5e9 askrene: add "auto.localchans" layer.
This populates information on both topology (i.e. unannounced channels) and capacity for the local node using `listpeerchannels`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
b8b8a40e9e askrene: split json_getroutes into two parts.
This will allow us to call an RPC function in the middle.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
61ccf18521 devtools/gossmap-compress: allow setting the nodeid explicitly for generated nodes.
This lets us make gossip which contains "real" nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
296b3ce20c plugin/askrene: add "auto.sourcefree" layer.
This marks all channels around the source node as free (no delay, no fee).  This is normally what we want, if we are calculating a path for ourselves.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
de19524b6d pytest: simple getroutes tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
45814bf8ac plugins/askrene: attach getroutes call to MCF code.
Now getroutes actually does something!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
79ceb59d7a plugins/askrene: remove local contexts.
In general, we should be using tmpctx unless there's a specific reason not to.
It's clear, and simplifies the code somewhat.

If tmpctx is not cleaned often enough, we can look at a per-MCF context, but this
seems like premature optimization.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
b1817b6c52 askrene: include the mcf and flow routines.
This make the code use askrene's "struct route_query".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
be054dcb3f askrene: make the flow.[ch] files compile.
This adapts them to their new locations, and copies a few more routines.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
1db5cf6dea askrene: simply fail if a flow amount exceeds 64 bits.
Rather than handling failure, simply report and exit the plugin.
Simplifies error handling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
7bf399cac5 askrene: remove code which tries to handle tal failures.
tal does not fail: the default handler (which we use) aborts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
37793b0ac2 askrene/flow: don't omit initial hop in flow_spend.
That will be done in the caller, not here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
e4b84f1ffb askrene: copy flow and dijkstra from renepay.
Still don't actually try compiling them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
d109fcb568 askrene: simplify minflow()
We let the caller choose mu, and iterate if necessary: it can also
check its limits for fees, etc.  Rationalize it to 0-100 inclusive for
human consumption.

This means we don't loop internally, and in fact there's only one
failure mode: we cannot find enough capacity.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
5999467dce askrene: copy mcf.[ch] from renepay with minimal modifications.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
22a32e6e09 pytest: test file for askrene.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
990bbdde27 askrene: fast lookup for capacities.
We don't know anything about most channels, so we create an array of
fp16_t containing them.  We zero out ones where we do know something,
and use the previous code as the slow path.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
eb335ad763 askrene: always set a dummy constraint when we add a local channel.
This means we never have to look up a local channel when asked the capacity.

We mark these dummy constraints with an MAX timestamp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
37bdaff8e2 askrene: flesh out getroutes() a little.
We apply all the gossmods for the layers they specified, and create a
naive routine to give the capacity of a channel given those layers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
c17c240f79 askrene: reservation implementation.
They tell us what paths they're using, so we can adjust capacity estimates
accordingly.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


Header from folded patch 'reserve-fixup.patch':

fixup! askrene: reservation implementation.
2024-08-07 20:35:30 +09:30
Rusty Russell
d7f983a5b5 askrene: add layers infrastructure.
These are the repositories of all information.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>


Header from folded patch 'layers-fixup.patch':

fixup! askrene: add layers infrastructure.
2024-08-07 20:35:30 +09:30
Rusty Russell
ad209182ba askrene: skeleton which does JSON API.
All the infrastructure and interfaces, but it doesn't do anything yet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
9b19eada30 common: promote useful routines from renepay.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
a29d135bee bitcoin: add short_channel_id_dir_eq.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
9935e28358 common: new parameter parsing routines.
param_u16 (for delay).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
11bfbf5deb libplugin: add data pointer for plugin convenience.
This avoids globals (and means memleak traverses the variables!): we
only change over the test plugin though, to avoid unnecessary churn.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
e11bab8bbb gossmap: don't process channel_announcement until amount is present.
This simplifies the callers significantly: all channel_announcements now
have an amount, so gossmap_chan_get_capacity() only fails on a local
modification.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
4d593463fa doc: document askrene.
It's experimental, so API may change, but it definitely does need explanation!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
Rusty Russell
8dbc1c5f2e offers: fix crash when receiving response to offer without offer_issuer_id
Now we actually check the other fields too, as per BOLT!

Reported-by: https://github.com/hMsats
Fixes: #7513
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-06 19:09:27 -07:00
Rusty Russell
28b93e1b5f pytest: make GenChannel arguments explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
e3f06b2602 devtools/gossmap-compress: print out node ids.
This helps code using generate_gossip_store() too, since it can map its identifiers
to the nodeids which were used.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
53faab0838 pytest: add routine to generate gossmap by topology.
We marshal it into the "compressed" format and get the decompresser to
build the actual gossmap.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
99e8e9246e devtools/gossmap-compress: use transparent zlib compression if available.
Before:
```
-rw-rw-r-- 1 rusty rusty 1643258 Jul 26 09:51 compressed
```

After:
```
-rw-rw-r-- 1 rusty rusty 508332 Jul 26 09:49 compressed
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
c93b4aafb2 devtools/gossmap-compress: decompress code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
cf936d296e devtools: add gossmap-compress to give minimal representation of gossmap topology.
Simple format, which doesn't include node information, just the channels.

Example:

```
$ ls -l gossip-store-2024-06-26
-rw------- 1 rusty rusty 98815543 Jul 26 09:47 gossip-store-2024-06-26
$ ./devtools/gossmap-compress -v compress gossip-store-2024-06-26 compressed
18693 nodes
61437 channels
46148 disabled channels (32620 no update)
9690 unique capacities
85 unique htlc_min
6867 unique htlc_max
807 unique basefee
2521 unique propfee
94 unique delay
$ ls -l compressed
-rw-rw-r-- 1 rusty rusty 1643258 Jul 26 09:51 compressed
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
daywalker90
0a73f918c1 macos: fix ci and autoclean
Changelog-None
2024-08-07 10:57:26 +09:30
ShahanaFarooqui
a82f1feb3a pyln-client: Deprecate category, description and long description from method
Category, description and long description from `json_command` and `plugin_command` have been removed in favour of getting them from json schema.  Reference PR: Add categories in RPC documentation #7485

Deprecating them in pyln-client as well. They will remove completely in future releases.

Changelog-Deprecated: pyln-client: category, description and long descriptions for RPC commands are deprecated now.
2024-08-07 10:19:55 +09:30
Rusty Russell
bb400238b1 gossipd: always ask first peer for all the gossip.
This is a hack, but we've had nodes missing gossip.  LDK does this,
so until we get a better workaround, use this one.

Changelog-Changed: Protocol: we now always ask the first peer for all its gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 10:15:03 +09:30
Rusty Russell
855d7366c7 CI: don't fall over if cannot connect to upload server.
```
>           warnings.warn(f"Error reporting testrun: {e}: {e.read()}")
E           AttributeError: 'URLError' object has no attribute 'read'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-06 02:00:35 -05:00
Rusty Russell
f77d4d7097 offers: fix flake in fetchinvoice.
In CI, this would sometimes fail: we would timeout waiting for the
fetchinvoice reply.  Never happened locally, so was annoying to debug.

What happened was simple: we called injectonionmessage then when it
returned, put the "sent" object in the linked list so we could recognize
any reply onion messages.

However, we were getting that reply before the plugin processed the response
to injectonionmessage.  This is possible because there are two fds for
plugins: one for it to receive notifications and hooks (like onion messages)
and one for normal RPC usage (like commands to inject onion messages).

The fix is simple: put in the list *before* calling JSON RPC.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-05 16:35:15 +02:00
Rusty Russell
2e06d814e5 lightningd: fix false positive in leak detection.
We ask each channeld to report its leaks, and keep a refcount of how
many are outstanding.  When the channeld replies, or dies, we decrement
the count in a destructor.

But if the last channeld we're waiting for dies, we can call the
destructor in an unexpected place, and thus run leak detection when
we were partway through some operation, which gives a false positive.
Here's a backtrace showing it:

```
0x5f93405897e3 send_backtrace
	common/daemon.c:33
0x5f93405381cf finish_report
	lightningd/memdump.c:139
0x5f93405382a0 leak_detect_req_done
	lightningd/memdump.c:172
0x5f9340705c41 notify
	ccan/ccan/tal/tal.c:243
0x5f9340705ca5 del_tree
	ccan/ccan/tal/tal.c:437
0x5f9340705ce8 del_tree
	ccan/ccan/tal/tal.c:447
0x5f93407061f3 tal_free
	ccan/ccan/tal/tal.c:532
0x5f9340563f5a subd_release_channel
	lightningd/subd.c:924
0x5f934050fb91 channel_set_owner
	lightningd/channel.c:31
0x5f9340511b84 channel_err
	lightningd/channel.c:1081
0x5f93405121a3 channel_fail_transient
	lightningd/channel.c:1095
0x5f934054e547 peer_channels_cleanup
	lightningd/peer_control.c:187
0x5f9340550411 peer_connected
	lightningd/peer_control.c:1689
0x5f9340525101 connectd_msg
	lightningd/connect_control.c:629
```

Instead, do the lightningd detection up-front, where it's in a
clean environment.

Reported-by: Shahana Farooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-04 11:47:54 +02:00
Rusty Russell
0b4b4c5e03 lightningd: don't create a second peer on stub recover if one already exists.
This was confusing to debug, but if the peer already exists we must not
create a second one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-04 10:48:36 +09:30
Rusty Russell
bf549130d6 global: update BOLT12 quotes.
This is a final sweep to match the current BOLT12 text:

	1563d13999d342680140c693de0b9d65aa522372 ("More bolt12 test vectors.")

Only two code changes, to change the order of checks to match the bolt,
and to give a warning on decode if a path is empty.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
f2a7b19281 plugins/offers: handle invreq_paths in invoice_requests, set them if necessary.
Changelog-EXPERIMENTAL: offers: `invoicerequest` will set a blinded path if we're an unannounced node.
Changelog-EXPERIMENTAL: offers: `sendinvoice` will use a blinded path in an invoice_request, if specified.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
71270ae795 lightningd: make the caller set invreq_metadata and invreq_payer_id for createinvoicerequest.
It's an internal undocumented interface, which makes this change less painful.

We *do* check that the invreq_metadata maps to the given invreq_payer_id, which would
is required for us to sign it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
74ef03d361 lightningd: use tweak on node_id of different key, for createinvoicerequest.
It's an internal difference, so doesn't actually break compatibility
(it would if we tried to prove we owned an old invoicerequest, but we
don't have infrastructure for that anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
fa33a2fece lightningd: make plugins set metadata for invoice requests.
They can do it now: before it would have been awkward to look up previous
payments to match it up for recurring offers (which need to use the same
key, hence the same invreq_metadata).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
401d85fce0 lightningd: make invreq metdata deterministic, not random for recurring invoices.
This is going to allow us to move it out from lightingd into plugin,
easily.  It's legal because the combination of offer id and label must
be unique (with recurrence, we use the same metadata anyay, since they
want to correlate with prior payments anyway).

We already broke recurrence in this PR, so we don't need another note to say
we're changing the key derivation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
98f07e92c2 common: fix up some minor typos in BOLT quotes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
e8a38f111d hsmd: create an hsm variant to sign a bolt12 invoice using a tweak on our key.
The current interface, if given a tweak, uses a *different secret key*
and tweaks it.  This was an early experiment: we will switch to using
a secret tweak for invoice_requests like we do for invoice path ids.

To make sure there's no funny business, *hsmd* hashes to form the
tweak (i.e. no zero tweaks!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30