Commit Graph

15083 Commits

Author SHA1 Message Date
Rusty Russell
99e8e9246e devtools/gossmap-compress: use transparent zlib compression if available.
Before:
```
-rw-rw-r-- 1 rusty rusty 1643258 Jul 26 09:51 compressed
```

After:
```
-rw-rw-r-- 1 rusty rusty 508332 Jul 26 09:49 compressed
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
c93b4aafb2 devtools/gossmap-compress: decompress code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
Rusty Russell
cf936d296e devtools: add gossmap-compress to give minimal representation of gossmap topology.
Simple format, which doesn't include node information, just the channels.

Example:

```
$ ls -l gossip-store-2024-06-26
-rw------- 1 rusty rusty 98815543 Jul 26 09:47 gossip-store-2024-06-26
$ ./devtools/gossmap-compress -v compress gossip-store-2024-06-26 compressed
18693 nodes
61437 channels
46148 disabled channels (32620 no update)
9690 unique capacities
85 unique htlc_min
6867 unique htlc_max
807 unique basefee
2521 unique propfee
94 unique delay
$ ls -l compressed
-rw-rw-r-- 1 rusty rusty 1643258 Jul 26 09:51 compressed
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 11:18:55 +09:30
daywalker90
0a73f918c1 macos: fix ci and autoclean
Changelog-None
2024-08-07 10:57:26 +09:30
ShahanaFarooqui
a82f1feb3a pyln-client: Deprecate category, description and long description from method
Category, description and long description from `json_command` and `plugin_command` have been removed in favour of getting them from json schema.  Reference PR: Add categories in RPC documentation #7485

Deprecating them in pyln-client as well. They will remove completely in future releases.

Changelog-Deprecated: pyln-client: category, description and long descriptions for RPC commands are deprecated now.
2024-08-07 10:19:55 +09:30
Rusty Russell
bb400238b1 gossipd: always ask first peer for all the gossip.
This is a hack, but we've had nodes missing gossip.  LDK does this,
so until we get a better workaround, use this one.

Changelog-Changed: Protocol: we now always ask the first peer for all its gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 10:15:03 +09:30
Rusty Russell
855d7366c7 CI: don't fall over if cannot connect to upload server.
```
>           warnings.warn(f"Error reporting testrun: {e}: {e.read()}")
E           AttributeError: 'URLError' object has no attribute 'read'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-06 02:00:35 -05:00
Rusty Russell
f77d4d7097 offers: fix flake in fetchinvoice.
In CI, this would sometimes fail: we would timeout waiting for the
fetchinvoice reply.  Never happened locally, so was annoying to debug.

What happened was simple: we called injectonionmessage then when it
returned, put the "sent" object in the linked list so we could recognize
any reply onion messages.

However, we were getting that reply before the plugin processed the response
to injectonionmessage.  This is possible because there are two fds for
plugins: one for it to receive notifications and hooks (like onion messages)
and one for normal RPC usage (like commands to inject onion messages).

The fix is simple: put in the list *before* calling JSON RPC.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-05 16:35:15 +02:00
Rusty Russell
2e06d814e5 lightningd: fix false positive in leak detection.
We ask each channeld to report its leaks, and keep a refcount of how
many are outstanding.  When the channeld replies, or dies, we decrement
the count in a destructor.

But if the last channeld we're waiting for dies, we can call the
destructor in an unexpected place, and thus run leak detection when
we were partway through some operation, which gives a false positive.
Here's a backtrace showing it:

```
0x5f93405897e3 send_backtrace
	common/daemon.c:33
0x5f93405381cf finish_report
	lightningd/memdump.c:139
0x5f93405382a0 leak_detect_req_done
	lightningd/memdump.c:172
0x5f9340705c41 notify
	ccan/ccan/tal/tal.c:243
0x5f9340705ca5 del_tree
	ccan/ccan/tal/tal.c:437
0x5f9340705ce8 del_tree
	ccan/ccan/tal/tal.c:447
0x5f93407061f3 tal_free
	ccan/ccan/tal/tal.c:532
0x5f9340563f5a subd_release_channel
	lightningd/subd.c:924
0x5f934050fb91 channel_set_owner
	lightningd/channel.c:31
0x5f9340511b84 channel_err
	lightningd/channel.c:1081
0x5f93405121a3 channel_fail_transient
	lightningd/channel.c:1095
0x5f934054e547 peer_channels_cleanup
	lightningd/peer_control.c:187
0x5f9340550411 peer_connected
	lightningd/peer_control.c:1689
0x5f9340525101 connectd_msg
	lightningd/connect_control.c:629
```

Instead, do the lightningd detection up-front, where it's in a
clean environment.

Reported-by: Shahana Farooqui
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-04 11:47:54 +02:00
Rusty Russell
0b4b4c5e03 lightningd: don't create a second peer on stub recover if one already exists.
This was confusing to debug, but if the peer already exists we must not
create a second one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-04 10:48:36 +09:30
Rusty Russell
bf549130d6 global: update BOLT12 quotes.
This is a final sweep to match the current BOLT12 text:

	1563d13999d342680140c693de0b9d65aa522372 ("More bolt12 test vectors.")

Only two code changes, to change the order of checks to match the bolt,
and to give a warning on decode if a path is empty.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
f2a7b19281 plugins/offers: handle invreq_paths in invoice_requests, set them if necessary.
Changelog-EXPERIMENTAL: offers: `invoicerequest` will set a blinded path if we're an unannounced node.
Changelog-EXPERIMENTAL: offers: `sendinvoice` will use a blinded path in an invoice_request, if specified.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
71270ae795 lightningd: make the caller set invreq_metadata and invreq_payer_id for createinvoicerequest.
It's an internal undocumented interface, which makes this change less painful.

We *do* check that the invreq_metadata maps to the given invreq_payer_id, which would
is required for us to sign it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
74ef03d361 lightningd: use tweak on node_id of different key, for createinvoicerequest.
It's an internal difference, so doesn't actually break compatibility
(it would if we tried to prove we owned an old invoicerequest, but we
don't have infrastructure for that anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
fa33a2fece lightningd: make plugins set metadata for invoice requests.
They can do it now: before it would have been awkward to look up previous
payments to match it up for recurring offers (which need to use the same
key, hence the same invreq_metadata).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
401d85fce0 lightningd: make invreq metdata deterministic, not random for recurring invoices.
This is going to allow us to move it out from lightingd into plugin,
easily.  It's legal because the combination of offer id and label must
be unique (with recurrence, we use the same metadata anyay, since they
want to correlate with prior payments anyway).

We already broke recurrence in this PR, so we don't need another note to say
we're changing the key derivation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
98f07e92c2 common: fix up some minor typos in BOLT quotes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
e8a38f111d hsmd: create an hsm variant to sign a bolt12 invoice using a tweak on our key.
The current interface, if given a tweak, uses a *different secret key*
and tweaks it.  This was an early experiment: we will switch to using
a secret tweak for invoice_requests like we do for invoice path ids.

To make sure there's no funny business, *hsmd* hashes to form the
tweak (i.e. no zero tweaks!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
09401e34b6 common/bolt12_id: generate alias tweak.
For now we only use a fake id for requesting invoices (as a payer_key), but we
will eventually use this generically, and we want plugins to be able to map them
too, so use the same scheme as path_id: a generated secret using the makesecret
API.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
ee47b9370f common/bolt12_id: new common routine for creating path secrets.
invoice_path_id is actually a generic path_id thing, so rename it.

We're going to use the same scheme for path secrets and the tweak to
node_id when we create a fake pubkey for invoice_requests, so a new
header is appropriate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
363e1ee44c common: update test to encode new test vectors now we handle them correctly.
1. Missing offer_description iff offer_amount also missing.
2. Missing offer_issuer_id iff offer_paths is present.
3. Short channel id on introduction point.
4. Experimental range.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
d740795139 common/bolt12: allow missing offer_issuer_id.
The latest spec allows this to be omitted iff there is a blinded path
and it would be made up anyway.

In that case, the key they will use to sign the invoice will be the final
blinded key in the path we use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
2ecf5e6bd5 BOLT12: reject zero-length blinded paths.
This is a good idea, but also specifically called out in the latest BOLT spec.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
18fc1b82b0 wire: add new field invreq_paths from spec.
This is the blinded path for offerless invoice_requests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
0f0a0cf153 doc: mark schema offer_node_id and warning_missing_offer_node_id deprecated.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
925e90ee8b wire: rename "offer_node_id" to "offer_issuer_id" to match latest BOLT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
f42e9b3fe6 wire: renumber experimental recurring offer fields.
The `invreq_recurrence_counter` clashes with the coming addition of
invreq_paths, so we might as well move them all to the new experimental
ranges.

Changelog-EXPERIMENTAL: Recurring offers had incompatible changes, won't work against older versions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Rusty Russell
ef2905895b offers: fix invoice return comparison.
This mistake was revealed when we start using experimental range for recurrence
(next commit).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-01 12:31:48 +09:30
Pavol Rusnak
5c494488e7 pyln-proto: update cryptography to v42
Changelog-None
2024-07-31 16:34:55 +02:00
michael1011
46c3920585 grpc-plugin: Run cargo fmt for minor style fixes 2024-07-31 08:53:12 +02:00
michael1011
9b1f331748 Revert "ci: Pin rustc to version 1.78 due to a type inference regression"
This reverts commit 76ad48cd1b.
2024-07-31 08:53:12 +02:00
michael1011
27df5aa03b grpc-plugin: Update rcgen
The version of rcgen that was used before does not compile with
the latest stable Rust release v1.80
2024-07-31 08:53:12 +02:00
ShahanaFarooqui
2ff3e55f08 plugin: Removing category, description and long_description from plugin_command struct
Changelog-None.
2024-07-31 14:42:58 +09:30
ShahanaFarooqui
b485a026f7 rpc: Removing description from json_command struct 2024-07-31 14:42:58 +09:30
ShahanaFarooqui
89c182e2be rpc: Removing category and verbose from json_command struct 2024-07-31 14:42:58 +09:30
ShahanaFarooqui
58e3ea574c docs: Adding readonly category for all get & list commands except listdatastore 2024-07-31 14:42:58 +09:30
Christian Decker
10acbffcf9 onchaind: Remove key derivation involving the signer
The signer may not be present at this time. If we want to keep the
check to protect against bit flips we should move it into `onchaind`
where it doesn't matter as much that the signer may be slow to
respond.
2024-07-29 17:08:33 +02:00
Christian Decker
d496126af8 tests: Adjust tests for the lack of unilateral attempt now 2024-07-29 17:08:33 +02:00
Christian Decker
6c972cdb32 close: Do not publish unilateral when witnessing a close onchain
Changelog-Changed: close: We no longer attempt to publish a unilateral close that'd fail anyway when we witness a close onchain.
2024-07-29 17:08:33 +02:00
Christian Decker
76ad48cd1b ci: Pin rustc to version 1.78 due to a type inference regression
Changelog-None
2024-07-26 15:22:04 +02:00
Rusty Russell
15fb37f6d1 common: fix endless loop in gossmap iteration.
If we need to iterate forward to find a timestamp (only happens if we have gossip older than
2 hours), we didn't exit the loop, as it didn't actually move the offset.

Fixes: https://github.com/ElementsProject/lightning/issues/7462
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-25 12:30:42 -07:00
Rusty Russell
88024fa8d6 common/bolt12, offers plugin: handle experimental ranges in bolt12 correctly.
The latest draft allows these experimental ranges, which involves more
changes than I expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: handle experimental ranges in offers/invoice_requests/invoices.
2024-07-23 09:54:47 +09:30
Rusty Russell
6d6716c9ec devtools/bolt12-cli: fix decoding of unknown fields.
We had known vs unknown backwards ("it's OK to be odd!")

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30
Rusty Russell
c342f204b2 common/wire: update, don't replace fields array in TLV structures.
Regnerating them entirely loses unknown fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30
Rusty Russell
ea4025b1ed plugins/offers: fix JSON output of decode when there are unknown fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30
Rusty Russell
1e2c6a5837 common: maintain unknown fields when unmarshalling/marshalling TLVs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: maintain unknown fields offers/invoice_requests correctly.
2024-07-23 09:54:47 +09:30
Rusty Russell
7735eb6767 doc: allow "offer_description" to be missing in decode.
This is now allowed to be omitted.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30
Rusty Russell
90b06bbd13 msggen: fix up previous generated "deprecated": false.
Between this and the last fix, we can now deprecate things in future.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30
Rusty Russell
0a1aa972d8 msggen: fix deprecated from "False" to "None".
Turns out we set this to False instead of None, so new field get "deprecated": False.
When we actually deprecate one, we get the following error:

	ValueError: Field Decode.offer_node_id changed `deprecated` annotation: v24.08 vs False

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30
Rusty Russell
21a83a8f79 wire: remove unused obsolete file.
This is from back when we removed EXPERIMENTAL builds.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-07-23 09:54:47 +09:30