Changelog-Added: Protocol: onion messages are now supported by default.
Changelog-Deprecated: Config: the --experimental-onion-messages option is ignored (on by default).
However fast we can handle them, it's antisocial to allow others to
make us spam the rest of the network.
Changelog-Protocol: onion messages: we limit incoming to 4 per second, allowing a little burst.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is now permitted in the offers PR, so we should support it. But
we can't just look up in the gossmap, since the "short_channel_id"
could be an alias. So we get lightningd to tell us all scid->peer
mappings, and look up in that.
Changelog-Added: Protocol: onion messages can now be forwarded by short_channel_id.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we set them (i.e. at lockin), when we fire up channeld (for
aliases, which we create at channel init, but aren't really useful
until we have finished channel opening), and at startup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unlike "sendonionmessage" which instructs us to send to a peer, this
process it locally (presumably, it contains the next hop). This is
useful because it allows us to process an onion message which starts
with us (a legal case for a blinded path supplied by someone else!).
It also opens the door to bolt12 self-pay.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Allows for caller to log, but more importantly, when we add a command to
inject onion messages, allows for us to capture the error.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For some reason CI now hits a race where it tries to analyze a still-being-generated file:
```
tools/headerversions.c:52:30: error: syntax error [syntaxError]
new = tal_fmt(NULL, template,
^
make: *** [Makefile:552: check-cppcheck] Error 123
```
Restrict it to real source files instead.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A bit tricky, since we get more than one message at a time. However,
this just means we go over quota for a bit, and will get caught when
those are sent (we do this for a single message already, so it's not
that much worse).
Note: this not only limits sending, but it limits the actuall query
processing, which is nice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This basically means moving the code from gossipd to connectd to handle
these queries.
This will get connectd have finer control over ratelimiting them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was removed from the spec on Apr 25, 2022. We stopped ever sending them
in 0.12.0 (2022-08-23). Now we remove receive support.
Changelog-Protocol: Removed support for zlib-compressed short-channel-ids in query responses.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is more efficient in a few ways:
1. It's trivial to get to the end of the gossip_store, we don't have
to iterate.
2. It tends to be mmaped so we don't have to call pread().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently stream gossip as fast as we can, even if they start at
timestamp 0. Instead, use a simple token bucket filter and only let
them have 1MB per second (500 bytes per second for testing).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: connectd: we now throttle outgoing gossip at 1MB/second per peer.
We never call `listconfigs` in our tests with `experimental-offers` enabled,
so we didn't notice that the schema is wrong: it does not expect the
"plugin" field in 'fetchinvoice-noconnect'.
The next patch folds the fetchinvoice plugin into the offers plugin,
which is enabled even if `experimental-offers` isn't (for `decode`),
so we notice it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Document and enforce the --experimental-anchors deprecation, which was somehow missed in v24.02
Changelog-Deprecated: Config: the --experimental-anchors option is ignored (on by default since v24.02).
Only sphinx internally uses the hmac field: it's actually a general descriptor
of onion contents, which we can use elsewhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't use the payment_secret in bolt12, but in onion_decode() we
do put the path_secret (if of correct length) into payment_secret. Not
realizing this confused me, so document that, and make sure we insist
on it being present for bolt12.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was when we handled pre-TLV onions where the first byte was 0. We haven't
done that for a while: you can tell, because process_onionpacket doesn't use
the parameter at all!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can't happen because we go the self-pay path in this case, but
once we fix that for bolt12, this can be reached.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were getting the following message in test_feerate_stress:
```
2024-07-08T02:15:45.5663941Z lightningd-2 2024-07-08T02:13:45.696Z **BROKEN** 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-connectd: Peer did not close, forcing close
```
I can reproduce it locally if I run the test enough, and finally found
the issue by printing the status of the fd when we time it out (using
routines from connectd.c).
The peer fd alternates between reading and writing. When we go to
discard it, we wake the write queue, so write_to_peer() get called.
It won't shutdown the socket if there are still subds attached, and
will wait again for a read.
The last subd exit has to also wake the write queue if we're draining,
so it can do the io_sock_shutdown. Otherwise, we hit the timeout,
causing the message above.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Theoretical only, but we could leak an fd if we closed a conn before
the fd was sent. This doesn't happen in our current codebase because
we only hand fds to connectd, which only closes at shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only hand fds to connectd for now, so this doesn't happen (we don't
destroy that queue except on shutdown). But it's still a potential issue.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If the peer sends you their signatures before we return from the *first*
openchannel_update then the state would still be in
MULTIFUNDCHANNEL_STARTED (not UPDATED) and we'd incorrectly switch the
state to MULTIFUNDCHANNEL_SIGNED, which would plunge us headfirst into
`check_sigs_ready`.
However the `mfc->txid` hadn't been set yet, so we'd crash when trying
to confirm that we've got sigs for the correct txid. Oops.
To fix this, we simply invert the check such that the *only* state that
will move into SIGNED is SECURED
Interestingly, this patch and the last take my total test time on my
build machine down by about 10%. From 403.93s (0:06:43) to 366.64s (0:06:06)
though there were some unrelated errors.
Changelog-Changed: connectd: I/O optimizations to significantly speed up larger nodes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A fairly simple change: ccan/io will now call the underlying I/O
routines repeatedly until they indicate they are unfinished, *or* fail
with EAGAIN. This should make a significant difference to large
nodes, which currently spend far too much time calling poll() to
discover a single fd is still writable (mainly, for streaming gossip).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: connectd: now should use far less CPU on large nodes.
common/version.o depends on common/version_gen.h explicitly already,
and header_versions_gen.h is only used by lightningd, so we can make
that dependency explicit.
This means when version changes (i.e. different git commit) we only
relink, not recompile.
Before:
```
real 0m6.578s
user 0m14.705s
sys 0m13.621s
```
After:
```
real 0m2.098s
user 0m6.763s
sys 0m4.844s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This slows compilation somewhat when we make small changes (e.g. making the tree dirty).
Instead, simply mark them as "pre-XXX" or "XXX".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Their dependencies seem correct, so we shouldn't have to build them every time.
Before:
```shell
$ time make -s
real 0m5.183s
user 0m5.134s
sys 0m11.715s
```
After:
```shell
$ time make -s
real 0m0.784s
user 0m0.775s
sys 0m1.159s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The file `cln-rpc/src/notifications.rs` is generated by `msggen`.
The `Makefile` has tooling to update the file when needed
- `make check-gen-updated` should error if file isn't properly updated
- `make gen` updates the file
This comit fixes both behaviors mentioned above.
As suggested by @cdecker, I compared `poetry show --tree` results from before/after `cd plugins/clnrest/ && poetry export --output requirements.txt && pip install -r requirements.txt` to check for any significant difference. There is no difference in the tree.
But poetry export removed coincurve & other dependencies from clnrest's requirements.txt.
Changelog-None.
clnrest installation instruction on the markdown tries to install `flask_restx` not `flask-restx`.
Reference: clnrest plugin complains of flask_restx missing (#7383)
Changelog-None.
