When a peer connects, we always send all our own gossip (even if they
had set the timestamp filters to filter it out). But we weren't
forcing it out to them when it changed, so this logic only applied to
unstable or frequently-restarting nodes.
So now, we tell all the peers whenever we tell gossipd about our new
gossip.
Fixes: #7276
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now send current peers our changed gossip (even if they set timestamp_filter otherwise), not just on reconnect.
We didn't actually check that we *send* the refreshed gossip, just
that we print the message saying we're going to.
So check that everyone received updated gossip when this happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes, for various reasons, a user disables an offer
and then wants to re-enable it. This should be allowed because,
from the CLN point of view, it is just an internal state.
If a user has constraints on the description of the invoice
because they are using services that link some sort of user ID
to an offer, it is important for the user to be able to re-enable the
offer, not create a new one. Creating a new offer would
require a different description.
Link: https://github.com/ElementsProject/lightning/issues/7360
Co-Developed-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
I will commit doc/schemas/lightning-*.json files separately to keep this commit easier to review.
- test-autogenerate-rpc-examples updates all example request & responses in doc/schemas/lightning-*.json files.
- Updated tools/fromschema.py to accommodate the sql JSON example requirement where it does not accept -o in the query but shell does (for queries containing the = sign).
Changelog-None.
It might be nice to let the bookkeeper keep track of external accounts
as well as the internal onchain wallet? To this end, we add some new
custom notifications, which the bookkeeper will ingest and add to its
ledger.
Suggested-By: @chrisguida
Changelog-Added: PLUGINS: `bookkeeper` now listens for two custom events: `utxo_deposit` and `utxo_spend`. This allows for 3rd party plugins to send onchain coin events to the `bookkeeper`. See the new plugins/bkpr/README.md for details on how these work!
Also redirect config creation prompts to stderr in order to not interfere
with json output on stdout.
Changelog-Added: reckless provides json output with option flag -j/--json
This populates information on both topology (i.e. unannounced channels) and capacity for the local node using `listpeerchannels`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This marks all channels around the source node as free (no delay, no fee). This is normally what we want, if we are calculating a path for ourselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids globals (and means memleak traverses the variables!): we
only change over the test plugin though, to avoid unnecessary churn.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This helps code using generate_gossip_store() too, since it can map its identifiers
to the nodeids which were used.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a hack, but we've had nodes missing gossip. LDK does this,
so until we get a better workaround, use this one.
Changelog-Changed: Protocol: we now always ask the first peer for all its gossip.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: `invoicerequest` will set a blinded path if we're an unannounced node.
Changelog-EXPERIMENTAL: offers: `sendinvoice` will use a blinded path in an invoice_request, if specified.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an internal undocumented interface, which makes this change less painful.
We *do* check that the invreq_metadata maps to the given invreq_payer_id, which would
is required for us to sign it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's an internal difference, so doesn't actually break compatibility
(it would if we tried to prove we owned an old invoicerequest, but we
don't have infrastructure for that anyway).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The latest draft allows these experimental ranges, which involves more
changes than I expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: handle experimental ranges in offers/invoice_requests/invoices.
Changelog-Added: Protocol: pay can now pay to bolt12 invoices if entry to blinded hop is specified as a short_channel_id (rather than node id).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This also lets us test offers and invoices work if we are the start of
the blinded path.
Changelog-Added: offers: automatically add a blinded path from a peer if we have no public channels, so unannounced nodes can have offers too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only need a connection with a peer, not an actual channel. So
add all peers to the local gossmap.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means only a single gossmap, and they already share the fetchinvoice-noconnect option
and autoconnect code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: the `fetchinvoice` plugin has been combined into the `offers` plugin.
Making sure that we are able to fetch the invoice
from an offer without description
E pyln.client.lightning.RpcError: RPC call failed: method:
offer, payload: {'amount': '2msat'}, error: {'code': -32602, 'message':
'bolt12: Offer does not contain a description: invalid token
\'"lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqyppvggz953rvg9rtxj8lalh43z8epwydjfrmffn3y3p5qz5cywpu09rr4vs"\''}
Link: https://github.com/ElementsProject/lightning/issues/7405
Link: https://github.com/ElementsProject/lightning/issues/7404
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: Protocol: onion messages are now supported by default.
Changelog-Deprecated: Config: the --experimental-onion-messages option is ignored (on by default).
However fast we can handle them, it's antisocial to allow others to
make us spam the rest of the network.
Changelog-Protocol: onion messages: we limit incoming to 4 per second, allowing a little burst.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A bit tricky, since we get more than one message at a time. However,
this just means we go over quota for a bit, and will get caught when
those are sent (we do this for a single message already, so it's not
that much worse).
Note: this not only limits sending, but it limits the actuall query
processing, which is nice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently stream gossip as fast as we can, even if they start at
timestamp 0. Instead, use a simple token bucket filter and only let
them have 1MB per second (500 bytes per second for testing).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Protocol: connectd: we now throttle outgoing gossip at 1MB/second per peer.
Document and enforce the --experimental-anchors deprecation, which was somehow missed in v24.02
Changelog-Deprecated: Config: the --experimental-anchors option is ignored (on by default since v24.02).
We don't use the payment_secret in bolt12, but in onion_decode() we
do put the path_secret (if of correct length) into payment_secret. Not
realizing this confused me, so document that, and make sure we insist
on it being present for bolt12.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Shahana decided this was the optimal UX path, though I insisted that we still
report the actual problem directly when in dev mode, as a compromise.
Suggested-by: https://github.com/Amperstrand
Changelog-Changed: JSON-RPC: Do not return the contents of invalid parameters in error messages, refer to logs (use 'check' to get full error messages)
Fixes: https://github.com/ElementsProject/lightning/issues/7338
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2024-06-24T05:09:32.5233603Z assert l1.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234187Z > assert l2.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234917Z E AssertionError: assert 'FUNDING_SPEND_SEEN' == 'ONCHAIN'
2024-06-24T05:09:32.5235464Z E - ONCHAIN
2024-06-24T05:09:32.5235773Z E + FUNDING_SPEND_SEEN
2024-06-24T05:09:32.5236096Z
2024-06-24T05:09:32.5236242Z tests/test_misc.py:2907: AssertionError
```
It's possible that l2 hasn't seen the onchain tx yet. What we really want to do is
wait for it, then test that l1 really can spend the output it has recovered.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
2024-06-24T05:14:14.9939033Z > l1.rpc.call('dev-feerate', [l2.info['id'], rate - 5])
2024-06-24T05:14:14.9939354Z
2024-06-24T05:14:14.9939466Z tests/test_connection.py:3439:
...
2024-06-24T05:14:14.9967617Z > raise RpcError(method, payload, resp['error'])
2024-06-24T05:14:14.9968833Z E pyln.client.lightning.RpcError: RPC call failed: method: dev-feerate, payload: ['022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 2290], error: {'code': -1, 'message': 'Peer bad state'}
```
The disconnect can actually take a while: wait for both sides to
believe they're reconnected. Also wait a little to make sure the
feerate change doesn't create a bad signature.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
lightningd-3 penalizes lightningd-2 but then it can see the preimage for the HTLC which
has already been timed out:
```
2024-06-24T02:41:29.4633900Z lightningd-3 2024-06-24T02:33:54.073Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-onchaind-chan#1: HTLC already resolved by THEIR_HTLC_TIMEOUT_TO_THEM when we found preimage
```
This is fair: the test deliberately takes l3 offline for long enough
that the HTLC can get timed out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I was looking into using the `threading.Condition` but since we're
already rather heavily using callbacks, this allows us to stay
single-threaded, and not having to completely hook the `setconfig`
function.
Changelog-Added: pyln-client: Added a notification mechanism for config changes
We didn't actually *change* the value you'd see, when we got a setconfig call!
Changelog-Added: pyln-client: implement setconfig hook for plugins so you can see changes in `dynamic` options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a difficult transition for us: this string appears in channel
types. We make the transition now in the understanding that it will
be more difficult in future.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listpeers` `features` array string "option_anchors_zero_fee_htlc_tx": use "option_anchors" (spec renamed it).
Changelog-Added: JSON-RPC: `listpeers` `features` array string uses "option_anchors" for feature 22/23, following renaming in BOLT 9.
Changelog-Changed: JSON-RPC: `listclosedchannels`, `listpeerchannels`, `openchannel_update`, `openchannel_init`, `fundchannel`, `fundchannel_start` and `multifundchannel`: `channel_type` array `names` now contains "anchors" instead of "anchors_zero_fee_htlc_tx".
Changelog-Changed: lightningd: `--list-features-only` now lists "option_anchors" instead of "option_anchors_zero_fee_htlc_tx".
These were removed from the spec.
We still support existing ones, though we were the only implementation
which ever did, and only in experimental mode, so we should be able to
upgrade them and avoid a forced close, with a bit of engineering...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We still support *existing* channels. Just not new ones (before they could,
in theory, explicitly ask for one).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we do have to set the network correctly though,
and also we can get query messages from lightningd which we have to filter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We documented them as deprecated in v23.08 but unintentionally didn't deprecate them in the code. Thus we are starting their actual deprecation cycle from v24.08 release.
Updating removal version for commands `commando-rune`, `commando-listrunes`, and `commando-blacklist`
Changelog-None.
Somehow, this documentation got lost during the Great Rewrite, so
restore that too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `offer` removed `@` prefix support from `recurrence_base` (use `recurrence_start_any_period` set to `false`)
We do this by literally creating the modern-style TLV, and pretending we found it in the onion.
This isolates us from messing with any callers, who don't even know.
Co-programmed-with: Alex Myers <alex@endothermic.dev>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7347
Changelog-Fixed: Protocol: forward legacy non-TLV onions which we removed in 22.11 and spec itself in Feb 2022. Still sent by LND nodes who haven't seen our node_announcement.
This fails, because l2 can't decode the onion:
```
lightningd-2 2024-05-28T21:43:35.137Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Rejecting their htlc 0 since onion is unprocessable WIRE_INVALID_ONION_HMAC ss=4202c24ea44d9029a2ea3abb24cded51da93164f8bb5cddce9cc824af9945435
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to fire up channeld to send this, but:
1. That's silly, we have all the information to make it ourselves.
2. We didn't do it if there was an error on the channel, which as of 24.02
there always is!
3. When it did work, running channeld *stops* onchaind, indefinitely slowing recovery.
Fixes: https://github.com/Blockstream/greenlight/issues/433
Changelog-Fixed: Protocol: we once again send CHANNEL_REESTABLISH responses on closing channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
LDK will pick the *upper* limit (see: https://github.com/lightningdevkit/rust-lightning/issues/3014)!
It should not do this, but since you can set a manual range for mutual close, it's probably better to disable this option for close, as it's even more dangerous than documented.
Changelog-Changed: config/JSON: --ignore-fee-limits / setchannel ignorefeelimits no longer applies to mutual close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7242
Thought it's intuitive, it's not documented to, and LDK's current behavior makes it quite dangerous to do so.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we can see the values in listconfigs, even if we haven't set
them yet.
In particular, we now see the following:
* autoclean-cycle.value_int=3600
* bitcoin-rpcclienttimeout.value_int=60
* bitcoin-retry-timeout.value_int=60
* funder-max-their-funding.value_str=4294967295sat
* funder-per-channel-min.value_str=10000sat
* funder-reserve-tank.value_str=0sat
* funder-fund-probability.value_int=100
Changelog-Changed: plugins: libplugin now shows plugin option default values (where they're non-trivial)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This demonstrates a number of issues reported by Carla: no surprise
since there was no test!
(We create a one-hop blinded path when we only have private channels).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our fetchinvoice always creates a reply path which terminates at their peer,
so we need a dev overrride for that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The feerate security margin is a multiplicative factor applied to the
feerate of some transactions in order to guarantee that the
transaction remains publishable and has a sufficient chance of being
confirmed, that we can base some of our decisions on that.
The multiplicative factor is >=1 and was so far a constant 2. This
might have been sensible in the low-fee environment, where the fees
are expected to oscillate, and almost guaranteeing that we will
eventually have rising feerates but in high-fee environments that is
no longer the case, and the 100% margin that the multiplicator 2
brings is excessive. We therefore opt to start out with 100%, then
linearly interpolate up to a given maxfeerate (which does not have to
be a real feerate ever reached, it just indicates the feerate after
which we apply the constant 10% margin.
Fixes#6974Closes#6976
[Fixed up all the other changes required, including spendable calcualtion
comments and unit test and pytest tests --RR]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: we now understand blinded paths which use a short-channel-id(+direction) as entry point.
We don't actually support it yet, but this threads through the type change,
puts it in "decode" etc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And deprecate the --max-locktime-blocks which allows them to set it.
Hilariously, the spec misspells CLTV as CTLV at one point, so we work around it for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We expect:
UNUSUAL.*WARNING: we have 1 channels but can file descriptors limited to 65536
We get:
lightningd: WARNING: we have 1 channels but can file descriptors limited to 32768!
This is strange, since the first line is from Python's hard limit. Presumably something is restricting the fd limit of children
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1024 is a common limit, and people are starting to hit that many channels, so we should increase it: twice the number of channels seems reasonable, though we only do this at restart time.
Changelog-Changed: lightningd: we now try to increase the number of file descriptors, if it's less than twice the number of channels at startup (and log if we cannot!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Listpeerchannels would update the local channel information setting the
liquidity in the outgoing channel to known_min=known_max=capacity,
when in fact it should be known_min=known_max=spendable.
This didn't trigger the bug, but worth explicitly testing: we spend a
to-remote output from a previous unilateral close, to spend an anchor
on a current unilateral close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We wire through --dev options into the hsmd, and test preapprove accept and deby
with both old and new protocols.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we can check that the setconfig would actually parse. We do this
by handing NULL to the calback to indicate it's a test, which may not
work in general but works for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `check` `setconfig` now checks that the new config setting would be valid.
We need to pass through setconfig in check mode, then we need to have libplugin
add (and use!) a `check_only` parameter to all option setting.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` `setconfig` on plugin options can now check the config value would be accepted.
Without this, everything came out as level INFO.
Changelog-Fixed: pyln-client: Fix Plugin.notify_message() not to ignore `level` parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This test reproduces a bug in the `cln-plugin`-crate.
Core Lightning supports a wildcard `*` that plugins can use to
subscribe to all notifications.
However, `cln-plugin` does not support this case.
It allows the developer to subscribe `*`.
But the plug-in crashes when the first notification is received
Uncommited channels are missing several fields which would normally be
populated by an active channel. This simply skips them for the purposes
of finding a route. The particular culprit was:
{
"peer_id": "038cd9f3679d5b39bb2105978467918d549572de472f07dd729e37c7a6377d41d5",
"peer_connected": true,
"state": "OPENINGD",
"owner": "lightning_openingd",
"opener": "local",
"to_us_msat": 8317559000,
"total_msat": 8317559000,
"features": [
"option_static_remotekey",
"option_anchors_zero_fee_htlc_tx"
]
}
Fixes#7197 - SEGV in direct_pay_listpeerchannels when field private missing
Changelog-Fixed: Fixed crash in pay plugin caused by parsing uncommitted dual open channels
Changelog-Changed: Plugins: bcli: Add a path that tries to fetch blocks
from a peer if we can not find them locally.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
Rather than `allow_broken_log`, we have `broken_log` which is a regex
indicating what log lines are expected. This tightens our tests
significantly, as it will catch *unexpected* brokenness.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Apparently NixOS didn't have Python (sometimes!) so let's not assume.
By reusing the JSON "parsing" code from cowsay, we can self-disable
to handle this case.
Reported-by: Shahana Farooqui <shahana.farooqui@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `clnrest` now correctly self-disables if Python not present at all.
Tests whether we're able to route around a low-fee channel in the graph. We
should be able to find the more expensive route if the cheaper route is
depleted.
In the Min. Cost Flow solver we put a constraint on arcs capacities,
such that the total flow that can be allocated through a channel does
not exceed htlc_max. This solves two previous problem of the htlc_max
constraint at the MCF level.
Add spendable/receivable and is_local fields to the callback function
used in gossmods_from_listpeerchannels. This allows to do more fine
grained use of the listpeerchannels call.
The first use case is renepay, for which we need to ignore the htlc_max
of the local channels only.
I was trying to debug test_zeroconf_open and getting very confused.
The reason: l0 is lightning-1, l1 is lightning-2, etc! And there are
two other tests where an l0 has been added at the front: fix them all
to avoid future confusion!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows for faster startup for Greenlight. We still require full sync
before *incoming* htlcs, and onchain operations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
LDK doesn't set this feature if they don't have any useful gossip (mobile nodes)
and it was agreed at the spec meeting that we should repurpose this feature
to mean "I don't have any useful gossip".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These were deprecated in v22.08 (invoice hook) and v0.8 (htlc_accepted hook), and
marked EOL in v23.02.
*PLEASE* complain if this breaks things for you: it's kind of a test canary of
the deprecation system!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugins: `invoice_payment` and `htlc_accepted` hook `failure_code` response (derepcated v22.08 and v0.8, EOL v23.02)
Indeed, we now need to adjust our tests to use --i-promise-to-fix-broken-api-user
for the specific APIs past end-of-life.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were duplicating the command name (e.g. "autocleaninvoice.autocleaninvoice"), and also not
handling listconfigs if they specified the (currently unused!) i-promise-to-fix-broken-api-user
option, which we are going to use next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We tried to put *everything* into the "all" output, which didn't work
if there were other outputs!
Fixes: https://github.com/ElementsProject/lightning/issues/6664
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `multifundchannel` with `all` as an amount works as expected.
It's a u64, we should pass by copy. This is a big sweeping change,
but mainly mechanical (change one, compile, fix breakage, repeat).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: runes: named parameters (e.g. `pnameamountmsat`) no longer need to remove underscores (i.e. `pnameamount_msat` now works as expected).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than speaking 'rune' we should speak english in error messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @ShahanaFarooqui
