There's a known issue with hsm passwords and valgrind:
```
write_all(master_fd, (password + '\n').encode("utf-8"))
> l1.daemon.wait_for_log("Server started with public key")
tests/test_plugin.py:4526:
...
if self.is_in_log(r):
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Server started with public key')]" in logs.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Based on the patch by bstin <barry.github@capsmx.com>, which added a separate command,
this simply extends "generatehsm" to allow more options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: hsmtool: generatehsm can run non-interactive, taking options on the cmdline.
This means it always tells us explicitly whether to keep watching or not,
and we know it's processed it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And we hook in the replay watch code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `onchaind` can miss conclusion of final txs in some cases, will now replay independently.
And use it for `exposesecret-passphrase`. This is probably overly
cautious, but it makes me feel a little better that we won't leak it
to someone with read-only access.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Being able to back up the hsm_secret is critical, but you cannot do
this through a UI, because of course we do not allow such access.
People have lost funds because they didn't back up.
This allows access to the hsm_secret if you use a password set in the
config file. (If it's not set, the command does not work). This is a
compromise, of course.
Changelog-Added: `exposesecret` command for encouraging hsm_secret backups.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This takes an {payment_id} and {description}.
It looks for all chain + channel events that match
that {payment_id} and updates the description for those events.
We return all the updated events. If no events are updated, an empty
list is returned.
Changelog-Added: PLUGINS: bookkeeper has a new RPC `bkpr-editdescriptionbypaymentid` which will update the description for any event with matching payment_id
This is the sister command of addpsbtoutput.
Adds inputs equal to or greater than the amount requests, reservers them, and reports important information back out to the user.
Changelog-Added: New low-level RPC command addpsbtinput to fund PSBTs directly and help with complex splices & dual-opens.
The ability to stfu channels in bulk is required to do complex multi channel operations. When stfu’ing in this manner, the available funds at the moment of stfu is returned to the user.
In order to cancel the stfu we also add a bulk tx_abort command.
Changelog-Added: `stfu_channels` and `abort_channels` are added for bulk multi-channel splice commands. These allow the user to pause (and resume) multiple channels in place.
This is needed to all multi-channel splices. When channeld can return the signatures to the user (based on signing order precedent), it now does from splice_update.
Additionally, we move sending of the initial psbt from splice_init down to splice_update. This is also necessary for correct psbt diff detection during multi-channel splices.
Changelog-Changed: splice_update can in some cases now return the remotely partiall signed psbt to the user, if so `signtures_secured` will be true.
This lets you place annotated biases on channels, to influence routing.
Uses include avoiding TOR nodes, slow channels or other local preferences.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: askrene is new anyway.
Without knowing what method was called, we can't have useful general logging
methods, so go through the pain of adding "const char *method" everywhere,
and add:
1. ignore_and_complete - we're done when jsonrpc returned
2. log_broken_and_complete - we're done, but emit BROKEN log.
3. plugin_broken_cb - if this happens, fail the plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we used to allow cmd to be NULL, we had to hand the plugin
everywhere. We no longer do.
1. Various jsonrpc_ functions no longer need the plugin arg.
2. send_outreq no longer needs a plugin arg.
3. The init function takes a command, not a plugin.
4. Remove command_deprecated_in_nocmd_ok.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I started getting "WIRE_TEMPORARY_CHANNEL_FAILURE: Too many HTLCs" after
two hundred xpay attempts.
This was nice (it found some bugs in injectpaymentonion's handling of
local errors, and in xpay's reporting), but shouldn't happen.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Start with a random capacity (linear prob), and remember in-progess
payments so we can simulate them using capacity properly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note the impedence mismatch between sendpay and getroutes: we have to shift
amounts and delays by 1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our gossmap_store uncompresser generates nodeids with well-known
privkeys, so we can decrypt and respond to HTLCs sent to such nodes.
By replacing channeld with a fake, we can connect a node to another
node, but then once the channel is established, allow payments to be
sent into the generated network, and respond appropriately.
This minimal version handles MPP timeouts, but doesn't insert any
delays or runtime capacity for channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: Testing only
Now pay learns, it sometimes learns not to try again:
```
> assert(len(l1.rpc.listpays()['pays']) == 2)
E AssertionError: assert 1 == 2
E + where 1 = len([{'amount_sent_msat': 0, 'bolt11': 'lnbcrt1pnjj7mysp5tfx8n6nyx7ehszgqn7gqm2r6n079p22u2yddtg797ka3pa9557tspp5f89z6genjqrl3knymvav9ajwcxrm5w7arxux06rrhjux88derjyqdq8v3jhxccxqyjw5qcqp9rzjqgkjyd3q5dv6gllh77kygly9c3kfy0d9xwyjyxsq2nq3c83u5vw4jqqqvuqqqqsqqqqqqqqpqqqqqzsqqc9qxpqysgqcuyr7qlyctf9w96fqg4wetqt7t5v938dagmv0r777n902utjufujzjxl3289r97yngft966zly3ehxfp469dh3lq0hkv6r684snvunqppuyvsl', 'created_at': 1730771812, 'destination': '035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d', ...}])
tests/test_pay.py:5147: AssertionError
```
We fix this by creating a fresh channel, so it will try.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can see peer_in before the state changes:
```
l3.daemon.wait_for_log("peer_in WIRE_ERROR")
> assert only_one(l3.rpc.listpeerchannels(l2.info['id'])['channels'])['state'] == 'AWAITING_UNILATERAL'
E AssertionError: assert 'CHANNELD_NORMAL' == 'AWAITING_UNILATERAL'
E - AWAITING_UNILATERAL
E + CHANNELD_NORMAL
```
From the logs, there is 0.2 seconds between them:
```
lightningd-3 2024-11-05T01:58:41.695Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_in WIRE_ERROR
lightningd-3 2024-11-05T01:58:41.726Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: billboard perm: Received ERROR channel cecf36a62a09d4f1bdb42aa61d0770964bf3b245b8943a3e5b86dafc572f63d1: Forcibly closed by `close` command timeout
lightningd-3 2024-11-05T01:58:41.745Z UNUSUAL 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Peer permanent failure in CHANNELD_NORMAL: channeld: received ERROR channel cecf36a62a09d4f1bdb42aa61d0770964bf3b245b8943a3e5b86dafc572f63d1: Forcibly closed by `close` command timeout (reason=protocol)
lightningd-3 2024-11-05T01:58:41.890Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: We have 3 anchor points to use
lightningd-3 2024-11-05T01:58:41.897Z DEBUG lightningd: Broadcasting txid ff2f44b37d96a81fcaa2a4b11746a06be70f3f800fbce941e61a47abb61f70c0
lightningd-3 2024-11-05T01:58:41.901Z DEBUG lightningd: sendrawtransaction: 02000000000101cecf36a62a09d4f1bdb42aa61d0770964bf3b245b8943a3e5b86dafc572f63d1000000000096b64c80044a01000000000000220020525df7a97bd0506c9ec41ee4e5f095e6e5316db01846a0a687404628017e88494a010000000000002200206db2ec9041ba3ccb6309dcad26015f32637e6869be09d3c3d3a1cb1439296f0b400d030000000000220020c2468acf761754e2533fcb13235326d1f1173b697b132048d6409be7818ed9a96a1f0c0000000000220020b1b561b95c1bccd50fb21bd417a95cabbc3efc351b1353063fe5e9f185d21c8a0400473044022036ab981a2642527c2019a4d15847f6b2fb1d0b92f1a2faff463ab109ed70c4d002205eeb0fcd610a9dba477ee7d66f02d0384080383fbb003d4ee423bdae0970b0430147304402202f7de3481ce00478e539cd9d6bac95ec3c80ec906c46ad2561118dca321b1458022048efd1aa04ff5fafbb99d14f7c9671072b035cbd2c031547793eb0cb6a62302d0147522102d595ae92b3544c3250fb772f214ad8d4c51425033740a5bcc357190add6d7e7a2102d6063d022691b2490ab454dee73a57c6ff5d308352b461ece69f3c284f2c241252ae6e14d920
lightningd-3 2024-11-05T01:58:41.907Z INFO 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is like `sendonion` but unwraps the onion as the first hop,
avoiding nasty special cases for blinded paths which start with this
node, and also self-pay.
Tests split into multiple ones after Christian's review.
Changelog-Added: JSON-RPC: `injectpaymentonion` for initiating an HTLC like a peer would do.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Removes the `COMPAT_V070` functionality for `listfowards`.
Changelog-Changed: The `listforwards` command will now return a value
of 0 for `received_time` for very old forward attempts.
Changelog-Added: JSON-RPC: `decode` now used modern BOLT 4 language for blinded paths, `first_path_key`.
Changelog-Deprecated: JSON-RPC: `decode` `blinding` in blinded path: use `first_path_key`.
Changelog-Added: Plugins: `onion_message_recv` and `onion_message_recv_secret` hooks now used modern BOLT 4 language for blinded paths, `first_path_key`.
Changelog-Deprecated: JSON-RPC: `onion_message_recv` and `onion_message_recv_secret` hooks `blinding` in blinded path: use `first_path_key`.
No code changes, just catching up with the BOLT changes which rework our
blinded path terminology (for the better!).
Another patch will sweep the rest of our internal names, this tries only to
make things compile and fix up the BOLT quotes.
1. Inside payload: current_blinding_point -> current_path_key
2. Inside update_add_htlc TLV: blinding_point -> blinded_path
3. Inside blinded_path: blinding -> first_path_key
4. Inside onion_message: blinding -> path_key.
5. Inside encrypted_data_tlv: next_blinding_override -> next_path_key_override
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we should support it by default.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `option_quiesce` enabled by default.
Changelog-Deprecated: Config: --experimental-quiesce: it's now the default.
I noticed that increasing mu a little bit sometimes made a big difference,
because by completely ignoring fees we were choosing the worst of two channels
in some cases.
Start at 1% fees; this saves a lot on initial fees in this test!
Here's the new stats on mu levels:
96 mu=1
90 mu=10
41 mu=20
30 mu=30
24 mu=40
19 mu=50
22 mu=60
8 mu=70
95 mu=80
19 mu=90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `askrene` is now better at finding low-fee paths.
While the `k=8` value worked for the current main network tests with the
amounts in those tests, it wasn't robust across a wider range of values
(as demonstrated when other test changes broke tests!).
Time to do this properly: calculate the ratio at the time we combine them,
using median values.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed this in the logs:
plugin-cln-askrene: notify msg unusual: The flows had a fee of 151950msat, greater than max of 53697msat, retrying with mu of 10%...
plugin-cln-askrene: notify msg unusual: The flows had a fee of 220126msat, greater than max of 53697msat, retrying with mu of 20%...
We would expect increasing mu to *reduce* the fee!
Turns out that our linear fee is a bad terrible approximation, because I
was using base_fee_penalty of 10.0.
|
| / __ <- real fee, with base: fee = base + propfee * amount.
| / __/
| _//
| __/
| __/_/
|/ _/
| _/ <- linearized fee: fee = linear * amount
|/
+-----------------------------------
These cross over where linear = propfee + base / amount. Assume we split the
payment into 10 parts, this implies that the base_fee_penalty should be 10 / amount
(this gives a slight penalty to the normal case, but that's ok).
This gives better results, too: we get down to 650099 sats in fees, vs 801613
before.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During "test_real_data", then only successes with reduced fees were 92 on "mu=10", and only
1 on "mu=30": the rest went to mu=100 and failed.
I tried numerous approaches, and in the end, opted for the simplest:
The typical range of probability costs looks likes:
min = 0, max = 924196240, mean = 10509.4, stddev = 1.9e+06
The typical range of linear fee costs looks like:
min = 0, max = 101000000, mean = 81894.6, stddev = 2.6e+06
This implies a k factor of 8 makes the two comparable.
This makes the two numbers comparable, and thus makes "mu" much more
effective. Here are the number of different mu values we succeeded at:
87 mu=0
90 mu=10
42 mu=20
24 mu=30
17 mu=40
19 mu=50
19 mu=60
11 mu=70
95 mu=80
19 mu=90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We ask it again, but reduce fees by 1msat from the previous answer.
This is really nasty, as it frequently exercises the case where we
only go over fee when we do the refinement step.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The fp16_t values are approximations (overestimate for htlc_max,
underestimate for htlc_min), so in the refinement step we should use
the exact values.
This also fixes a logic bug: flow_remaining_capacity returned the
total capacity, not the additional capacity!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `askrene` now honors exact htlc_maximum_msat limits.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `getroutes` now applies `auto.sourcefree` layer in the order specified, so doesn't alter channels changed in later layers.
It was failing because the channel_hint from one attempt would prevent
us from retrying. By changing the amounts so that the channel_hints do
not concern them (value smaller than estimate) we can make things work
as before again.
