The spec has moved a bit here: the `outgoing_cltv_value` in the final onion
is basically the blockheight now (plus the 1 block delta we give ourselves).
Also, we were doubling ours, since p->min_final_cltv_expiry was already set
to p->blindedpay->cltv_expiry_delta above.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't actually support it yet, but this threads through the type change,
puts it in "decode" etc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The `chainlag` is defined as the positive difference between the
height of the last block processed by the node and the best height
known by the bitcoin backend. The chainlag is positive when we are
still catching up with the blockchain, and `0` otherwise.
The `chainlag` is used as an additional offset to the CLTV values when
sending payments, allowing payments to be sent even before the chain
sync completes.
Changelog-Fixed: Plugins: pay now correctly estimates channel capacity
after payment failure, fixing some retry cases.
The `estimated_capacity` was properly substracted from the channel
hint, but adding the amount back did not work. The amount was added back
and then immediately substracted again. This caused the
`estimated_capacity` to ever decrease. This commit makes sure re-adding
the amount to the `estimated_capacity` works as expected.
It's a u64, we should pass by copy. This is a big sweeping change,
but mainly mechanical (change one, compile, fix breakage, repeat).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This has the benefit of being shorter, as well as more reliable (you
will get a link error if we can't print it, not a runtime one!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that run-route-infloop chose some worse-looking paths after
routing was fixed, eg the second node:
Before:
Destination node, success, probability, hops, fees, cltv, scid...
02b3aa1e4ed31be83cca4bd367b2c01e39502cb25e282a9b4520ad376a1ba0a01a,1,0.991856,2,1004,40,2572260x39x0/1,2131897x45x0/0
After:
Destination node, success, probability, hops, fees, cltv, scid...
02b3aa1e4ed31be83cca4bd367b2c01e39502cb25e282a9b4520ad376a1ba0a01a,1,0.954540,3,1046,46,2570715x21x0/1,2346882x26x14/1,2131897x45x0/0
This is because although the final costs don't reflect it, routing was taking
into account local channels, and 2572260x39x0/1 has a base fee of 2970.
There's an easy fix: when we the pay plugin creates localmods for our
gossip graph, add all local channels with delay and fees equal to 0.
We do the same thing in our unit test. This improves things across
the board:
Linear success probability (when found): min-max(mean +/- stddev)
Before: 0.487040-0.999543(0.952548+/-0.075)
After: 0.486985-0.999750(0.975978+/-0.053)
Hops:
Before: 1-5(2.98374+/-0.77)
After: 1-5(2.09593+/-0.63)
Fees:
Before: 0-50848(922.457+/-2.7e+03)
After: 0-50041(861.621+/-2.7e+03)
Delay (blocks):
Before: 0-196(65.8081+/-60)
After: 0-190(60.3285+/-60)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `pay` route algorithm doesn't bias against our own "expensive" channels any more.
The "path_score" callback was supposed to evaluate the *entire path*,
but that was counter-intuitive and opened the door to a cost function
bug which caused this path cost to be less than the closer path.
In particular, the capacity bias code didn't understand this at all.
1. Rename the function to `channel_score` and remove the "distance"
parameter (always "1" since you're supposed to be evaluating a
single hop).
2. Rename "cost" to the more specific "fee": "score" is our
actual cost function result (we avoid the word "cost" as it
may get confused with satoshi amounts).
3. For capacity biassing, we do want to know the amount, but
explicitly hand that as a separate parameter "total".
4. Fix a minor bug where total handed to scoring function previously
included channel fee (this is wrong: fee is paid before sending into
channel).
5. Remove the now-unused total_delay member from the dijkstra
struct.
Here are the results of our test now (routing 4194303 msat, which
didn't crash the old code, so we could compare). In both cases
we could find routes to 615 nodes:
Linear success probability (when found): min-max(mean +/- stddev)
Before: 0.484764-0.999750(0.9781+/-0.049)
After: 0.487040-0.999543(0.952548+/-0.075)
Hops:
Before: 1-5(2.13821+/-0.66)
After: 1-5(2.98374+/-0.77)
Fees:
Before: 0-50041(2173.75+/-5.3e+03)
After: 0-50848(922.457+/-2.7e+03)
Delay (blocks):
Before: 0-294(83.1642+/-68)
After: 0-196(65.8081+/-60)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: https://github.com/ElementsProject/lightning/issues/7092
Changelog-Fixed: Plugins: `pay` would occasionally crash on routing.
Changelog-Fixed: Plugins: `pay` route algorithm fixed and refined to balance fees and capacity far better.
It gave 0. A lot.
Firstly, rmsat was often very small, because delays are often small. Much
smaller than the actual fee.
We really just want to offset the bias and multiply it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We attempted to introduce a "capacity bias" so we would penalize
channels where we were using a large amount of their total capacity.
Unfortunately, this was both naive, and applied wrongly:
-log((capmsat + 1 - amtmsat) / (capmsat + 1));
As an integer gives 0 up to about 65% capacity. We then applied this
by multiplying:
(cmsat * rmsat * bias) / (cmsat + rmsat + bias + 1);
Giving a total score of 0 in these cases! If we're using the
arithmetic mean we really need to add 1 to bias. We might as well
use a double the whole way through, for a slightly more fine-grained
approach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So far we would call `preapproveinvoice` once for each payment split,
i.e., at least once per HTLC, and potentially more often. There is no
point in doing so repeatedly, and especially in remote signer setup
this is actually counterproductive due to the additional roundtrips.
Changelog-Changed pay: Improved performance by removing repeated `preapproveinvoice` calls
CI revealed one:
```
cc plugins/libplugin-pay.c
plugins/libplugin-pay.c: In function ‘payment_getroute’:
plugins/libplugin-pay.c:888:17: error: ‘errstr’ may be used uninitialized [-Werror=maybe-uninitialized]
888 | payment_fail(p, "%s", errstr);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
plugins/libplugin-pay.c:851:21: note: ‘errstr’ was declared here
851 | const char *errstr;
| ^~~~~~
cc1: all warnings being treated as errors
```
My local compiler gave another:
```
channeld/channeld.c: In function ‘resume_splice_negotiation’:
channeld/channeld.c:3734:23: error: ‘final_tx’ may be used uninitialized [-Werror=maybe-uninitialized]
3734 | msg = towire_channeld_splice_confirmed_signed(tmpctx, final_tx,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3735 | chan_output_index);
| ~~~~~~~~~~~~~~~~~~
channeld/channeld.c:3461:28: note: ‘final_tx’ was declared here
3461 | struct bitcoin_tx *final_tx;
| ^~~~~~~~
cc1: all warnings being treated as errors
make: *** [Makefile:298: channeld/channeld.o] Error 1
```
So fix both.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As side-effect, getroute(0) is special too.
Reported-by: MiddleW4y in Discord
Fixes: #6577
Changelog-Fixed: `pay` will still use an invoice routehint if path to it doesn't take 1-msat payments.
Update gossip routiens and various other hecks on the channel state to consider AWAITING_SPLICE to be routable and treated similar to CHANNELD_NORMAL.
Small updates to psbt interface
Changelog-None
The alias may not be set for non-alias channels after they
confirm. The other branch is safe because we only consider active
channels.
Changelog-None
Fixes#6450
This is the simplest solution, not the best, but there's significant risk in try to remove the "we have a path" assumption in the code pay code.
Includes removing a `tal_steal` which was incorrect: the buffer has the same lifetime as the plugin, so if we steal it then things get messy when we free the struct payment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `pay` will now pay your own invoices if you try.
Debugging a number of payments showed that we sometimes waste a number
of attempts routing through a channel via its alias, rather than its
scid. This is because while we annotate the scid when it has been set,
we do not do so for the alias. The alias then is picked for routing
despite not having enough capacity, failing the attempt locally.
It can also happen that we alternate between scid and alias, doubling
the number of failed attempts before we can make progress.
This patch sets the hint for the alias to a capacity of 0 and disables
it as if the peer were offline. This means when available we'll always
use the scid, which is also far easier to read in the logs.
Changelog-Fixed: pay: We now track spendable amounts when routing on both the local alias as well as the short channel ID
Otherwise we later copy the uninitialized memory to descendants,
triggering undefined behavior:
plugins/libplugin-pay.c:2882:34: runtime error: load of value 190, which is not a valid value for type 'bool'
"BOLT 4: Remove legacy format, make var_onion_optin compulsory."
This also renamed the redundant "tlv_payload" to "payload", so we
replace "tlv_tlv_payload" with "tlv_payload" everyhere!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With the warning that we were trying to put "inf" into a u64, we can
see that this calculation was wrong to use integers!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The htlc_budget only exists iff the hint is a 'local' one; we were
failing to write to the htlc_budget field for non-local cases.
To avoid this, we make `local` into a struct that contains the fields
that pertain to local-only payments (in this case, `htlc_budget`).
Valgrind error file: valgrind-errors.1813487
==1813487== Conditional jump or move depends on uninitialised value(s)
==1813487== at 0x4A9C958: __vfprintf_internal (vfprintf-internal.c:1687)
==1813487== by 0x4AB0F99: __vsnprintf_internal (vsnprintf.c:114)
==1813487== by 0x1D2EF9: do_vfmt (str.c:66)
==1813487== by 0x1D3006: tal_vfmt_ (str.c:92)
==1813487== by 0x11A60A: paymod_log (libplugin-pay.c:167)
==1813487== by 0x11B749: payment_chanhints_apply_route (libplugin-pay.c:534)
==1813487== by 0x11EB36: payment_compute_onion_payloads (libplugin-pay.c:1707)
==1813487== by 0x12000F: payment_continue (libplugin-pay.c:2135)
==1813487== by 0x1245B9: adaptive_splitter_cb (libplugin-pay.c:3800)
==1813487== by 0x11FFB6: payment_continue (libplugin-pay.c:2123)
==1813487== by 0x1206BC: retry_step_cb (libplugin-pay.c:2301)
==1813487== by 0x11FFB6: payment_continue (libplugin-pay.c:2123)
==1813487==
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:__vfprintf_internal
fun:__vsnprintf_internal
fun:do_vfmt
fun:tal_vfmt_
fun:paymod_log
fun:payment_chanhints_apply_route
fun:payment_compute_onion_payloads
fun:payment_continue
fun:adaptive_splitter_cb
fun:payment_continue
fun:retry_step_cb
[sesh] 0:[tmux]*Z
Suggested-By: @nothingmuch
If we only specify the node_id, we get the "first" channel.
Closes: #5803
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` uses the correct local channel for payments when there are multiple available (not just always the first!)
Valgrind correctly reports it as uninitialized for this log message, and
the only way this can happen is channel_hints_update() when we receive a
temporary_channel_failure. Put a dummy value here in this case.
```
Valgrind error file: valgrind-errors.23404
==23404== Conditional jump or move depends on uninitialised value(s)
==23404== at 0x49E4B56: __vfprintf_internal (vfprintf-internal.c:1516)
==23404== by 0x49F6519: __vsnprintf_internal (vsnprintf.c:114)
==23404== by 0x1EBCEB: do_vfmt (str.c:66)
==23404== by 0x1EBDF8: tal_vfmt_ (str.c:92)
==23404== by 0x11A336: paymod_log (libplugin-pay.c:167)
==23404== by 0x11B4B2: payment_chanhints_apply_route (libplugin-pay.c:534)
==23404== by 0x11E999: payment_compute_onion_payloads (libplugin-pay.c:1707)
==23404== by 0x11FF4C: payment_continue (libplugin-pay.c:2135)
==23404== by 0x1245C0: adaptive_splitter_cb (libplugin-pay.c:3800)
==23404== by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404== by 0x1205FE: retry_step_cb (libplugin-pay.c:2301)
==23404== by 0x11FEF3: payment_continue (libplugin-pay.c:2123)
==23404==
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-route-blinding" BOLTVERSION=guilt/offers
```
Other than textual changes, this does:
1. Ensures we put total_amount_msat in onion final hop (reported by @t-bast).
2. Require that they put total_amount_msat in onion final hop.
3. Return `invalid_onion_blinding` exactly as defined by the spec (i.e. less
aggressive when we're the final hop) (also reported by @t-bast, but I already
knew).
See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (total_amount_sat required, Eclair compat)
Don't parse the listpeers.channels output ourselves: with two extra fields
we can simply reuse json_to_listpeers_channels().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of returning a peers -> channels heirarchy, return (as callers
want!) a flat array of channels.
This is actually most of the transition work to make them work with
listpeerchannels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We no longer use offers for "I want to send you money", but we'll use
invoice_requests directly. Create a new table for them, and
associated functions.
The "localofferid" for "pay" and "sendpay" is now "localinvreqid".
This is an experimental-only option, so document the change under
experimental only.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `pay` and `sendpay` `localofferid` is now `localinvreqid`.
We simply take the first one, and route to the start of that. Then we
append the blinded path to the onion construction.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It means we consume the channel completely to the best of our
knowledge, so let that through.
Changelog-Fixed: pay: Squeezed out the last `msat` from our local view of the network
In the case of the local channel we set the estimation to the exact
value spendable, which is important when we want to drain a channel,
because there we actually want to get the last msat.
