I previously mistyped the rather lengthy conditions for failures, so
let's dissect it into its smaller components and add rationale behind
the individual parts of the decision.
This adds a new state `PAYMENT_STEP_RETRY_GETROUTE` which is used to
retry just that one step, without spawning a completely new
attempt. It's a new state so that modifiers do not act on it twice.
Changelog-Fixed: pay: Improved the performance of the `pay` command considerably by avoiding conflicting changes to our local network view.
We were delaying the channel_hint update till after the `createonion`
call which gave us the same situation we had with concurrent
`getroute` calls. Now we update the hints as soon as the plugins have
had their say in the route construction. If we still fail, either
because a modifier changed the route causing the failure, or because
we interleaved the route computation for multiple parts, we reset the
attempt and retry inline (i.e., without creating a new sub-payment).
Notice that interleaved route computations now only happen if the
modifier makes an async call to some RPC or similar.
1. One place returned false instead of -1.
2. The names implied it returned a bool, and it doesn't.
Fix both, and curse C's loose typing a little.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Technically there *are* two feerates that we need to know:
- the feerate to use for the funding transaction, and
- the feerate to tell our peer to use for our commitment txs/htlc txs
As written, `multifundchannel` uses the same feerate for both. This
optional parameter will allow us to differentiate between the two, which
will be exceedingly handy for anchor output worlds. ;)
FIXME: test this
Changelog-Added: JSON API: `multifundchannel` has a new optional argument, 'commitment_feerate', which can be used to differentiate between the funding feerate and the channel's initial commitment feerate
This is a fairly direct translation. Even so, it should be faster in
most cases, and and we can do more sophisticated things if we want.
This also handles disabled channels better.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: plugins: `pay` will now try disabled channels as a last resort.
Finally, extends the 'close_to' functionality up to the flagship 'open a
channel' command.
Changelog-Added: JSON-API `fundchannel` now accepts an optional 'close_to' param, a bitcoin address that the channel funding should be sent to on close. Requires `opt_upfront_shutdownscript`
We're rarely in a hurry here, and bitcoind is aggressive with fees.
You can always spend this output if you really have to, using CPFP.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: mutual closing feerate reduced to "slow" to avoid overpaying.
I got a corrupt file, which looked like multiple concurrent attempts
to build it. So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).
Also move into plugins/ where it logically belongs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Deprecated, but this can happen:
```
==1578== Conditional jump or move depends on uninitialised value(s)
==1578== at 0x12B30E: amount_msat_add (amount.c:224)
==1578== by 0x11270B: add_amount_sent (pay.c:1734)
==1578== by 0x112D89: listsendpays_done (pay.c:1831)
==1578== by 0x114F4B: handle_rpc_reply (libplugin.c:555)
==1578== by 0x115704: rpc_read_response_one (libplugin.c:685)
==1578== by 0x115821: rpc_conn_read_response (libplugin.c:705)
==1578== by 0x148E40: next_plan (io.c:59)
==1578== by 0x1499BD: do_plan (io.c:407)
==1578== by 0x1499FB: io_ready (io.c:417)
==1578== by 0x14BBC1: io_loop (poll.c:445)
==1578== by 0x117A82: plugin_main (libplugin.c:1322)
==1578== by 0x113ABC: main (pay.c:2096)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #3926
(probably)
Changelog-Fixed: pay: Also limit the number of splits if the payee seems to have a low number of channels that can enter it, given the max-concurrent-htlcs limit.
The routehints paymod shares the storage of the array d->routehints and
p->invoice->routes, but once it operates, it possibly leaves it as a stale
pointer to memory it used to have.
Since other paymods may be interested in the invoice details, including
the routehints in the invoice, we should ensure the p->invoice->routes
remains valid whenever we try mutating that array.
As revealed by the failure of tests in #3936, where we ended up trying
to send a partial payment using legacy style, we are not handling
style properly.
1. BOLT9 has features, so we can *know* that the destination supports
MPP. We may not have seen a node_announcement.
2. We can't assume that nodes inside routehints support TLV.
3. We can't assume direct peers support TLV.
The keysend code tried to fix this up, so I'm not sure that this caused
the issue in #3968, though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `pay` will now make reliable multi-part payments to nodes it doesn't have a node_announcement for.
This will update the fee output if it exists, rather than unilaterally
adding a new one.
Also, if the fee output already exists, we should make sure that it
doesn't interfere with the outnums of the other outputs
Fixes: #2679
Changelog-Added: JSON-RPC: New `multiwithdraw` command to batch multiple onchain sends in a single transaction. Note it shuffles inputs and outputs, does not use BIP69.
Changelog-Added: We now have `multifundchannel` as a builtin plugin command to fund multiple channels to different peers all in a single onchain transaction.
Header from folded patch 'fixup-use-json_add_psbt.patch':
fixup!
Header from folded patch 'use-goto-no-ok-chain.patch':
fixup!
Header from folded patch 'destinations-at-parse-time.patch':
fixup!
Header from folded patch 'multifundchannel__use_jsmntoks_to_pass_through_json_string,_not_strings.patch':
multifundchannel: use jsmntoks to pass through json string, not strings
Passing in "" for utxos would crash lightningd on the command-line
otherwise. Now returns an error.
Header from folded patch 'update_plugins-multifundchannel.c.patch':
Update plugins/multifundchannel.c
Co-authored-by: Darosior <darosior@protonmail.com>
Changelog-Deprecated: plugin: `bcli` replacements should note that `sendrawtransaction` now has a second required Boolean argument, `allowhighfees`, which if `true`, means ignore any fee limits and just broadcast the transaction. Use `--deprecated-apis` to use older `bcli` replacement plugins that only support a single argument.
This removes the reservation cleanup at startup, too, now they're all
using 'reserved_til'.
This changes test_withdraw, since it asserted that outputs were marked
spent as soon as we broadcast a transaction: now they're reserved until
it's mined. Similarly, test_addfunds_from_block assumed we'd see funds
as soon as we broadcast the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` now randomizes input and output order, not BIP69.
This is a little lazy, but simpler than extracting the common parts
or making withdraw a plugin which calls txprepare (which should be
deprecated soon in favor of fundpsbt etc).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some minor phrasing differences cause test changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: txprepare reservations stay across restarts: use fundpsbt/reservepsbt/unreservepsbt
Changelog-Removed: txprepare `destination` `satoshi` argument form removed (deprecated v0.7.3)
This uses `fundpsbt` and similar to simulate the txprepare command.
It has one difference (when complete), in that it those reservations
are now timed and don't get reset on restart.
It also doesn't have the restriction that `all` can only be used with
no other output, as I didn't realize that when I implemented it!
Note that change is now inserted in a random position, not sorted
into BIP69 order.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids overwriting the ones in git, and generally makes things neater.
We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were using `tal_fmt` to truncate off the last byte of the
response (newline), which required an allocation, a call to `vsnprintf` and a
copy of the block contents. This being >2MB in size on mainnet was rather
expensive.
We now just signal the end of the string by overwriting the trailing newline,
and stealing directly onto the result.
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS. Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using `waitblockheight 0` is a very slightly faster query than `getinfo`.
Also, avoid querying blockheight for child payments (allow `waitblockheight`
paymod to provide the blockheight returned from the `waitblockheight`, and
just resample the starting blockheight from the parent).
Changelog-None: pointless micro-optimization
This was checked with `gcc -S -O2` to see how an optimized build
would compile the function.
The original code completed calls into each child (and the `.s`
file showed that GCC 9.x was not smart enough to do early-out).
This modification explicitly does early-out, and avoids call-return
stack overhead for the common case where a payment is an ancestor
of a long line of single-child payments due to retrying.
Changelog-None: pointless micro-optimization
This is the simplest possible fix: increase the target amount until we get
the desired number of parts, while still bucketizing payments together that
are in approximately the same size.
The current logic puts all payments that are in the range x < amount <= 16*x
in the same bucket, making them harder to distinguish.
Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
We have sanity checks in there that it's a valid point. Simply store
the JSON token like we do with others.
time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null
Before:
real 0m2.054s
user 0m0.114s
sys 0m0.024s
After:
real 0m1.781s
user 0m0.127s
sys 0m0.013s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null
Before:
real 0m12.447s
user 0m0.143s
sys 0m0.008s
After:
real 0m2.054s
user 0m0.114s
sys 0m0.024s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
memmem is also O(n^2), though it's faster. Now we have infrastructure,
let's do incremental parsing.
time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null
Before:
real 0m13.674s
user 0m0.131s
sys 0m0.024s
After:
real 0m12.447s
user 0m0.143s
sys 0m0.008s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This doesn't make any difference, since lightningd generally sends us
short commands (command responses are via the rpc loop, which is
already done), but it's harmless.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The jsmn parser is a beautiful piece of code. In particular, you can parse
part of a string, then continue where you left off.
We don't take advantage of this, however, meaning for large JSON objects
we parse them multiple times before finally having enough to complete.
Expose the parser state and tokens through the API, so the caller can pass
them in repeatedly. For the moment, every caller is allocates each time
(except the unit tests).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to change the API on the more complete JSON parser, so
make and use a simple API for the easy cases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Tested on a test node which had made 50,000 payment, with no optimization.
For comparison, time for 'listsendpays' was 0.983s.
time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null
Before:
real 0m52.415s
user 0m0.127s
sys 0m0.044s
After:
real 0m42.741s
user 0m0.149s
sys 0m0.016s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: libplugin: significant speedups for reading large JSON replies (e.g. calling listsendpays on large nodes, or listchannels / listnodes).
It's currently always 0, but it won't be once we replace txprepare.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel` has new `outnum` field indicating which output of the transaction funds the channel.
Reported-by: ZmnSCPxj
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
Only advance through routehints if no route was found at all, or if the
estimated capacity at the routehint is lower than the amount that we
have to send through the routehint.
Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
listpays: make doc-all missed
Changelog-Added: JSON-RPC: `listpays` can be used to query payments using the `payment_hash`
Changelog-Added: JSON-RPC: `listpays` now includes the `payment_hash`
Changelog-None: cleanup only.
Before this change:
```
$ ls -l plugins/bcli
-rwxrwxr-x 1 zmnscpxj zmnscpxj 1914976 Aug 5 21:54 plugins/bcli
```
After this change:
```
$ ls -l plugins/bcli
-rwxrwxr-x 1 zmnscpxj zmnscpxj 1830552 Aug 5 22:00 plugins/bcli
```
We already duplicate a lot of code between `lightningd` and eeach
builtin plugin because we do not use .so for `common/`, but including
an object file with code that is not referenced in the executable is
fairly low-hanging size optimization fruit.
Changelog-Fixed: pay: Fixed a bug where routehints would be ignored if the payment exceeded 10,000 satoshi. This is particularly bad if the payee is only reachable via routehints in an invoice.
It's not all that rare to do these operations, and requiring annotations
for it is a little painful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Arguably a low-priority bug since no current node ever generates routehints longer
than one hop.
However, it is possible as an edge case, if the destination is directly accessible
*and* supports multiple channels, that we route through the destination, one of the
*other* channels it has not in the routehint, to the entry point, and then through
the routehint.
This change removes the risk of the above edge case.
Changelog-None: arguably a low-priority bug.
We've had problems with blocksonly in the past and bitcoind still allows
to use outdated (thus potentially dangerous estimates) while running
bitcoin with -blocksonly.
ZmnSCPxj mentionned that we still don't document this, but i figured
that in this specific case an explicit check and error seems preferable.
Changelog-Added: We now explicitly check at startup that our default Bitcoin backend (bitcoind) does relay transactions.
Proposed-by: ZmnSCPxj <zmnscpxj@protonmail.com>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This allows us to kill two birds with one stone: once connected, we use
the output of the successful call to do some sanity checks (only
checking the version for now, but more are yet to come!).
Changelog-Added: We now explicitly check at startup the version of our default Bitcoin backend (bitcoind).
Co-Authored-by: ZmnSCPxj <zmnscpxj@protonmail.com>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
We'd previously take the failed attempt and estimate the failing channel's
capacity at 3/4 of the attempted amount, which is rather aggressive. This
reduces this aggressiveness to use the exact amount tried, but excluding on
equality. This still skips attempting the same route with the same amount, but
also permits attempts that are in the range [3/4, 1] of the failed attempt
amount to still be attempted.
We sum up the amounts just like we do with `amount_sent`, however we may have
parts that don't have that annotation, so make it optional.
Suggested-by: Rusty Russell <@rustyrussell>
While not directly necessary, it still feeds the `listpays` result, and so we
should pass it along if we can, so we don't have to rely solely on the
`amount_sent` field, which includes the fees.
Reported-by: Rusty Russell <@rustyrussell>
The shortcut in the retry_mod that we can skip retrying if getroute fails or
we have no result is only valid if the parameters don't change. As we iterate
through the routehints the parameters change, and so we must signal to the
retry_mod that it can retry even in those cases.
The child payments will sometimes depend on the step of the parent, and making
sure that the parent state is correct before we create the children is
therefore important.
This uses @cdecker's idea of excluding the routehinted channel from the route,
and also consumes the route hints as it goes so that it makes progress.
I don't know if this is correct, but it reliably passes tests/test_pay.py::test_tlv_or_legacy
now.
We store an offset of the current routehint in the modifier data. It gets
incremented on retry, and it gets reset to 0 on split. This is because once we
split we have a different amount and a previously unusable routehint becomes
usable again.
We have two places we need to do that now: in the root payment after we
checked if the destination is reachable, and in any other payment directly in
the initialization-step callback.
It was spread over the step callback, but we only need to initialize the
routehints array there, child-payments can just inherit most of the information.
This does two things: it checks if the destination of the payment is at all
reachable without routehints, and if it is it adds a direct attempt as option
to the routehints in the form of a NULL routehint. It also simplifies the
selection of the routehint since the direct case is no longer special, instead
we just return a NULL routehint as if it were a normal routehint.
The adaptive MPP test was showing an issue with always using a routehint, even
when it wasn't necessary: we would insist on routhing to the entrypoint of the
routehint, even through the actual destination. If a channel on that loop
would result being over capacity we'd slam below 0, and then increase again by
unapplying the route. The solution really is not to insist on routing through
a routehint, so we implement random skipping of routehints, and we rotate them
if we have multiples.
As the hints get new fields added it is easy to forget to amend one of the
places we create them, since we already have an update method let's use that
to handle all additions to the array of known channel hints.
We were removing the current hint from the list and not inheriting the current
routehint, so we'd be forgetting a hint at each retry. Now we keep the array
unchanged in the root, and simply skip the ones that are not usable given the
current information we have about the channels (in the form of channel_hints).
Fixes#3861
This may be related to the issue #3862, however the water was muddied by it
being the wrong error to return, and the node should not expect this courtesy
feature to be present at all...
There is little point in trying to split if the resulting HTLCs exceed the
maximum number of HTLCs we can add to our channels. So abort if a split would
result in more HTLCs than our channels can support.
The presplit modifier could end up exceeding the maximum number of HTLCs we
can add to a channel right out the gate, so we switch to a dynamic presplit if
that is the case. The presplit will now at most use 1/3rd of the available
HTLCs on the channels if the normal split would exceed the number of availabe
HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
It turns out that by aggressively splitting payments we may end up exhausting
the number of HTLCs we can add to a channel quickly. By tracking the number of
HTLCs we can still add, and excluding the channels to which we cannot add any
more we increase the route diversity, and avoid quickly exhausting the HTLC
budget.
In the next commit we'll also implement an early abort if we've exhausted all
channels, so we don't end up splitting indefinitely and we can also optimize
the initial split to not run afoul of that limit.
Changelog-Fixed: plugin: `bcli` no longer logs a harmless warning about being unable to connect to the JSON-RPC interface.
Changelog-Added: plugin: Plugins can opt out of having an RPC connection automatically initialized on startup.
This PR includes the fix discussed on PR #3855. This fix was tested with the use case described inside the issue and worked.
Fixes: #3855
Changelog-None
The code had incorrect assertions, partially because it didn't clearly
distinguish errors from the final node (which, barring blockheight issues,
mean a complete failre) and intermediate nodes.
In particular, we can't trust the *values*, so we need to distinguish
these by the *sender*.
If a route is of length 2 (A, B):
- erring_index == 0 means us complaining about channel A.
- erring_index == 1 means A.node complaining about channel B.
- erring_index == 2 means the final destination node B.node.
This is particularly of note because Travis does NOT run test_pay_routeboost!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were using the current constraints, including any shadow route and other
modifications, when computing the remainder that the second child should
use. Instead we should use the `start_constraints` on the parent payment,
which is a copy of `constraints` created in `payment_start` exactly for this
purpose.
Also added an assert for the invariant on the multiplier.
When using mpp we need to always have partids>0, since we bumped the partid
for the root, but not the next_id we'd end up with partid=1 being
duplicated. Not a big problem since we never ended up sending the root to
lightningd, instead skipping it, but it was confusing me while trying to trace
sub-payment's ancestry.
We skip most payment steps and all sub-payments, so consolidate the skip
conditions in one if-statement. We also not use `payment_set_step` to skip any
modifiers after us after the step change.
We now check against both constraints on the modifier and the payment before
applying either. This "fixes" the assert that was causing the crash in #3851,
but we are still looking for the source of the inconsistency where the
modifier constraints, initialized to 1/4th of the payment, suddenly get more
permissive than the payment itself.
This was highlighted in #3851, so I added an assertion. After the rewrite in
the next commit we would simply skip if any of the constraints were not
maintained, but this serves as the canary in the coalmine, so we don't paper over.
While we were unsetting the `payment->cmd` in case of a success to signal that
we should not return to the JSON-RPC command twice, we were not doing that in
the case of failures. This was causing multiple responses to a single incoming
command, and `lightningd` was correctly killing the plugin. This issue was
introduced through early returns (anything setting `payment->abort=true`) and
was caused in Rusty's case through an MPP timeout.
Fixes#3847
Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
We were rather pedanticly failing the plugin if we were unable to parse the
`waitsendpay` result, but had coded all the modifiers in such a way that they
can handle a `NULL` result (verified in the code and manually by randomly
failing the parsing). So we now just log the result we failed to parse and
merrily go our way.
Worst case is that we end up retrying the same route multiple times, since we
can't blacklist any nodes / channels without understanding the error, but that
is still in the scope of what we must handle anyway.
This modifier splits a payment that has been attempted a number of times (by a
modifier earlier in the mod chain) and has failed consistently. It splits the
amount roughly in half, with a but if random fuzz, and then starts a new round
of attempts for the two smaller amounts.
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
With the `presplit`-modifier we actually skip execution of the root altogether
which results in the root not having a result at all. Instead we should use
the result returned by `payment_collect_result`.
With MPP we require that the sum of parts is equal to the `total_msat` amount
declared in the onion. Since that can't be changed once the first part arrives
we need a way to disable amount fuzzing for MPP.
Technically an API break, but nobody relies on these I hope!
Note that the feerates warning was buried inside the style object:
it should be top-level.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were applying the fee exemption to all payments individually, which is ok
until we switch to MPP, where amounts change. Also the log entry was referring
to the total amount, and not the fee of the payment.
This was causing the state flapping test to fail, since we were yielding
control of the io_loop, waiting for the blockheight to be reached, and not
setting the status beforehand. An interim `paystatus` call would then find a
failed leaf and deduce the entire payment failed. Setting it back to the
previous state keeps the overall payment pending while we wait.
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.
Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.
Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
This makes use of the payment modifier structure to just add the preimage to
the TLV payload for the last hop.
Changelog-Added: JSON-RPC: The `keysend` command allows sending to a node without requiring an invoice first.
It turns out that the `failcodename` doesn't get populated if the `failcode`
isn't a known error from the enum (duh...) so don't fail parsing if it's
missing.
This commit collects the changes required to the tests caused by the changes
to the `pay` and `paystatus` commands. They are also rather good hints as to
what these changes entail.
We want to differentiate a wrong block-height from other failure reasons, such
as an unknown `payment_hash`, so we skip the `waitblockheight` if we're
already at the correct height.
Add/remove the HTLC amount from the channel hints so concurrent getroute calls
have the correct exclusions. This can sometimes underflow, if we're unlucky
and call getroute too closely, but that's not a big issue, it can only result
in a failed MPP attempt too many, nothing fatal, and it'll get corrected based
on the result returned by the failing node.
These are primarily the fee and cltv constraints that we need to keep up to
date in order to give modifiers a correct view of what is and what isn't
allowed.
We could end up with multiple channel hints, which is a bit wasteful. We now
look for existing ones before adding a new one, and if one exists we use the
more restrictive parameters.
Suggested-by: Lisa Neigut <@niftynei>
We're lucky that we can distinguish the severity of the failure based on the
failcode, so we bubble up the one with the maximum failcode, and let callers
inspect details if they need more information.
We can have quite detailed information about our local channels, so call
`listpeers` before the `getroute` call on the root payment, to seed that
information in the channel_hints.
Since we end up consolidating some of the return values for `pay` and
`paystatus` and change the public interface we need to add the compatibility
flag and guard the switchover behind it.
most likely unused since the switch to libwally for internal blockchain
things.
these method names were clashing with ones that are to be introduced
with some libwally cleanups, so getting rid of them pre-emptively keeps
us libwally compatible
We need to keep them around so we can inspect them later. We'll also need a
background cleanup every once in a while to free some memory. More on that in
a future commit.
We were just handwaving the partid generation, which broke some tests that
expected the first payment attempt to always have partid=0, so here we just
track the partids assigned in the payment tree, starting at 0.
The status of what started as a simple JSON-RPC call is now spread across an
entire tree of partial payments and payment attempts. So we collect the status
in a single struct in order to report back success of failure.
This is just for testing for now, TLV payload computation will come next. We
stage all the payloads in deserialized form so modifiers can modify them more
easily and serialize them only before actually calling `createonion`.
This is necessary so we can build the absolute locktimes in the next step. For
now just fetch the blockheight on each (sub-)payment, later we can reuse the
root blockheight if it ends up using too much traffic.
A payment is considered finished if it is in a final state (success or
failure) or all its sub-payments are finished. If that's the case we notify
`payment_finished` and bubble up through `payment_child_finished`, eventually
bubbling up to the root, which can then report success of failure back to the
RPC command that initiated the whole process.
This is likely a bit of overkill for this type of functionality, but it is a
nice first use-case of how functionality can be compartmentalized into
modifiers. If makes swapping retry mechanisms in and out really simple.
This should make it easy for JSON-RPC functions and modifiers to get the
associated data for a given modifier name. Useful if a modifier needs to
access its parent's modifier data, or in other functions that need to access
modifier data, e.g., when passing destination pointers into the `param()`
call.
This commit can be reverted/skipped once we have implemented all the logic and
have feature parity with the normal `pay`. It's main purpose is to expose the
unfinished functionality to test it, without completely breaking the existing
`pay` command.
```
plugins/keysend.c:136:47: error: format specifies type 'unsigned long' but the argument has type 'time_t' (aka 'int') [-Werror,-Wformat]
ki->label = tal_fmt(ki, "keysend-%lu.%09lu", now.ts.tv_sec, now.ts.tv_nsec);
~~~ ^~~~~~~~~~~~~
%d
```
Changelog-None
The autocleaning will only happen if the autocleaninvoice-cycle startup
option is passed, which cannot happen if the plugin is started
post-startup.
Thus, it's less misleading for users to restrict its usage to startup.
Changelog-Added: plugins: The `autoclean` plugin is now static (you cannot manage it with the `plugin` RPC command anymore).
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
We did not take the value of --commit-fee into account : this removes
the unused option from lightningd and instead registers it in bcli,
where we set the actual feerate of commitment transactions. This also
corrects the documentation.
Changelog-Fixed: config: we now take the --commit-fee parameter into account.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Since we now over-write the wally malloc/free functions, we need to do
so for tests as well. Here we pull up all of the common setup/teardown
logic into a separate place, and update the tests that use libwally to
use the new common_setup core
Changelog-None
And the percentage of the initial amount, not the constently increasing
one !
Changelog-Fixed: pay: we now respect maxfeepercent, even for tiny amounts.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
They are already logged both in bcli, and in lightningd.
This just adds a lot of noise to the logs. We keep successed attempts
though for the tests.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
The documentation was wrong, and I copied my mistake to `libplugin` where it
was then ignored instead of ORed into the node's featurebits. This fixes both.
As discussed with Christian, prepending the length to the payload returned
is awkward, but it's the only way to set a legacy payload. As this will
be soon deprecated, simplify the external API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So far we were relying on `lightningd` to create an ad-hoc invoice when
telling it to `resolve` with a given preimage. We now switch to having the
plugin create the invoice, remove the mandatory `keysend_preimage`
field (which would upset `lightningd` otherwise), and then return the modified
payload with the instructions to `continue` instead of resolving.
This ties back in with the existing payment/invoice handling code. Invoices
are created only if we don't have a label clash (unlikely since we have the
nano-time in the label), or the `payment_hash` was already used for another
invoice (at which point `lightningd` will automatically reject the payment and
we're a bit poorer for it, but meh :-)
The generated wrappers will ignore the raw fields and will only consider the
shortcut fields. This function takes the raw fields and serializes them
instead.
This still uses the experimental TLV-type, but once the type is standardized
we can add detection for the new type quite easily.
Changelog-Added: pay: The `keysend` plugin implements the ability to receive spontaneous payments (keysend)
While we removed the `satoshi` param in #3603 it appears that the
`fundchannel` plugin was still passing it to the `fundchannel_start`
call. This fixes up the help text. Notice that technically the help text
changes the param name, but since it was internally always called `amount`
this change doesn't break the API, the help was just wrong.
They now use -fno-common by default, so duplicated variables cause
a link error:
/usr/bin/ld: common/utils.o:(.bss+0x10): multiple definition of `chainparams'; plugins/libplugin.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make: *** [Makefile:408: plugins/autoclean] Error 1
This was introduced in 9ebfdf0b8c.
Fixes: #3597
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Multiple definition of chainparams on Fedora (or other really recent gcc)
The shadow route algorithm is extending the route randomly using channels
adjacent to the current destination, in the hope to create a plausible route
extension. However, instead of only retrieving the channels adjacent to the
destination it was retrieving all channels in the entire topology, and
selecting a random channel from there. This resulted in a very large request
for all channels being processed, and then mostly not being used, but also in
shadow extensions to the path which were not plausible (they didn't extend the
real path, just random edges). This is fixed by restricting the call to
`listchannels` to the channels with the current destination as source.
On my laptop retrieving all channels in the current mainnet takes
approximately 1.2 seconds, and given the geometric series expansion of the 50%
extension probability this indeed would result in an overhead of 1.2 seconds
to the `pay` command. In contrast specifying a source results in an overhead
of ~30ms.
So good news everyone, your pay commands just shaved 1.17 seconds off their
runtime.
Changelog-Changed: pay: Improved the performance of the `pay`-plugin by limiting the `listchannels` when computing the shadow route.
Changelog-Fixed: pay: The `pay`-plugin was generating non-contiguous shadow routes
It's almost always "their_features" and "our_features" respectively, so
make those names clear.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out that unnecessary: all callers can access the feature_set,
so make it much more like a normal primitive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A CONSERVATIVE/3 target for them.
Some noisy changes to the tests as we had to update the estimatesmartfee
mock.
Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
So we can't tell people they should use amount, until v0.8.2 is
released. Another 6 months before we can deprecated the 'satoshi'
field here :(
Fixes: d149ba2f3a
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON: `fundchannel_start` returns `amount` even when deprecated APIs are enabled.
Changelog-Deprecated: JSON: `fundchannel_start` `satoshi` field really deprecated now (use `amount`).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON: `fundchannel` and `fundchannel_start` `satoshi` parameter removed (renamed to `amount` in 0.7.3).
This is a common thing to do, so create a macro.
Unfortunately, it still needs the type arg, because the paramter may
be const, and the return cannot be, and C doesn't have a general
"(-const)" cast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before this patch we used to send `double`s over the wire by just
copying them. This is not portable because the internal represenation
of a `double` is implementation specific.
Instead of this, multiply any floating-point numbers that come from
the outside (e.g. JSONs) by 1 million and round them to integers when
handling them.
* Introduce a new param_millionths() that expects a floating-point
number and returns it multipled by 1000000 as an integer.
* Replace param_double() and param_percent() with param_millionths()
* Previously the riskfactor would be allowed to be negative, which must
have been unintentional. This patch changes that to require a
non-negative number.
Changelog-None
Spark does this, for example:
{"method":"pay","params":["lnbc..."],"id":22}
Which doesn't have a jsonrpc field. The result is that the command
doesn't terminate, there is nothing in the logs, stderr contains
"pay: JSON-RPC message does not contain "jsonrpc" field", and
from then on "Unknown command 'pay'".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We are going to initialize a plugin before its creation, so log as
UNUSUAL instead.
Also, `pay` and `fundchannel` inits are using rpc_delve(), so we need to
io_new_conn() (which sets the socket as non blocking) after calling the
plugin's init.
This is also taken and adapted from lightningd/bitcoind.
The call to 'getblockchaininfo' is replaced by 'echo' as we don't
make use of the result and the former can sometimes be slow (e.g. on
IBD).
Most is taken from lightningd/bitcoind and adapted. This currently
exposes 5 commands:
- `getchaininfo`, currently called at startup to check the network and
whether we are on IBD.
- `getrawblockbyheight`, which basically does the `getblockhash` +
`getblock` trick.
- `getfeerate`
- `sendrawtransaction`
- `getutxout`, used to gather infos about an output and currently used by
`getfilteredblock` in `lightningd/bitcoind`.
We don't take the callback result into account, so it can better be void.
Having a general callback parameter is handy, because for bcli we want
to pass it the struct bcli.
As a separated commit because it was pre-existent (changelog + xfail test).
This also fix a logical problem in lightningd/plugin_control: we were
assuming a plugin started with 'plugin start' but which did not comport
a 'dynamic' entry in its manifest to be dynamic, though it should have
been treated as static.
Changelog-fixed: plugins: Dynamic C plugins can now be managed when lightningd is up
This adds helpers to start and send a jsonrpc request using json_stream
in order to benefit from the helpers.
This then simplifies existing plugins RPC requests by using json_stream
helpers.
This pass to json_stream helpers for commands outputs, but keeps
compatibility with existing plugins which use jout as of now, by not
starting/closing the "result"/"error" objects.
Now we have streams and a global object, we can use them for io_plans.
This makes the logic closer from lightningd/plugin (it has been mostly
stolen from here), and simplifies the main.
This also allows plugins to use io_plans (thanks to the io_loop) for
asynchronous connections.
This commit only handle incoming commands.
Now that we have json_stream in common/, we can move all the related
helpers from lightningd/json to common/json. This way everyone can
benefit of them (including libplugin, the plugins themselves,
potentially lightning-cli), not lightningd alone!
Note that the Makefile of the common/test/ had to be modified, because
the new helpers make use of common/wireaddr... Which turns out to
\#include <lightingd/lightningd.h> ! So we couldnt just include the .c
and add mocks if we redefined some structs (hello run-param).
GCC 10 defaults to `-fno-common`. no longer automatically sharing
global variable definitions, which makes it important to define
them in only one place (otherwise there will be duplicate definition
errors). Add `extern` qualifiers where (I think) is the best place for
them.
Before this patch we used `int` for error codes. The problem with
`int` is that we try to pass it to/from wire and the size of `int` is
not defined by the standard. So a sender with 4-byte `int` would write
4 bytes to the wire and a receiver with 2-byte `int` (for example) would
read just 2 bytes from the wire.
To resolve this:
* Introduce an error code type with a known size:
`typedef s32 errcode_t`.
* Change all error code macros to constants of type `errcode_t`.
Constants also play better with gdb - it would visualize the name of
the constant instead of the numeric value.
* Change all functions that take error codes to take the new type
`errcode_t` instead of `int`.
* Introduce towire / fromwire functions to send / receive the newly added
type `errcode_t` and use it instead of `towire_int()`.
In addition:
* Remove the now unneeded `towire_int()`.
* Replace a hardcoded error code `-2` with a new constant
`INVOICE_EXPIRED_DURING_WAIT` (903).
Changelog-Changed: The waitinvoice command would now return error code 903 to designate that the invoice expired during wait, instead of the previous -2
Changelog-Fixed: Detect a previously non-permanent error (`final_cltv_too_soon`) that has been merged into a permanent error (`incorrect_or_unknown_payment_details`), and retry that failure case in `pay`.
it's that time of year (merry xmas!)
enables the ability to push_msat on fundchannel
Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
This won't usually be visible to the end-user, since the pay plugin doesn't
do multi-part yet (and mpp requires EXPERIMENTAL_FEATURES), but we're ready
once it does.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The pay plugin has been supplying the bolt11 string since 0.7.0, so only
ancient "pay" commands would be omitted by this change.
You can create a no-bolt11 "sendpay" manually, but then you'll find it
in 'listsendpays'.
Changelog-Removed: JSON: `listpays` won't shown payments made via sendpay without a bolt11 string, or before 0.7.0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: RPC: The 'fundchannel' command now tries to connect to the peer before funding the channel, no need to 'connect' before 'fundchannel' anymore !
When doing the random walk through the channel, we now add the fees
(both the base and the proportional one) for that channel in addition to
the cltv delta.
Changelog-Added: Payment amount fuzzing is restored, but through shadow routing.
I was wondering why TAGS was missing some functions, and finally
tracked it down: PRINTF_FMT() confuses etags if it's at the start
of a function, and it ignores the rest of the file.
So we put PRINTF_FMT at the end, but that doesn't work for
*definitions*, only *declarations*. So we remove it from definitions
and add gratuitous declarations in the few static places.1
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently the only source for amount_asset is the value getter on a tx output,
and we don't hand it too far around (mainly ignoring it if it isn't the
chain's main currency). Eventually we could bubble them up to the wallet, use
them to select outputs or actually support assets in the channels.
Since we don't hand them around too widely I thought it was ok for them to be
pass-by-value rather than having to allocate them and pass them around by
reference. They're just 41 bytes currently so the overhead should be ok.
Signed-off-by: Christian Decker <@cdecker>
The fundchannel plugin needs to know how to build a transaction, so we need to
tell it which chainparams to use. Also adds `chainparams` as a global, since
that seems to be the way to do things in plugins.
in order to preserve current behavior, we cap at max if specified 'all';
otherwise we fail since the amount requested is larger than the
channel max capacity
Previously, we'd fail on 'all'. `fundchannel_start` needs an amount
in order to start a funding transaction.
The way that we approach this is to first call `txprepare` with a
placeholder address and the 'all' amount; this will return the maximum
amount available. We then clamp this to the max_funding (currently
hardcoded, in the future we'd want to consult our/the peer's features) and
then use the amount on the output in the prepared transaction as the
funding amount. We then pass this amount to fundchannel_start,
after we've started it successfully we cancel the held placeholder
transaction and prepare a second transaction for the exact amount,
using the funding address that fundchannel_start passed back.
The string cut & paste hack was nasty; make ->failure a json_out
object so we can splice it in properly.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now hand around struct json_out members, rather than using formatted
strings, so plugins need to construct them properly.
There's no automatic conversion between ' and " any more, so those
are eliminated too. pay still uses some manual construction of elements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since this handles escaping for us, this automatically fixes our previous
escaping issued.
Fixes: #2612
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are generalized from our internal implementations.
The main difference is that 'struct json_escaped' is now 'struct
json_escape', so we replace that immediately.
The difference between lightningd's json-writing ringbuffer and the
more generic ccan/json_out is that the latter has a better API and
handles escaping transparently if something slips through (though
it does offer direct accessors so you can mess things up yourself!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Allow timers to be freed in their callback.
2. Clarify in header that we have to terminate our timer with timer_finished()
eventually.
3. We don't currently have plugins with more than one outstanding timer, but
it certainly would be possible, so fix in_timer to be a counter.
Suggested-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I tried to just do gossipd, but it was uncontainable, so this ended up being
a complete sweep.
We didn't get much space saving in gossipd, even though we should save
24 bytes per node.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Node ids are pubkeys, but we only use them as pubkeys for routing and checking
gossip messages. So we're packing and unpacking them constantly, and wasting
some space and time.
This introduces a new type, explicitly the SEC1 compressed encoding
(33 bytes). We ensure its validity when we load from the db, or get it
from JSON. We still use 'struct pubkey' for peer messages, which checks
validity.
Results from 5 runs, min-max(mean +/- stddev):
store_load_msec,vsz_kb,store_rewrite_sec,listnodes_sec,listchannels_sec,routing_sec,peer_write_all_sec
39475-39572(39518+/-36),2880732,41.150000-41.390000(41.298+/-0.085),2.260000-2.550000(2.336+/-0.11),44.390000-65.150000(58.648+/-7.5),32.740000-33.020000(32.89+/-0.093),44.130000-45.090000(44.566+/-0.32)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
*best is checked for null before the comparison against the uninitialized
variable ever happens, so this isn't a real issue.
Initialize it to zero so that we don't fail to compile on certain gcc versions.
plugins/pay.c: In function ‘add_shadow_route’:
plugins/pay.c:644:18: error: ‘sample’ may be used uninitialized in this function
if (!best || v > sample) {
~~^~~~~~~~
Signed-off-by: William Casarin <jb55@jb55.com>
New name is less confusing, and most people should be transitioning to
listpays rather than this anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is to future-proof against multi-part-payments: the low-level commands
will start returning multiple results once we have that, so prepare
transition plan now.
Closes: #2372
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the same deprecation, but one level up. For the moment, we
still support invoices with a `h` field (where description will be
necessary) but that will be removed once this option is removed.
Note that I just changed pylightning without backwards compatibility,
since the field was unlikely to be used, but we could do something
more complex here?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to do it in various places, but we shouldn't do it lightly:
the primitives are there to help us get overflow handling correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is particularly interesting because we handle overflow during route
calculation now; this could happen in theory once we wumbo.
It fixes a thinko when we print out routehints, too: we want to print
them out literally, not print out the effect they have on fees (which
is in the route, which we also print).
This ABI change doesn't need a CHANGELOG, since paystatus is new since
release.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current param_sat accepts "any": rename and move that to invoice.c
where it's called. We rename it to param_msat_or_any and invoice.c
is our first (trivial) amount_msat user.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to still accept it when parsing the database, but this flag
should allow upgrade testing for devs building on top
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Up until now, riskfactor was useless due to implementation bugs, and
also the default setting is wrong (too low to have an effect on
reasonable payment scenarios).
Let's simplify the definition (by assuming that P(failure) of a node
is 1), to make it a simple percentage. I examined the current network
fees to see what would work, and under this definition, a default of
10 seems reasonable (equivalent to 1000 under the old definition).
It is *this* change which finally fixes our test case! The riskfactor
is now 40msat (1500000 * 14 * 10 / 5259600 = 39.9), comparable with
worst-case fuzz is 50msat (1001 * 0.05 = 50).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the direct cause of the failure of the original
test_pay_direct test and it makes sense: invoice routehints may not be
necessary, so try without them *first* rather than last.
We didn't mention the use of routehints in CHANGELOG at all yet, so
do that now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Christian points out that we can iterate by ->size rather than calling
json_next() to find the end (which traverses the entire object!).
Now ->size is reliable (since previous patch), this is OK.
Reported-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Avoid the unnecessary extra var, and don't use "capacity" since
that usually refers to static capacity.
Reported-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So add a new 'strategy' field. This makes it clearer what is going
on, currently one of:
* "Initial attempt"
* "Excluded channel <scid>"
* "Removed route hint"
* "Excluded expensive channel <scid>"
* "Excluded delaying channel <scid>"
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We sanitize the routes: firstly, we assume appending so eliminate the
first hop if the route points straight to us. Secondly, eliminate empty
hints. Thirdly, trim overlong hints.
Then we just use the first route hint.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use the 'exclude' option to getroute for successive attempts. This
is more robust than having gossipd disable for some limited time.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I wrote this sync first, then rewrote async, then developed libplugin.
But committing all that just wastes reviewer time, so I present it as
if it was always asnc and using the library helper.
Currently the command it registers is 'pay2', but when it's complete
we'll remove the internal 'pay' and rename it. This does a single
'getroute/sendpay' call. No retries, no options.
Shockingly, this by itself is almost sufficient to pass our current test
suite with `pay`->`pay2`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
