mirrors/core-lightning
1
0
Fork 0
mirror of https://github.com/ElementsProject/lightning.git synced 2024-11-19 18:11:28 +01:00
Code Issues Projects Releases Wiki Activity
4a9c162211
core-lightning/plugins/libplugin-pay.c

3126 lines
98 KiB
C
Raw Normal View History

paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
#include <bitcoin/preimage.h>
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
#include <ccan/array_size/array_size.h>
#include <ccan/tal/str/str.h>
#include <common/json_stream.h>
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
#include <common/pseudorand.h>
plugins/Makefile: Change plugins build procedure. Changelog-None
2020-06-19 08:22:25 +02:00
#include <common/type_to_string.h>
#include <plugins/libplugin-pay.h>
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
paymod: Add the final cltv delta to the getroute call
2020-05-21 23:24:30 +02:00
#define DEFAULT_FINAL_CLTV_DELTA 9
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
struct payment *payment_new(tal_t *ctx, struct command *cmd,
struct payment *parent,
struct payment_modifier **mods)
{
struct payment *p = tal(ctx, struct payment);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
static u64 next_id = 0;
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
p->children = tal_arr(p, struct payment *, 0);
p->parent = parent;
p->modifiers = mods;
p->cmd = cmd;
p->start_time = time_now();
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
p->result = NULL;
paymod: Add reason why a payment was attempted This is a slight change in interface in that the string no longer spells out that a specific node was excluded.
2020-05-29 17:08:20 +02:00
p->why = NULL;
paymod: Wrap the getroute request in a struct Keeping the arguments in their non-serialized form allows us to amend them as we apply modifiers and learn new information.
2020-06-01 14:57:37 +02:00
p->getroute = tal(p, struct getroute_request);
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
p->label = NULL;
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
p->failreason = NULL;
paymod: Expose riskfactor and wire through to getroute
2020-06-11 15:53:55 +02:00
p->getroute->riskfactorppm = 10000000;
paymod: Set the STOPPED_RETRYING status code if we stopped retrying
2020-07-06 22:39:04 +02:00
p->abort = false;
paymod: Always initialize p->route We're using it in a couple of places to see if we even performed the attempt, so we need to make sure it's initialized.
2020-07-22 17:47:59 +02:00
p->route = NULL;
plugins/pay: Exclude the entrypoint to a routehint to avoid cycles This uses @cdecker's idea of excluding the routehinted channel from the route, and also consumes the route hints as it goes so that it makes progress. I don't know if this is correct, but it reliably passes tests/test_pay.py::test_tlv_or_legacy now.
2020-07-23 06:50:10 +02:00
p->temp_exclusion = NULL;
paymod: Routehintmod signals that we can retry if getroute fails The shortcut in the retry_mod that we can skip retrying if getroute fails or we have no result is only valid if the parameters don't change. As we iterate through the routehints the parameters change, and so we must signal to the retry_mod that it can retry even in those cases.
2020-07-23 16:07:39 +02:00
p->failroute_retry = false;
plugins/pay: hand bolt11 arg to sendonion if we have one (i.e. for `pay`) [ Extracted into standalone patch and comment added by RR ]
2020-07-26 03:13:56 +02:00
p->bolt11 = NULL;
plugins/libplugin-pay.c: Store the route description, and re-report on failure.
2020-08-25 03:54:52 +02:00
p->routetxt = NULL;
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
/* Copy over the relevant pieces of information. */
if (parent != NULL) {
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
assert(cmd == NULL);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
tal_arr_expand(&parent->children, p);
paymod: Wrap the getroute request in a struct Keeping the arguments in their non-serialized form allows us to amend them as we apply modifiers and learn new information.
2020-06-01 14:57:37 +02:00
p->destination = parent->destination;
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
p->amount = parent->amount;
pay: Inherit payment label to all children
2020-08-11 16:41:50 +02:00
p->label = parent->label;
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
p->payment_hash = parent->payment_hash;
paymod: Use reasonable partids We were just handwaving the partid generation, which broke some tests that expected the first payment attempt to always have partid=0, so here we just track the partids assigned in the payment tree, starting at 0.
2020-05-18 17:46:50 +02:00
p->partid = payment_root(p->parent)->next_partid++;
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
p->plugin = parent->plugin;
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
/* Re-establish the unmodified constraints for our sub-payment. */
p->constraints = *parent->start_constraints;
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
p->deadline = parent->deadline;
mpp: Detect if destination supports MPP from invoice and abort early We abort on the root since that is the coordination point for all parts of the payment.
2020-07-10 15:21:47 +02:00
p->invoice = parent->invoice;
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
p->id = parent->id;
plugins/libplugin-pay.c: Propagate local_id from parent to child payment object.
2020-08-25 06:38:23 +02:00
p->local_id = parent->local_id;
paymod: Use reasonable partids We were just handwaving the partid generation, which broke some tests that expected the first payment attempt to always have partid=0, so here we just track the partids assigned in the payment tree, starting at 0.
2020-05-18 17:46:50 +02:00
} else {
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
assert(cmd != NULL);
paymod: Use reasonable partids We were just handwaving the partid generation, which broke some tests that expected the first payment attempt to always have partid=0, so here we just track the partids assigned in the payment tree, starting at 0.
2020-05-18 17:46:50 +02:00
p->partid = 0;
p->next_partid = 1;
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
p->plugin = cmd->plugin;
paymod: Add a local_channel_hints modifier to collect local channels We can have quite detailed information about our local channels, so call `listpeers` before the `getroute` call on the root payment, to seed that information in the channel_hints.
2020-05-22 17:20:36 +02:00
p->channel_hints = tal_arr(p, struct channel_hint, 0);
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
p->excluded_nodes = tal_arr(p, struct node_id, 0);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
p->id = next_id++;
plugins/libplugin-pay.c: Propagate local_id from parent to child payment object.
2020-08-25 06:38:23 +02:00
/* Caller must set this. */
p->local_id = NULL;
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
}
/* Initialize all modifier data so we can point to the fields when
* wiring into the param() call in a JSON-RPC handler. The callback
* can also just `memcpy` the parent if this outside access is not
* required. */
p->modifier_data = tal_arr(p, void *, 0);
for (size_t i=0; mods[i] != NULL; i++) {
if (mods[i]->data_init != NULL)
tal_arr_expand(&p->modifier_data,
mods[i]->data_init(p));
else
tal_arr_expand(&p->modifier_data, NULL);
}
return p;
}
paymod: Use reasonable partids We were just handwaving the partid generation, which broke some tests that expected the first payment attempt to always have partid=0, so here we just track the partids assigned in the payment tree, starting at 0.
2020-05-18 17:46:50 +02:00
struct payment *payment_root(struct payment *p)
{
if (p->parent == NULL)
return p;
else
return payment_root(p->parent);
}
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
static void
paymod_log_header(struct payment *p, const char **type, u64 *id)
{
struct payment *root = payment_root(p);
/* We prefer to show the command ID here since it is also known
* by `lightningd`, so in theory it can be used to correlate
* debugging logs between the main `lightningd` and whatever
* plugin is using the paymod system.
* We only fall back to a unique id per root payment if there
* is no command with an id associated with this payment.
*/
if (root->cmd && root->cmd->id) {
*type = "cmd";
*id = *root->cmd->id;
} else {
*type = "id";
*id = root->id;
}
}
static void
paymod_log(struct payment *p, enum log_level l, const char *fmt, ...)
{
const char *type;
u64 id;
char *txt;
va_list ap;
va_start(ap, fmt);
txt = tal_vfmt(tmpctx, fmt, ap);
va_end(ap);
paymod_log_header(p, &type, &id);
plugin_log(p->plugin, l, "%s %"PRIu64" partid %"PRIu32": %s",
type, id, p->partid, txt);
}
static void
paymod_err(struct payment *p, const char *fmt, ...)
{
const char *type;
u64 id;
char *txt;
va_list ap;
va_start(ap, fmt);
txt = tal_vfmt(tmpctx, fmt, ap);
va_end(ap);
paymod_log_header(p, &type, &id);
plugin_err(p->plugin, "%s %"PRIu64" partid %"PRIu32": %s",
type, id, p->partid, txt);
}
paymod: Add a `getinfo` call on payment_start to get the blockheight This is necessary so we can build the absolute locktimes in the next step. For now just fetch the blockheight on each (sub-)payment, later we can reuse the root blockheight if it ends up using too much traffic.
2020-05-13 17:44:31 +02:00
/* Generic handler for RPC failures that should end up failing the payment. */
static struct command_result *payment_rpc_failure(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
payment_fail(p,
"Failing a partial payment due to a failed RPC call: %.*s",
toks->end - toks->start, buffer + toks->start);
paymod: Add a `getinfo` call on payment_start to get the blockheight This is necessary so we can build the absolute locktimes in the next step. For now just fetch the blockheight on each (sub-)payment, later we can reuse the root blockheight if it ends up using too much traffic.
2020-05-13 17:44:31 +02:00
return command_still_pending(cmd);
}
paymod: Make payment_tree_result public to users can query state
2020-05-27 14:48:25 +02:00
struct payment_tree_result payment_collect_result(struct payment *p)
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
{
struct payment_tree_result res;
size_t numchildren = tal_count(p->children);
res.sent = AMOUNT_MSAT(0);
paymod: Count all attempts, not just the ones with a result With the presplitter in particular we would have n attempts but the array contains n+1 entries, which is kinda weird.
2020-07-19 19:52:18 +02:00
res.attempts = 1;
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
res.treestates = p->step;
res.leafstates = 0;
res.preimage = NULL;
paymod Collect the final failure when aggregating the tree results We're lucky that we can distinguish the severity of the failure based on the failcode, so we bubble up the one with the maximum failcode, and let callers inspect details if they need more information.
2020-05-28 12:25:58 +02:00
res.failure = NULL;
if (p->step == PAYMENT_STEP_FAILED && p->result != NULL)
res.failure = p->result;
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
if (numchildren == 0) {
res.leafstates |= p->step;
if (p->result && p->result->state == PAYMENT_COMPLETE) {
res.sent = p->result->amount_sent;
res.preimage = p->result->payment_preimage;
}
}
for (size_t i = 0; i < numchildren; i++) {
struct payment_tree_result cres =
payment_collect_result(p->children[i]);
/* Some of our subpayments have succeeded, aggregate how much
* we sent in total. */
if (!amount_msat_add(&res.sent, res.sent, cres.sent))
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_err(
p,
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
"Number overflow summing partial payments: %s + %s",
type_to_string(tmpctx, struct amount_msat,
&res.sent),
type_to_string(tmpctx, struct amount_msat,
&cres.sent));
/* Bubble up the first preimage we see. */
if (res.preimage == NULL && cres.preimage != NULL)
res.preimage = cres.preimage;
res.leafstates |= cres.leafstates;
res.treestates |= cres.treestates;
res.attempts += cres.attempts;
paymod Collect the final failure when aggregating the tree results We're lucky that we can distinguish the severity of the failure based on the failcode, so we bubble up the one with the maximum failcode, and let callers inspect details if they need more information.
2020-05-28 12:25:58 +02:00
/* We bubble the failure result with the highest failcode up
* to the root. */
if (res.failure == NULL ||
(cres.failure != NULL &&
cres.failure->failcode > res.failure->failcode)) {
res.failure = cres.failure;
}
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
}
return res;
}
paymod: Add a `getinfo` call on payment_start to get the blockheight This is necessary so we can build the absolute locktimes in the next step. For now just fetch the blockheight on each (sub-)payment, later we can reuse the root blockheight if it ends up using too much traffic.
2020-05-13 17:44:31 +02:00
static struct command_result *payment_getinfo_success(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
const jsmntok_t *blockheighttok =
json_get_member(buffer, toks, "blockheight");
json_to_number(buffer, blockheighttok, &p->start_block);
payment_continue(p);
return command_still_pending(cmd);
}
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
void payment_start(struct payment *p)
{
paymod: Wrap the getroute request in a struct Keeping the arguments in their non-serialized form allows us to amend them as we apply modifiers and learn new information.
2020-06-01 14:57:37 +02:00
struct payment *root = payment_root(p);
plugins/libplugin-pay.c: Propagate local_id from parent to child payment object.
2020-08-25 06:38:23 +02:00
/* Should have been set in root payment, or propagated from root
* payment to all child payments. */
assert(p->local_id);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
p->step = PAYMENT_STEP_INITIALIZED;
p->current_modifier = -1;
paymod: Add a `getinfo` call on payment_start to get the blockheight This is necessary so we can build the absolute locktimes in the next step. For now just fetch the blockheight on each (sub-)payment, later we can reuse the root blockheight if it ends up using too much traffic.
2020-05-13 17:44:31 +02:00
paymod: Wrap the getroute request in a struct Keeping the arguments in their non-serialized form allows us to amend them as we apply modifiers and learn new information.
2020-06-01 14:57:37 +02:00
/* Pre-generate the getroute request, so modifiers can have their say,
* before we actually call `getroute` */
p->getroute->destination = p->destination;
p->getroute->max_hops = ROUTING_MAX_HOPS;
if (root->invoice != NULL && root->invoice->min_final_cltv_expiry != 0)
p->getroute->cltv = root->invoice->min_final_cltv_expiry;
else
p->getroute->cltv = DEFAULT_FINAL_CLTV_DELTA;
p->getroute->amount = p->amount;
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
p->start_constraints = tal_dup(p, struct payment_constraints, &p->constraints);
paymod: Add a `getinfo` call on payment_start to get the blockheight This is necessary so we can build the absolute locktimes in the next step. For now just fetch the blockheight on each (sub-)payment, later we can reuse the root blockheight if it ends up using too much traffic.
2020-05-13 17:44:31 +02:00
/* TODO If this is not the root, we can actually skip the getinfo call
* and just reuse the parent's value. */
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
send_outreq(p->plugin,
jsonrpc_request_start(p->plugin, NULL, "getinfo",
paymod: Add a `getinfo` call on payment_start to get the blockheight This is necessary so we can build the absolute locktimes in the next step. For now just fetch the blockheight on each (sub-)payment, later we can reuse the root blockheight if it ends up using too much traffic.
2020-05-13 17:44:31 +02:00
payment_getinfo_success,
payment_rpc_failure, p));
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
}
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
static void channel_hints_update(struct payment *p,
const struct short_channel_id scid,
int direction, bool enabled, bool local,
pay: Be less aggressive when estimating channel capacity We'd previously take the failed attempt and estimate the failing channel's capacity at 3/4 of the attempted amount, which is rather aggressive. This reduces this aggressiveness to use the exact amount tried, but excluding on equality. This still skips attempting the same route with the same amount, but also permits attempts that are in the range [3/4, 1] of the failed attempt amount to still be attempted.
2020-07-27 17:42:54 +02:00
const struct amount_msat *estimated_capacity,
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
u16 *htlc_budget)
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
{
struct payment *root = payment_root(p);
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
struct channel_hint hint;
/* If the channel is marked as enabled it must have an estimate. */
assert(!enabled || estimated_capacity != NULL);
/* Try and look for an existing hint: */
for (size_t i=0; i<tal_count(root->channel_hints); i++) {
struct channel_hint *hint = &root->channel_hints[i];
if (short_channel_id_eq(&hint->scid.scid, &scid) &&
hint->scid.dir == direction) {
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
bool modified = false;
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
/* Prefer to disable a channel. */
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
if (!enabled && hint->enabled) {
hint->enabled = false;
modified = true;
}
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
/* Prefer the more conservative estimate. */
if (estimated_capacity != NULL &&
amount_msat_greater(hint->estimated_capacity,
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
*estimated_capacity)) {
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
hint->estimated_capacity = *estimated_capacity;
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
modified = true;
}
if (htlc_budget != NULL && *htlc_budget < hint->htlc_budget) {
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
hint->htlc_budget = *htlc_budget;
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
modified = true;
}
if (modified)
paymod_log(p, LOG_DBG,
"Updated a channel hint for %s: "
"enabled %s, "
"estimated capacity %s",
type_to_string(tmpctx,
struct short_channel_id_dir,
&hint->scid),
hint->enabled ? "true" : "false",
type_to_string(tmpctx,
struct amount_msat,
&hint->estimated_capacity));
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
return;
}
}
/* No hint found, create one. */
hint.enabled = enabled;
hint.scid.scid = scid;
hint.scid.dir = direction;
hint.local = local;
if (estimated_capacity != NULL)
hint.estimated_capacity = *estimated_capacity;
if (htlc_budget != NULL)
hint.htlc_budget = *htlc_budget;
tal_arr_expand(&root->channel_hints, hint);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_DBG,
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
"Added a channel hint for %s: enabled %s, estimated capacity %s",
type_to_string(tmpctx, struct short_channel_id_dir, &hint.scid),
hint.enabled ? "true" : "false",
type_to_string(tmpctx, struct amount_msat,
&hint.estimated_capacity));
}
static void payment_exclude_most_expensive(struct payment *p)
{
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
struct route_hop *e = &p->route[0];
struct amount_msat fee, worst = AMOUNT_MSAT(0);
for (size_t i = 0; i < tal_count(p->route)-1; i++) {
if (!amount_msat_sub(&fee, p->route[i].amount, p->route[i+1].amount))
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_err(p, "Negative fee in a route.");
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
if (amount_msat_greater_eq(fee, worst)) {
e = &p->route[i];
worst = fee;
}
}
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
channel_hints_update(p, e->channel_id, e->direction, false, false,
NULL, NULL);
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
}
static void payment_exclude_longest_delay(struct payment *p)
{
struct route_hop *e = &p->route[0];
u32 delay, worst = 0;
for (size_t i = 0; i < tal_count(p->route)-1; i++) {
delay = p->route[i].delay - p->route[i+1].delay;
if (delay >= worst) {
e = &p->route[i];
worst = delay;
}
}
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
channel_hints_update(p, e->channel_id, e->direction, false, false,
NULL, NULL);
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
}
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
static struct amount_msat payment_route_fee(struct payment *p)
{
struct amount_msat fee;
if (!amount_msat_sub(&fee, p->route[0].amount, p->amount)) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p,
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
LOG_BROKEN,
"gossipd returned a route with a negative fee: sending %s "
"to deliver %s",
type_to_string(tmpctx, struct amount_msat,
&p->route[0].amount),
type_to_string(tmpctx, struct amount_msat, &p->amount));
abort();
}
return fee;
}
/* Update the constraints by subtracting the delta_fee and delta_cltv if the
* result is positive. Returns whether or not the update has been applied. */
static WARN_UNUSED_RESULT bool
payment_constraints_update(struct payment_constraints *cons,
const struct amount_msat delta_fee,
const u32 delta_cltv)
{
if (delta_cltv > cons->cltv_budget)
return false;
/* amount_msat_sub performs a check before actually subtracting. */
if (!amount_msat_sub(&cons->fee_budget, cons->fee_budget, delta_fee))
return false;
cons->cltv_budget -= delta_cltv;
return true;
}
paymod: Apply and unapply routes to the channel hints Add/remove the HTLC amount from the channel hints so concurrent getroute calls have the correct exclusions. This can sometimes underflow, if we're unlucky and call getroute too closely, but that's not a big issue, it can only result in a failed MPP attempt too many, nothing fatal, and it'll get corrected based on the result returned by the failing node.
2020-06-09 15:34:46 +02:00
/* Given a route and a couple of channel hints, apply the route to the channel
* hints, so we have a better estimation of channel's capacity. We apply a
* route to a channel hint before calling `sendonion` so subsequent `route`
* calls don't accidentally try to use those out-of-date estimates. We unapply
* if the payment failed, i.e., all HTLCs we might have added have been torn
* down again. Finally we leave the update in place if the payment went
* through, since the balances really changed in that case. The `remove`
* argument indicates whether we want to apply (`remove=false`), or clear a
* prior application (`remove=true`). */
static void payment_chanhints_apply_route(struct payment *p, bool remove)
{
struct route_hop *curhop;
struct channel_hint *curhint;
struct payment *root = payment_root(p);
assert(p->route != NULL);
for (size_t i = 0; i < tal_count(p->route); i++) {
curhop = &p->route[i];
for (size_t j = 0; j < tal_count(root->channel_hints); j++) {
curhint = &root->channel_hints[j];
if (short_channel_id_eq(&curhint->scid.scid,
&curhop->channel_id) &&
curhint->scid.dir == curhop->direction) {
paymod: Track how many HTLCs each channel can still add It turns out that by aggressively splitting payments we may end up exhausting the number of HTLCs we can add to a channel quickly. By tracking the number of HTLCs we can still add, and excluding the channels to which we cannot add any more we increase the route diversity, and avoid quickly exhausting the HTLC budget. In the next commit we'll also implement an early abort if we've exhausted all channels, so we don't end up splitting indefinitely and we can also optimize the initial split to not run afoul of that limit.
2020-07-20 20:28:31 +02:00
/* Update the number of htlcs for any local
* channel in the route */
if (curhint->local && remove)
curhint->htlc_budget++;
else if (curhint->local)
curhint->htlc_budget--;
paymod: Apply and unapply routes to the channel hints Add/remove the HTLC amount from the channel hints so concurrent getroute calls have the correct exclusions. This can sometimes underflow, if we're unlucky and call getroute too closely, but that's not a big issue, it can only result in a failed MPP attempt too many, nothing fatal, and it'll get corrected based on the result returned by the failing node.
2020-06-09 15:34:46 +02:00
if (remove && !amount_msat_add(
&curhint->estimated_capacity,
curhint->estimated_capacity,
curhop->amount)) {
/* This should never happen, it'd mean
* that we unapply a route that would
* result in a msatoshi
* wrap-around. */
abort();
} else if (!amount_msat_sub(
&curhint->estimated_capacity,
curhint->estimated_capacity,
curhop->amount)) {
/* This can happen in case of multipl
* concurrent getroute calls using the
* same channel_hints, no biggy, it's
* an estimation anyway. */
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_UNUSUAL,
paymod: Apply and unapply routes to the channel hints Add/remove the HTLC amount from the channel hints so concurrent getroute calls have the correct exclusions. This can sometimes underflow, if we're unlucky and call getroute too closely, but that's not a big issue, it can only result in a failed MPP attempt too many, nothing fatal, and it'll get corrected based on the result returned by the failing node.
2020-06-09 15:34:46 +02:00
"Could not update the channel hint "
"for %s. Could be a concurrent "
"`getroute` call.",
type_to_string(
tmpctx,
struct short_channel_id_dir,
&curhint->scid));
}
}
}
}
}
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
static struct command_result *payment_getroute_result(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
const jsmntok_t *rtok = json_get_member(buffer, toks, "route");
paymod: Compute fee and CLTV delta limits for a payment So far we got away with not caring about these but since we're implementing modifiers that impact these limits, we better keep track of them.
2020-06-03 15:59:12 +02:00
struct amount_msat fee;
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
assert(rtok != NULL);
paymod: Consolidate parsing RPC results in libplugin We handle these in a number of different ways, and regularly get the parsing and logic for optional fields wrong, so let's consolidate them here.
2020-06-09 18:30:14 +02:00
p->route = json_to_route(p, buffer, rtok);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
p->step = PAYMENT_STEP_GOT_ROUTE;
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
fee = payment_route_fee(p);
paymod: Compute fee and CLTV delta limits for a payment So far we got away with not caring about these but since we're implementing modifiers that impact these limits, we better keep track of them.
2020-06-03 15:59:12 +02:00
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
/* Ensure that our fee and CLTV budgets are respected. */
if (amount_msat_greater(fee, p->constraints.fee_budget)) {
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
payment_exclude_most_expensive(p);
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
p->route = tal_free(p->route);
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
payment_fail(
p, "Fee exceeds our fee budget: %s > %s, discarding route",
type_to_string(tmpctx, struct amount_msat, &fee),
type_to_string(tmpctx, struct amount_msat,
&p->constraints.fee_budget));
paymod: Compute fee and CLTV delta limits for a payment So far we got away with not caring about these but since we're implementing modifiers that impact these limits, we better keep track of them.
2020-06-03 15:59:12 +02:00
return command_still_pending(cmd);
}
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
if (p->route[0].delay > p->constraints.cltv_budget) {
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
u32 delay = p->route[0].delay;
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
payment_exclude_longest_delay(p);
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
p->route = tal_free(p->route);
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
payment_fail(p, "CLTV delay exceeds our CLTV budget: %d > %d",
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
delay, p->constraints.cltv_budget);
paymod: Compute fee and CLTV delta limits for a payment So far we got away with not caring about these but since we're implementing modifiers that impact these limits, we better keep track of them.
2020-06-03 15:59:12 +02:00
return command_still_pending(cmd);
}
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
/* Now update the constraints in fee_budget and cltv_budget so
* modifiers know what constraints they need to adhere to. */
if (!payment_constraints_update(&p->constraints, fee, p->route[0].delay)) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_BROKEN,
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
"Could not update constraints.");
abort();
}
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
/* Allow modifiers to modify the route, before
* payment_compute_onion_payloads uses the route to generate the
* onion_payloads */
payment_continue(p);
return command_still_pending(cmd);
}
static struct command_result *payment_getroute_error(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
int code;
const jsmntok_t *codetok = json_get_member(buffer, toks, "code"),
*msgtok = json_get_member(buffer, toks, "message");
json_to_int(buffer, codetok, &code);
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
p->route = NULL;
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
payment_fail(
p, "Error computing a route to %s: %.*s (%d)",
type_to_string(tmpctx, struct node_id, p->getroute->destination),
json_tok_full_len(msgtok), json_tok_full(buffer, msgtok), code);
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
/* Let payment_finished_ handle this, so we mark it as pending */
return command_still_pending(cmd);
}
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
static const struct short_channel_id_dir *
payment_get_excluded_channels(const tal_t *ctx, struct payment *p)
paymod: Include excludes resulting from channel_hints for getrotue
2020-05-22 17:40:18 +02:00
{
struct payment *root = payment_root(p);
struct channel_hint *hint;
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
struct short_channel_id_dir *res =
tal_arr(ctx, struct short_channel_id_dir, 0);
paymod: Include excludes resulting from channel_hints for getrotue
2020-05-22 17:40:18 +02:00
for (size_t i = 0; i < tal_count(root->channel_hints); i++) {
hint = &root->channel_hints[i];
if (!hint->enabled)
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
tal_arr_expand(&res, hint->scid);
paymod: Include excludes resulting from channel_hints for getrotue
2020-05-22 17:40:18 +02:00
else if (amount_msat_greater_eq(p->amount,
hint->estimated_capacity))
pay: Be less aggressive when estimating channel capacity We'd previously take the failed attempt and estimate the failing channel's capacity at 3/4 of the attempted amount, which is rather aggressive. This reduces this aggressiveness to use the exact amount tried, but excluding on equality. This still skips attempting the same route with the same amount, but also permits attempts that are in the range [3/4, 1] of the failed attempt amount to still be attempted.
2020-07-27 17:42:54 +02:00
/* We exclude on equality because we've set the
* estimate to the smallest failed attempt. */
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
tal_arr_expand(&res, hint->scid);
pay: Be less aggressive when estimating channel capacity We'd previously take the failed attempt and estimate the failing channel's capacity at 3/4 of the attempted amount, which is rather aggressive. This reduces this aggressiveness to use the exact amount tried, but excluding on equality. This still skips attempting the same route with the same amount, but also permits attempts that are in the range [3/4, 1] of the failed attempt amount to still be attempted.
2020-07-27 17:42:54 +02:00
paymod: Track how many HTLCs each channel can still add It turns out that by aggressively splitting payments we may end up exhausting the number of HTLCs we can add to a channel quickly. By tracking the number of HTLCs we can still add, and excluding the channels to which we cannot add any more we increase the route diversity, and avoid quickly exhausting the HTLC budget. In the next commit we'll also implement an early abort if we've exhausted all channels, so we don't end up splitting indefinitely and we can also optimize the initial split to not run afoul of that limit.
2020-07-20 20:28:31 +02:00
else if (hint->local && hint->htlc_budget == 0)
/* If we cannot add any HTLCs to the channel we
* shouldn't look for a route through that channel */
tal_arr_expand(&res, hint->scid);
paymod: Include excludes resulting from channel_hints for getrotue
2020-05-22 17:40:18 +02:00
}
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
return res;
}
static const struct node_id *payment_get_excluded_nodes(const tal_t *ctx,
struct payment *p)
{
struct payment *root = payment_root(p);
return root->excluded_nodes;
}
/* Iterate through the channel_hints and exclude any channel that we are
* confident will not be able to handle this payment. */
static void payment_getroute_add_excludes(struct payment *p,
struct json_stream *js)
{
const struct node_id *nodes;
const struct short_channel_id_dir *chans;
json_array_start(js, "exclude");
/* Collect and exclude all channels that are disabled or we know have
* insufficient capacity. */
chans = payment_get_excluded_channels(tmpctx, p);
for (size_t i=0; i<tal_count(chans); i++)
json_add_short_channel_id_dir(js, NULL, &chans[i]);
/* Now also exclude nodes that we think have failed. */
nodes = payment_get_excluded_nodes(tmpctx, p);
for (size_t i=0; i<tal_count(nodes); i++)
json_add_node_id(js, NULL, &nodes[i]);
plugins/pay: Exclude the entrypoint to a routehint to avoid cycles This uses @cdecker's idea of excluding the routehinted channel from the route, and also consumes the route hints as it goes so that it makes progress. I don't know if this is correct, but it reliably passes tests/test_pay.py::test_tlv_or_legacy now.
2020-07-23 06:50:10 +02:00
/* And make sure we don't route in a circle via the routehint! */
plugins/libplugin-pay.c: Properly handle exclusions for routehints with two hops or more. Arguably a low-priority bug since no current node ever generates routehints longer than one hop. However, it is possible as an edge case, if the destination is directly accessible *and* supports multiple channels, that we route through the destination, one of the *other* channels it has not in the routehint, to the entry point, and then through the routehint. This change removes the risk of the above edge case. Changelog-None: arguably a low-priority bug.
2020-07-31 09:59:20 +02:00
if (p->temp_exclusion)
for (size_t i = 0; i < tal_count(p->temp_exclusion); ++i)
json_add_string(js, NULL, p->temp_exclusion[i]);
plugins/pay: Exclude the entrypoint to a routehint to avoid cycles This uses @cdecker's idea of excluding the routehinted channel from the route, and also consumes the route hints as it goes so that it makes progress. I don't know if this is correct, but it reliably passes tests/test_pay.py::test_tlv_or_legacy now.
2020-07-23 06:50:10 +02:00
paymod: Include excludes resulting from channel_hints for getrotue
2020-05-22 17:40:18 +02:00
json_array_end(js);
}
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
static void payment_getroute(struct payment *p)
{
struct out_req *req;
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
req = jsonrpc_request_start(p->plugin, NULL, "getroute",
paymod: Implement getroute call
2020-05-12 19:51:37 +02:00
payment_getroute_result,
payment_getroute_error, p);
paymod: Wrap the getroute request in a struct Keeping the arguments in their non-serialized form allows us to amend them as we apply modifiers and learn new information.
2020-06-01 14:57:37 +02:00
json_add_node_id(req->js, "id", p->getroute->destination);
json_add_amount_msat_only(req->js, "msatoshi", p->getroute->amount);
json_add_num(req->js, "cltv", p->getroute->cltv);
json_add_num(req->js, "maxhops", p->getroute->max_hops);
paymod: Expose riskfactor and wire through to getroute
2020-06-11 15:53:55 +02:00
json_add_member(req->js, "riskfactor", false, "%lf",
p->getroute->riskfactorppm / 1000000.0);
paymod: Include excludes resulting from channel_hints for getrotue
2020-05-22 17:40:18 +02:00
payment_getroute_add_excludes(p, req->js);
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
send_outreq(p->plugin, req);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
}
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
static u8 *tal_towire_legacy_payload(const tal_t *ctx, const struct legacy_payload *payload)
{
const u8 padding[] = {0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
/* Prepend 0 byte for realm */
u8 *buf = tal_arrz(ctx, u8, 1);
towire_short_channel_id(&buf, &payload->scid);
towire_u64(&buf, payload->forward_amt.millisatoshis); /* Raw: low-level serializer */
towire_u32(&buf, payload->outgoing_cltv);
towire(&buf, padding, ARRAY_SIZE(padding));
assert(tal_bytelen(buf) == 1 + 32);
return buf;
}
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
static struct payment_result *tal_sendpay_result_from_json(const tal_t *ctx,
const char *buffer,
const jsmntok_t *toks)
{
const jsmntok_t *idtok = json_get_member(buffer, toks, "id");
const jsmntok_t *hashtok = json_get_member(buffer, toks, "payment_hash");
const jsmntok_t *partidtok = json_get_member(buffer, toks, "partid");
const jsmntok_t *senttok = json_get_member(buffer, toks, "amount_sent_msat");
const jsmntok_t *statustok = json_get_member(buffer, toks, "status");
const jsmntok_t *preimagetok = json_get_member(buffer, toks, "payment_preimage");
const jsmntok_t *codetok = json_get_member(buffer, toks, "code");
const jsmntok_t *datatok = json_get_member(buffer, toks, "data");
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
const jsmntok_t *erridxtok, *msgtok, *failcodetok, *rawmsgtok,
*failcodenametok, *errchantok, *errnodetok, *errdirtok;
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
struct payment_result *result;
/* Check if we have an error and need to descend into data to get
* details. */
if (codetok != NULL && datatok != NULL) {
idtok = json_get_member(buffer, datatok, "id");
hashtok = json_get_member(buffer, datatok, "payment_hash");
partidtok = json_get_member(buffer, datatok, "partid");
senttok = json_get_member(buffer, datatok, "amount_sent_msat");
statustok = json_get_member(buffer, datatok, "status");
}
/* Initial sanity checks, all these fields must exist. */
if (idtok == NULL || idtok->type != JSMN_PRIMITIVE ||
hashtok == NULL || hashtok->type != JSMN_STRING ||
senttok == NULL || senttok->type != JSMN_STRING ||
statustok == NULL || statustok->type != JSMN_STRING) {
return NULL;
}
result = tal(ctx, struct payment_result);
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
if (codetok != NULL)
json_to_u32(buffer, codetok, &result->code);
else
result->code = 0;
paymod: Use reasonable partids We were just handwaving the partid generation, which broke some tests that expected the first payment attempt to always have partid=0, so here we just track the partids assigned in the payment tree, starting at 0.
2020-05-18 17:46:50 +02:00
/* If the partid is 0 it'd be omitted in waitsendpay, fix this here. */
if (partidtok != NULL)
json_to_u32(buffer, partidtok, &result->partid);
else
result->partid = 0;
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
json_to_u64(buffer, idtok, &result->id);
json_to_msat(buffer, senttok, &result->amount_sent);
if (json_tok_streq(buffer, statustok, "pending")) {
result->state = PAYMENT_PENDING;
} else if (json_tok_streq(buffer, statustok, "complete")) {
result->state = PAYMENT_COMPLETE;
} else if (json_tok_streq(buffer, statustok, "failed")) {
result->state = PAYMENT_FAILED;
} else {
goto fail;
}
if (preimagetok != NULL) {
result->payment_preimage = tal(result, struct preimage);
json_to_preimage(buffer, preimagetok, result->payment_preimage);
}
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
/* Now extract the error details if the error code is not 0 */
if (result->code != 0) {
erridxtok = json_get_member(buffer, datatok, "erring_index");
errnodetok = json_get_member(buffer, datatok, "erring_node");
errchantok = json_get_member(buffer, datatok, "erring_channel");
errdirtok = json_get_member(buffer, datatok, "erring_direction");
failcodetok = json_get_member(buffer, datatok, "failcode");
failcodenametok =json_get_member(buffer, datatok, "failcodename");
msgtok = json_get_member(buffer, toks, "message");
rawmsgtok = json_get_member(buffer, datatok, "raw_message");
if (failcodetok == NULL || failcodetok->type != JSMN_PRIMITIVE ||
paymod: Fix waitsendpay error parsing for unknown failure codes It turns out that the `failcodename` doesn't get populated if the `failcode` isn't a known error from the enum (duh...) so don't fail parsing if it's missing.
2020-07-02 14:39:28 +02:00
(failcodenametok != NULL && failcodenametok->type != JSMN_STRING) ||
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
(erridxtok != NULL && erridxtok->type != JSMN_PRIMITIVE) ||
(errnodetok != NULL && errnodetok->type != JSMN_STRING) ||
(errchantok != NULL && errchantok->type != JSMN_STRING) ||
(errdirtok != NULL && errdirtok->type != JSMN_PRIMITIVE) ||
msgtok == NULL || msgtok->type != JSMN_STRING ||
(rawmsgtok != NULL && rawmsgtok->type != JSMN_STRING))
goto fail;
if (rawmsgtok != NULL)
result->raw_message = json_tok_bin_from_hex(result, buffer, rawmsgtok);
else
result->raw_message = NULL;
paymod: Fix waitsendpay error parsing for unknown failure codes It turns out that the `failcodename` doesn't get populated if the `failcode` isn't a known error from the enum (duh...) so don't fail parsing if it's missing.
2020-07-02 14:39:28 +02:00
if (failcodenametok != NULL)
result->failcodename = json_strdup(result, buffer, failcodenametok);
else
result->failcodename = NULL;
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
json_to_u32(buffer, failcodetok, &result->failcode);
result->message = json_strdup(result, buffer, msgtok);
if (erridxtok != NULL) {
result->erring_index = tal(result, u32);
json_to_u32(buffer, erridxtok, result->erring_index);
} else {
result->erring_index = NULL;
}
if (errdirtok != NULL) {
result->erring_direction = tal(result, int);
json_to_int(buffer, errdirtok, result->erring_direction);
} else {
result->erring_direction = NULL;
}
if (errnodetok != NULL) {
result->erring_node = tal(result, struct node_id);
json_to_node_id(buffer, errnodetok,
result->erring_node);
} else {
result->erring_node = NULL;
}
if (errchantok != NULL) {
result->erring_channel =
tal(result, struct short_channel_id);
json_to_short_channel_id(buffer, errchantok,
result->erring_channel);
} else {
result->erring_channel = NULL;
}
}
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
return result;
fail:
return tal_free(result);
}
paymod: Add the contents of paystatus This proved to be rather difficult given the tight coupling of the old structs and the output of the command.
2020-05-29 15:25:00 +02:00
/* Try to infer the erring_node, erring_channel and erring_direction from what
* we know, but don't override the values that are returned by `waitsendpay`. */
static void payment_result_infer(struct route_hop *route,
struct payment_result *r)
{
paymod: Do not assume that parsing the waitsendpay result succeeds Suggested-by: ZmnSCPxj Signed-off-by: Christian Decker Reference: #3846
2020-07-17 13:33:02 +02:00
int i, len;
assert(r != NULL);
paymod: Add the contents of paystatus This proved to be rather difficult given the tight coupling of the old structs and the output of the command.
2020-05-29 15:25:00 +02:00
if (r->code == 0 || r->erring_index == NULL || route == NULL)
return;
paymod: Do not assume that parsing the waitsendpay result succeeds Suggested-by: ZmnSCPxj Signed-off-by: Christian Decker Reference: #3846
2020-07-17 13:33:02 +02:00
len = tal_count(route);
paymod: Add the contents of paystatus This proved to be rather difficult given the tight coupling of the old structs and the output of the command.
2020-05-29 15:25:00 +02:00
i = *r->erring_index;
assert(i <= len);
if (r->erring_node == NULL)
r->erring_node = &route[i-1].nodeid;
/* The above assert was enough for the erring_node, but might be off
* by one on channel and direction, in case the destination failed on
* us. */
if (i == len)
return;
if (r->erring_channel == NULL)
r->erring_channel = &route[i].channel_id;
if (r->erring_direction == NULL)
r->erring_direction = &route[i].direction;
}
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
/* If a node takes too much fee or cltv, the next one reports it. We don't
* know who to believe, but log it */
static void report_tampering(struct payment *p,
size_t report_pos,
const char *style)
paymod: Call waitsendpay on the sendonion we just queued
2020-05-14 15:57:21 +02:00
{
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
const struct node_id *id = &p->route[report_pos].nodeid;
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
if (report_pos == 0) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_UNUSUAL,
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
"Node #%zu (%s) claimed we sent them invalid %s",
report_pos + 1,
type_to_string(tmpctx, struct node_id, id),
style);
} else {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_UNUSUAL,
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
"Node #%zu (%s) claimed #%zu (%s) sent them invalid %s",
report_pos + 1,
type_to_string(tmpctx, struct node_id, id),
report_pos,
type_to_string(tmpctx, struct node_id,
&p->route[report_pos-1].nodeid),
style);
plugin: Do not get upset if it can't parse waitsendpay result We were rather pedanticly failing the plugin if we were unable to parse the `waitsendpay` result, but had coded all the modifiers in such a way that they can handle a `NULL` result (verified in the code and manually by randomly failing the parsing). So we now just log the result we failed to parse and merrily go our way. Worst case is that we end up retrying the same route multiple times, since we can't blacklist any nodes / channels without understanding the error, but that is still in the scope of what we must handle anyway.
2020-07-17 16:07:27 +02:00
}
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
}
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
static bool
failure_is_blockheight_disagreement(const struct payment *p,
u32 *blockheight)
{
struct amount_msat unused;
assert(p && p->result);
if (p->result->failcode == 17 /* Former final_expiry_too_soon */)
*blockheight = p->start_block + 1;
else if (!fromwire_incorrect_or_unknown_payment_details(
p->result->raw_message,
&unused, blockheight))
/* If it's incorrect_or_unknown_payment_details, that tells us
* what height they're at */
return false;
/* If we are already at the desired blockheight there is no point in
* waiting, and it is likely just some other error. Notice that
* start_block gets set by the initial getinfo call for each
* attempt.*/
if (*blockheight <= p->start_block)
return false;
return true;
}
plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes.
2020-08-15 11:56:41 +02:00
static char *describe_failcode(const tal_t *ctx, enum onion_type failcode)
{
char *rv = tal_strdup(ctx, "");
if (failcode & BADONION) {
tal_append_fmt(&rv, "BADONION|");
failcode &= ~BADONION;
}
if (failcode & PERM) {
tal_append_fmt(&rv, "PERM|");
failcode &= ~PERM;
}
if (failcode & NODE) {
tal_append_fmt(&rv, "NODE|");
failcode &= ~NODE;
}
if (failcode & UPDATE) {
tal_append_fmt(&rv, "UPDATE|");
failcode &= ~UPDATE;
}
tal_append_fmt(&rv, "%u", failcode);
return rv;
}
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
static struct command_result *
handle_final_failure(struct command *cmd,
struct payment *p,
const struct node_id *final_id,
enum onion_type failcode)
{
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
u32 unused;
/* Need to check for blockheight disagreement case here,
* otherwise we would set the abort flag too eagerly.
*/
if (failure_is_blockheight_disagreement(p, &unused)) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_DBG,
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
"Blockheight disagreement, not aborting.");
goto nonerror;
}
plugins/libplugin-pay.c: Store the route description, and re-report on failure.
2020-08-25 03:54:52 +02:00
paymod_log(p, LOG_DBG,
"Final node %s reported %04x (%s) on route %s",
type_to_string(tmpctx, struct node_id, final_id),
failcode, onion_type_name(failcode),
p->routetxt);
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
/* We use an exhaustive switch statement here so you get a compile
* warning when new ones are added, and can think about where they go */
switch (failcode) {
case WIRE_FINAL_INCORRECT_CLTV_EXPIRY:
report_tampering(p, tal_count(p->route)-1, "cltv");
goto error;
case WIRE_FINAL_INCORRECT_HTLC_AMOUNT:
report_tampering(p, tal_count(p->route)-1, "amount");
goto error;
paymod: Do not assume that parsing the waitsendpay result succeeds Suggested-by: ZmnSCPxj Signed-off-by: Christian Decker Reference: #3846
2020-07-17 13:33:02 +02:00
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
/* BOLT #4:
*
* A _forwarding node_ MAY, but a _final node_ MUST NOT:
*...
* - return an `invalid_onion_version` error.
*...
* - return an `invalid_onion_hmac` error.
*...
* - return an `invalid_onion_key` error.
*...
* - return a `temporary_channel_failure` error.
*...
* - return a `permanent_channel_failure` error.
*...
* - return a `required_channel_feature_missing` error.
*...
* - return an `unknown_next_peer` error.
*...
* - return an `amount_below_minimum` error.
*...
* - return a `fee_insufficient` error.
*...
* - return an `incorrect_cltv_expiry` error.
*...
* - return an `expiry_too_soon` error.
*...
* - return an `expiry_too_far` error.
*...
* - return a `channel_disabled` error.
*/
case WIRE_INVALID_ONION_VERSION:
case WIRE_INVALID_ONION_HMAC:
case WIRE_INVALID_ONION_KEY:
case WIRE_TEMPORARY_CHANNEL_FAILURE:
case WIRE_PERMANENT_CHANNEL_FAILURE:
case WIRE_REQUIRED_CHANNEL_FEATURE_MISSING:
case WIRE_UNKNOWN_NEXT_PEER:
case WIRE_AMOUNT_BELOW_MINIMUM:
case WIRE_FEE_INSUFFICIENT:
case WIRE_INCORRECT_CLTV_EXPIRY:
case WIRE_EXPIRY_TOO_FAR:
case WIRE_EXPIRY_TOO_SOON:
case WIRE_CHANNEL_DISABLED:
goto strange_error;
case WIRE_INVALID_ONION_PAYLOAD:
case WIRE_INVALID_REALM:
case WIRE_PERMANENT_NODE_FAILURE:
case WIRE_TEMPORARY_NODE_FAILURE:
case WIRE_REQUIRED_NODE_FEATURE_MISSING:
#if EXPERIMENTAL_FEATURES
case WIRE_INVALID_ONION_BLINDING:
#endif
case WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS:
case WIRE_MPP_TIMEOUT:
goto error;
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
}
paymod: Parse and store the waitsendpay result This is necessary so we can later aggregate across an entire tree of attempts and report success or failure to the RPC caller.
2020-05-14 20:22:40 +02:00
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
strange_error:
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_UNUSUAL,
plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes.
2020-08-15 11:56:41 +02:00
"Final node %s reported strange error code %04x (%s)",
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
type_to_string(tmpctx, struct node_id, final_id),
plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes.
2020-08-15 11:56:41 +02:00
failcode, describe_failcode(tmpctx, failcode));
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
error:
p->result->code = PAY_DESTINATION_PERM_FAIL;
payment_root(p)->abort = true;
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
nonerror:
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
payment_fail(p, "%s", p->result->message);
return command_still_pending(cmd);
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
}
static struct command_result *
handle_intermediate_failure(struct command *cmd,
struct payment *p,
const struct node_id *errnode,
const struct route_hop *errchan,
enum onion_type failcode)
{
struct payment *root = payment_root(p);
plugins/libplugin-pay.c: Store the route description, and re-report on failure.
2020-08-25 03:54:52 +02:00
paymod_log(p, LOG_DBG,
"Intermediate node %s reported %04x (%s) at %s on route %s",
type_to_string(tmpctx, struct node_id, errnode),
failcode, onion_type_name(failcode),
type_to_string(tmpctx, struct short_channel_id,
&errchan->channel_id),
p->routetxt);
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
/* We use an exhaustive switch statement here so you get a compile
* warning when new ones are added, and can think about where they go */
switch (failcode) {
/* BOLT #4:
paymod: Add a comment about how we derive errors from erring_index Mainly to help my future self remember
2020-07-19 21:31:41 +02:00
*
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
* An _intermediate hop_ MUST NOT, but the _final node_:
*...
* - MUST return an `incorrect_or_unknown_payment_details` error.
*...
* - MUST return `final_incorrect_cltv_expiry` error.
*...
* - MUST return a `final_incorrect_htlc_amount` error.
paymod: Add a comment about how we derive errors from erring_index Mainly to help my future self remember
2020-07-19 21:31:41 +02:00
*/
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
case WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS:
case WIRE_FINAL_INCORRECT_CLTV_EXPIRY:
case WIRE_FINAL_INCORRECT_HTLC_AMOUNT:
/* FIXME: Document in BOLT that intermediates must not return this! */
case WIRE_MPP_TIMEOUT:
goto strange_error;
paymod: Add a comment about how we derive errors from erring_index Mainly to help my future self remember
2020-07-19 21:31:41 +02:00
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
case WIRE_PERMANENT_CHANNEL_FAILURE:
case WIRE_CHANNEL_DISABLED:
case WIRE_UNKNOWN_NEXT_PEER:
case WIRE_REQUIRED_CHANNEL_FEATURE_MISSING:
/* All of these result in the channel being marked as disabled. */
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
channel_hints_update(root, errchan->channel_id,
errchan->direction, false, false, NULL,
NULL);
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
break;
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
case WIRE_TEMPORARY_CHANNEL_FAILURE: {
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
/* These are an indication that the capacity was insufficient,
* remember the amount we tried as an estimate. */
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
channel_hints_update(root, errchan->channel_id,
pay: Be less aggressive when estimating channel capacity We'd previously take the failed attempt and estimate the failing channel's capacity at 3/4 of the attempted amount, which is rather aggressive. This reduces this aggressiveness to use the exact amount tried, but excluding on equality. This still skips attempting the same route with the same amount, but also permits attempts that are in the range [3/4, 1] of the failed attempt amount to still be attempted.
2020-07-27 17:42:54 +02:00
errchan->direction, true, false,
&errchan->amount, NULL);
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
goto error;
}
case WIRE_INCORRECT_CLTV_EXPIRY:
report_tampering(p, errchan - p->route, "cltv");
goto error;
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
case WIRE_INVALID_ONION_VERSION:
case WIRE_INVALID_ONION_HMAC:
case WIRE_INVALID_ONION_KEY:
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
case WIRE_PERMANENT_NODE_FAILURE:
case WIRE_TEMPORARY_NODE_FAILURE:
case WIRE_REQUIRED_NODE_FEATURE_MISSING:
case WIRE_INVALID_ONION_PAYLOAD:
case WIRE_INVALID_REALM:
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
#if EXPERIMENTAL_FEATURES
case WIRE_INVALID_ONION_BLINDING:
#endif
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
tal_arr_expand(&root->excluded_nodes, *errnode);
goto error;
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
case WIRE_AMOUNT_BELOW_MINIMUM:
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
case WIRE_FEE_INSUFFICIENT:
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
case WIRE_EXPIRY_TOO_FAR:
case WIRE_EXPIRY_TOO_SOON:
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
goto error;
paymod: Parse error from waitsendpay and exclude failed chans
2020-05-26 18:57:44 +02:00
}
paymod: Call waitsendpay on the sendonion we just queued
2020-05-14 15:57:21 +02:00
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
strange_error:
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_UNUSUAL,
plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes.
2020-08-15 11:56:41 +02:00
"Intermediate node %s reported strange error code %04x (%s)",
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
type_to_string(tmpctx, struct node_id, errnode),
plugins/libplugin-pay.c: Describe the bits of unrecognized failure codes.
2020-08-15 11:56:41 +02:00
failcode, describe_failcode(tmpctx, failcode));
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
error:
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
payment_fail(p, "%s", p->result->message);
paymod: Call waitsendpay on the sendonion we just queued
2020-05-14 15:57:21 +02:00
return command_still_pending(cmd);
}
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
/* From the docs:
*
* - *erring_index*: The index of the node along the route that
* reported the error. 0 for the local node, 1 for the first hop,
* and so on.
*
* The only difficulty is mapping the erring_index to the correct hop.
* We split into the erring node, and the error channel, since they're
* used in different contexts. NULL error_channel means it's the final
* node, whose errors are treated differently.
*/
static bool assign_blame(const struct payment *p,
const struct node_id **errnode,
const struct route_hop **errchan)
{
int index;
if (p->result->erring_index == NULL)
return false;
index = *p->result->erring_index;
/* BADONION errors are reported on behalf of the next node. */
if (p->result->failcode & BADONION)
index++;
/* Final node *shouldn't* report BADONION, but don't assume. */
if (index >= tal_count(p->route)) {
*errchan = NULL;
*errnode = &p->route[tal_count(p->route) - 1].nodeid;
return true;
}
*errchan = &p->route[index];
if (index == 0)
*errnode = p->local_id;
else
*errnode = &p->route[index - 1].nodeid;
return true;
}
static struct command_result *
payment_waitsendpay_finished(struct command *cmd, const char *buffer,
const jsmntok_t *toks, struct payment *p)
{
const struct node_id *errnode;
const struct route_hop *errchan;
assert(p->route != NULL);
p->end_time = time_now();
p->result = tal_sendpay_result_from_json(p, buffer, toks);
if (p->result == NULL) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_UNUSUAL,
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
"Unable to parse `waitsendpay` result: %.*s",
json_tok_full_len(toks),
json_tok_full(buffer, toks));
payment_set_step(p, PAYMENT_STEP_FAILED);
payment_continue(p);
return command_still_pending(cmd);
}
payment_result_infer(p->route, p->result);
if (p->result->state == PAYMENT_COMPLETE) {
payment_set_step(p, PAYMENT_STEP_SUCCESS);
payment_continue(p);
return command_still_pending(cmd);
}
payment_chanhints_apply_route(p, true);
if (!assign_blame(p, &errnode, &errchan)) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_UNUSUAL,
pay: handle returned errors more gracefully. The code had incorrect assertions, partially because it didn't clearly distinguish errors from the final node (which, barring blockheight issues, mean a complete failre) and intermediate nodes. In particular, we can't trust the *values*, so we need to distinguish these by the *sender*. If a route is of length 2 (A, B): - erring_index == 0 means us complaining about channel A. - erring_index == 1 means A.node complaining about channel B. - erring_index == 2 means the final destination node B.node. This is particularly of note because Travis does NOT run test_pay_routeboost! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-20 07:47:55 +02:00
"No erring_index set in `waitsendpay` result: %.*s",
json_tok_full_len(toks),
json_tok_full(buffer, toks));
/* FIXME: Pick a random channel to fail? */
payment_set_step(p, PAYMENT_STEP_FAILED);
payment_continue(p);
return command_still_pending(cmd);
}
if (!errchan)
return handle_final_failure(cmd, p, errnode,
p->result->failcode);
return handle_intermediate_failure(cmd, p, errnode, errchan,
p->result->failcode);
}
paymod: Parse createonion result and call sendonion
2020-05-14 15:37:48 +02:00
static struct command_result *payment_sendonion_success(struct command *cmd,
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
paymod: Call waitsendpay on the sendonion we just queued
2020-05-14 15:57:21 +02:00
struct out_req *req;
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
req = jsonrpc_request_start(p->plugin, NULL, "waitsendpay",
paymod: Call waitsendpay on the sendonion we just queued
2020-05-14 15:57:21 +02:00
payment_waitsendpay_finished,
payment_waitsendpay_finished, p);
json_add_sha256(req->js, "payment_hash", p->payment_hash);
json_add_num(req->js, "partid", p->partid);
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
send_outreq(p->plugin, req);
paymod: Call waitsendpay on the sendonion we just queued
2020-05-14 15:57:21 +02:00
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
return command_still_pending(cmd);
}
paymod: Parse createonion result and call sendonion
2020-05-14 15:37:48 +02:00
static struct command_result *payment_createonion_success(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
struct out_req *req;
struct route_hop *first = &p->route[0];
struct secret *secrets;
paymod: Apply and unapply routes to the channel hints Add/remove the HTLC amount from the channel hints so concurrent getroute calls have the correct exclusions. This can sometimes underflow, if we're unlucky and call getroute too closely, but that's not a big issue, it can only result in a failed MPP attempt too many, nothing fatal, and it'll get corrected based on the result returned by the failing node.
2020-06-09 15:34:46 +02:00
payment_chanhints_apply_route(p, false);
paymod: Consolidate parsing RPC results in libplugin We handle these in a number of different ways, and regularly get the parsing and logic for optional fields wrong, so let's consolidate them here.
2020-06-09 18:30:14 +02:00
p->createonion_response = json_to_createonion_response(p, buffer, toks);
paymod: Parse createonion result and call sendonion
2020-05-14 15:37:48 +02:00
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
req = jsonrpc_request_start(p->plugin, NULL, "sendonion",
paymod: Parse createonion result and call sendonion
2020-05-14 15:37:48 +02:00
payment_sendonion_success,
payment_rpc_failure, p);
json_add_hex_talarr(req->js, "onion", p->createonion_response->onion);
json_object_start(req->js, "first_hop");
json_add_short_channel_id(req->js, "channel", &first->channel_id);
json_add_num(req->js, "direction", first->direction);
json_add_amount_msat_only(req->js, "amount_msat", first->amount);
json_add_num(req->js, "delay", first->delay);
json_add_node_id(req->js, "id", &first->nodeid);
json_object_end(req->js);
json_add_sha256(req->js, "payment_hash", p->payment_hash);
jsonrpc: Add `msatoshi` argument to `sendonion` to annotate While not directly necessary, it still feeds the `listpays` result, and so we should pass it along if we can, so we don't have to rely solely on the `amount_sent` field, which includes the fees. Reported-by: Rusty Russell <@rustyrussell>
2020-07-27 16:01:49 +02:00
json_add_amount_msat_only(req->js, "msatoshi", p->amount);
paymod: Parse createonion result and call sendonion
2020-05-14 15:37:48 +02:00
json_array_start(req->js, "shared_secrets");
secrets = p->createonion_response->shared_secrets;
for(size_t i=0; i<tal_count(secrets); i++)
json_add_secret(req->js, NULL, &secrets[i]);
json_array_end(req->js);
json_add_num(req->js, "partid", p->partid);
paymod: Add label to sendonion calls These get reflected in the `listsendpays` command, and are quite useful.
2020-06-11 15:45:32 +02:00
if (p->label)
json_add_string(req->js, "label", p->label);
plugins/pay: hand bolt11 arg to sendonion if we have one (i.e. for `pay`) [ Extracted into standalone patch and comment added by RR ]
2020-07-26 03:13:56 +02:00
if (p->bolt11)
json_add_string(req->js, "bolt11", p->bolt11);
listpays mod 1: add destination inside the response when bolt11 is null Changelog-Added: JSON-RPC: `listpays` now lists the `destination` if it was provided (e.g., via the `pay` plugin or `keysend` plugin)
2020-07-31 17:19:22 +02:00
if (p->destination)
json_add_node_id(req->js, "destination", p->destination);
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
send_outreq(p->plugin, req);
paymod: Parse createonion result and call sendonion
2020-05-14 15:37:48 +02:00
return command_still_pending(cmd);
}
paymod: Implement TLV onion payload generation
2020-05-20 13:58:39 +02:00
/* Temporary serialization method for the tlv_payload.data until we rework the
* API that is generated from the specs to use the setter/getter interface. */
static void tlvstream_set_tlv_payload_data(struct tlv_field **stream,
struct secret *payment_secret,
u64 total_msat)
{
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
u8 *ser = tal_arr(NULL, u8, 0);
towire_secret(&ser, payment_secret);
towire_tu64(&ser, total_msat);
tlv: Allow passing a raw pointer and a length to tlvstream_set_raw Allows us to do fewer allocations, since the argument doesn't have to be tal allocated itself. Suggested-by: Rusty Russell <@rustyrussell>
2020-07-09 10:39:22 +02:00
tlvstream_set_raw(stream, TLV_TLV_PAYLOAD_PAYMENT_DATA, ser, tal_bytelen(ser));
tal_free(ser);
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
}
tlv: Allow passing a raw pointer and a length to tlvstream_set_raw Allows us to do fewer allocations, since the argument doesn't have to be tal allocated itself. Suggested-by: Rusty Russell <@rustyrussell>
2020-07-09 10:39:22 +02:00
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
static void payment_add_hop_onion_payload(struct payment *p,
struct createonion_hop *dst,
struct route_hop *node,
struct route_hop *next,
bool final,
struct secret *payment_secret)
{
struct createonion_request *cr = p->createonion_request;
paymod: Add the courtesy +1 to the CLTVs to allow for a new block This may be related to the issue #3862, however the water was muddied by it being the wrong error to return, and the node should not expect this courtesy feature to be present at all...
2020-07-21 20:45:38 +02:00
u32 cltv = p->start_block + next->delay + 1;
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
u64 msat = next->amount.millisatoshis; /* Raw: TLV payload generation*/
paymod: Fix the onion payload construction I wrongly used a pointer to the array that'd move on append, so an extra dereference was required here.
2020-07-02 18:32:47 +02:00
struct tlv_field **fields;
pay: Fix final TLV payload if not going through MPP modifiers Reported-by: ZmnSCPxj Signed-off-by: Christian Decker <@cdecker> Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
2020-08-09 15:26:46 +02:00
struct payment *root = payment_root(p);
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
static struct short_channel_id all_zero_scid = {.u64 = 0};
/* This is the information of the node processing this payload, while
* `next` are the instructions to include in the payload, which is
* basically the channel going to the next node. */
dst->style = node->style;
dst->pubkey = node->nodeid;
switch (node->style) {
case ROUTE_HOP_LEGACY:
dst->legacy_payload = tal(cr->hops, struct legacy_payload);
dst->legacy_payload->forward_amt = next->amount;
if (!final)
dst->legacy_payload->scid = next->channel_id;
else
dst->legacy_payload->scid = all_zero_scid;
dst->legacy_payload->outgoing_cltv = cltv;
break;
case ROUTE_HOP_TLV:
dst->tlv_payload = tlv_tlv_payload_new(cr->hops);
paymod: Fix the onion payload construction I wrongly used a pointer to the array that'd move on append, so an extra dereference was required here.
2020-07-02 18:32:47 +02:00
fields = &dst->tlv_payload->fields;
tlvstream_set_tu64(fields, TLV_TLV_PAYLOAD_AMT_TO_FORWARD,
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
msat);
paymod: Fix the onion payload construction I wrongly used a pointer to the array that'd move on append, so an extra dereference was required here.
2020-07-02 18:32:47 +02:00
tlvstream_set_tu32(fields, TLV_TLV_PAYLOAD_OUTGOING_CLTV_VALUE,
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
cltv);
if (!final)
paymod: Fix the onion payload construction I wrongly used a pointer to the array that'd move on append, so an extra dereference was required here.
2020-07-02 18:32:47 +02:00
tlvstream_set_short_channel_id(fields,
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
TLV_TLV_PAYLOAD_SHORT_CHANNEL_ID,
&next->channel_id);
if (payment_secret != NULL) {
assert(final);
pay: Fix final TLV payload if not going through MPP modifiers Reported-by: ZmnSCPxj Signed-off-by: Christian Decker <@cdecker> Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
2020-08-09 15:26:46 +02:00
tlvstream_set_tlv_payload_data(
fields, payment_secret,
root->amount.millisatoshis); /* Raw: TLV payload generation*/
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
}
break;
}
paymod: Implement TLV onion payload generation
2020-05-20 13:58:39 +02:00
}
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
static void payment_compute_onion_payloads(struct payment *p)
{
paymod: Implement legacy onion payload computation This is just for testing for now, TLV payload computation will come next. We stage all the payloads in deserialized form so modifiers can modify them more easily and serialize them only before actually calling `createonion`.
2020-05-14 15:22:49 +02:00
struct createonion_request *cr;
size_t hopcount;
paymod: Implement TLV onion payload generation
2020-05-20 13:58:39 +02:00
struct payment *root = payment_root(p);
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
char *routetxt = tal_strdup(tmpctx, "");
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
p->step = PAYMENT_STEP_ONION_PAYLOAD;
paymod: Implement legacy onion payload computation This is just for testing for now, TLV payload computation will come next. We stage all the payloads in deserialized form so modifiers can modify them more easily and serialize them only before actually calling `createonion`.
2020-05-14 15:22:49 +02:00
hopcount = tal_count(p->route);
/* Now compute the payload we're about to pass to `createonion` */
cr = p->createonion_request = tal(p, struct createonion_request);
cr->assocdata = tal_arr(cr, u8, 0);
towire_sha256(&cr->assocdata, p->payment_hash);
cr->session_key = NULL;
cr->hops = tal_arr(cr, struct createonion_hop, tal_count(p->route));
/* Non-final hops */
for (size_t i = 0; i < hopcount - 1; i++) {
/* The message is destined for hop i, but contains fields for
* i+1 */
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
payment_add_hop_onion_payload(p, &cr->hops[i], &p->route[i],
&p->route[i + 1], false, NULL);
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
tal_append_fmt(&routetxt, "%s -> ",
type_to_string(tmpctx, struct short_channel_id,
&p->route[i].channel_id));
paymod: Implement legacy onion payload computation This is just for testing for now, TLV payload computation will come next. We stage all the payloads in deserialized form so modifiers can modify them more easily and serialize them only before actually calling `createonion`.
2020-05-14 15:22:49 +02:00
}
/* Final hop */
paymod: Move onion payload construction into its own function Suggested-by: Rusty Russell <@rustyrussell>
2020-06-19 15:53:38 +02:00
payment_add_hop_onion_payload(
p, &cr->hops[hopcount - 1], &p->route[hopcount - 1],
&p->route[hopcount - 1], true, root->payment_secret);
plugins/libplugin-pay.c: Show routes being tried, also print updates to channel hints not just initial creations.
2020-08-25 03:37:25 +02:00
tal_append_fmt(&routetxt, "%s",
type_to_string(tmpctx, struct short_channel_id,
&p->route[hopcount - 1].channel_id));
paymod_log(p, LOG_DBG,
"Created outgoing onion for route: %s", routetxt);
paymod: Implement legacy onion payload computation This is just for testing for now, TLV payload computation will come next. We stage all the payloads in deserialized form so modifiers can modify them more easily and serialize them only before actually calling `createonion`.
2020-05-14 15:22:49 +02:00
plugins/libplugin-pay.c: Store the route description, and re-report on failure.
2020-08-25 03:54:52 +02:00
p->routetxt = tal_steal(p, routetxt);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
/* Now allow all the modifiers to mess with the payloads, before we
* serialize via a call to createonion in the next step. */
payment_continue(p);
}
static void payment_sendonion(struct payment *p)
{
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
struct out_req *req;
paymod: Implement TLV onion payload generation
2020-05-20 13:58:39 +02:00
u8 *payload, *tlv;
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
req = jsonrpc_request_start(p->plugin, NULL, "createonion",
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
payment_createonion_success,
payment_rpc_failure, p);
json_array_start(req->js, "hops");
for (size_t i = 0; i < tal_count(p->createonion_request->hops); i++) {
json_object_start(req->js, NULL);
struct createonion_hop *hop = &p->createonion_request->hops[i];
json_add_node_id(req->js, "pubkey", &hop->pubkey);
paymod: Implement TLV onion payload generation
2020-05-20 13:58:39 +02:00
if (hop->style == ROUTE_HOP_LEGACY) {
payload = tal_towire_legacy_payload(tmpctx, hop->legacy_payload);
json_add_hex_talarr(req->js, "payload", payload);
}else {
tlv = tal_arr(tmpctx, u8, 0);
towire_tlvstream_raw(&tlv, hop->tlv_payload->fields);
payload = tal_arr(tmpctx, u8, 0);
towire_bigsize(&payload, tal_bytelen(tlv));
towire(&payload, tlv, tal_bytelen(tlv));
json_add_hex_talarr(req->js, "payload", payload);
tal_free(tlv);
}
tal_free(payload);
paymod: Call `createonion` to generate the onion and associated data After we gave each modifier a chance to have its say, we can now proceed to generate the onion that's going to be sent in the next step.
2020-05-14 15:30:08 +02:00
json_object_end(req->js);
}
json_array_end(req->js);
json_add_hex_talarr(req->js, "assocdata",
p->createonion_request->assocdata);
if (p->createonion_request->session_key)
json_add_secret(req->js, "sessionkey",
p->createonion_request->session_key);
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
send_outreq(p->plugin, req);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
}
/* Mutual recursion. */
static void payment_finished(struct payment *p);
paymod: Implement bubbling results up the hierarchy of payments A payment is considered finished if it is in a final state (success or failure) or all its sub-payments are finished. If that's the case we notify `payment_finished` and bubble up through `payment_child_finished`, eventually bubbling up to the root, which can then report success of failure back to the RPC command that initiated the whole process.
2020-05-11 20:02:16 +02:00
/* A payment is finished if a) it is in a final state, of b) it's in a
* child-spawning state and all of its children are in a final state. */
static bool payment_is_finished(const struct payment *p)
{
mpp: Consider an abort as the payment being finished If one part sets the root to be aborted, there is little point in continuing to wait for the remainder, return to the caller immediately.
2020-07-13 16:08:13 +02:00
if (p->step == PAYMENT_STEP_FAILED || p->step == PAYMENT_STEP_SUCCESS || p->abort)
paymod: Implement bubbling results up the hierarchy of payments A payment is considered finished if it is in a final state (success or failure) or all its sub-payments are finished. If that's the case we notify `payment_finished` and bubble up through `payment_child_finished`, eventually bubbling up to the root, which can then report success of failure back to the RPC command that initiated the whole process.
2020-05-11 20:02:16 +02:00
return true;
else if (p->step == PAYMENT_STEP_SPLIT || p->step == PAYMENT_STEP_RETRY) {
bool running_children = false;
for (size_t i = 0; i < tal_count(p->children); i++)
running_children |= !payment_is_finished(p->children[i]);
return !running_children;
} else {
return false;
}
}
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
static enum payment_step payment_aggregate_states(struct payment *p)
{
enum payment_step agg = p->step;
for (size_t i=0; i<tal_count(p->children); i++)
agg |= payment_aggregate_states(p->children[i]);
return agg;
}
/* A payment is finished if a) it is in a final state, of b) it's in a
* child-spawning state and all of its children are in a final state. */
static bool payment_is_success(struct payment *p)
{
return (payment_aggregate_states(p) & PAYMENT_STEP_SUCCESS) != 0;
}
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
/* Function to bubble up completions to the root, which actually holds on to
* the command that initiated the flow. */
paymod: Implement bubbling results up the hierarchy of payments A payment is considered finished if it is in a final state (success or failure) or all its sub-payments are finished. If that's the case we notify `payment_finished` and bubble up through `payment_child_finished`, eventually bubbling up to the root, which can then report success of failure back to the RPC command that initiated the whole process.
2020-05-11 20:02:16 +02:00
static void payment_child_finished(struct payment *p,
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
struct payment *child)
{
paymod: Implement bubbling results up the hierarchy of payments A payment is considered finished if it is in a final state (success or failure) or all its sub-payments are finished. If that's the case we notify `payment_finished` and bubble up through `payment_child_finished`, eventually bubbling up to the root, which can then report success of failure back to the RPC command that initiated the whole process.
2020-05-11 20:02:16 +02:00
if (!payment_is_finished(p))
return;
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
/* Should we continue bubbling up? */
paymod: Implement bubbling results up the hierarchy of payments A payment is considered finished if it is in a final state (success or failure) or all its sub-payments are finished. If that's the case we notify `payment_finished` and bubble up through `payment_child_finished`, eventually bubbling up to the root, which can then report success of failure back to the RPC command that initiated the whole process.
2020-05-11 20:02:16 +02:00
payment_finished(p);
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
}
paymod: Add attempts array to `pay` return value
2020-06-09 15:41:05 +02:00
static void payment_add_attempt(struct json_stream *s, const char *fieldname, struct payment *p, bool recurse)
{
bool finished = p->step >= PAYMENT_STEP_RETRY,
success = p->step == PAYMENT_STEP_SUCCESS;
/* A fieldname is only reasonable if we're not recursing. Otherwise the
* fieldname would be reused for all attempts. */
assert(!recurse || fieldname == NULL);
json_object_start(s, fieldname);
if (!finished)
json_add_string(s, "status", "pending");
else if (success)
json_add_string(s, "status", "success");
else
json_add_string(s, "status", "failed");
if (p->failreason != NULL)
json_add_string(s, "failreason", p->failreason);
paymod: Add a bit more information to the `pay` response Amount, parent_part_id and own partid can be helpful when debugging.
2020-07-12 14:30:49 +02:00
json_add_u64(s, "partid", p->partid);
json_add_amount_msat_only(s, "amount", p->amount);
if (p->parent != NULL)
json_add_u64(s, "parent_partid", p->parent->partid);
paymod: Add attempts array to `pay` return value
2020-06-09 15:41:05 +02:00
json_object_end(s);
for (size_t i=0; i<tal_count(p->children); i++) {
payment_add_attempt(s, fieldname, p->children[i], recurse);
}
}
static void payment_json_add_attempts(struct json_stream *s,
const char *fieldname, struct payment *p)
{
assert(p == payment_root(p));
json_array_start(s, fieldname);
payment_add_attempt(s, NULL, p, true);
json_array_end(s);
}
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
/* This function is called whenever a payment ends up in a final state, or all
* leafs in the subtree rooted in the payment are all in a final state. It is
* called only once, and it is guaranteed to be called in post-order
* traversal, i.e., all children are finished before the parent is called. */
static void payment_finished(struct payment *p)
{
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
struct payment_tree_result result = payment_collect_result(p);
struct json_stream *ret;
struct command *cmd = p->cmd;
pay: Do not duplicate "message" field in response to `pay`
2020-07-10 11:17:41 +02:00
const char *msg;
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
/* Either none of the leaf attempts succeeded yet, or we have a
* preimage. */
assert((result.leafstates & PAYMENT_STEP_SUCCESS) == 0 ||
result.preimage != NULL);
plugin: Do not return multiple times from `pay` While we were unsetting the `payment->cmd` in case of a success to signal that we should not return to the JSON-RPC command twice, we were not doing that in the case of failures. This was causing multiple responses to a single incoming command, and `lightningd` was correctly killing the plugin. This issue was introduced through early returns (anything setting `payment->abort=true`) and was caused in Rusty's case through an MPP timeout. Fixes #3847 Reported-by: Rusty Russell <@rustyrussell> Signed-off-by: Christian Decker <@cdecker>
2020-07-18 08:37:28 +02:00
if (p->parent == NULL) {
/* We are about to reply, unset the pointer to the cmd so we
* don't attempt to return a response twice. */
p->cmd = NULL;
if (cmd == NULL) {
/* This is the tree root, but we already reported
* success or failure, so noop. */
return;
} else if (payment_is_success(p)) {
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
assert(result.treestates & PAYMENT_STEP_SUCCESS);
assert(result.leafstates & PAYMENT_STEP_SUCCESS);
assert(result.preimage != NULL);
plugin: Do not return multiple times from `pay` While we were unsetting the `payment->cmd` in case of a success to signal that we should not return to the JSON-RPC command twice, we were not doing that in the case of failures. This was causing multiple responses to a single incoming command, and `lightningd` was correctly killing the plugin. This issue was introduced through early returns (anything setting `payment->abort=true`) and was caused in Rusty's case through an MPP timeout. Fixes #3847 Reported-by: Rusty Russell <@rustyrussell> Signed-off-by: Christian Decker <@cdecker>
2020-07-18 08:37:28 +02:00
ret = jsonrpc_stream_success(cmd);
paymod: Add attempts array to `pay` return value
2020-06-09 15:41:05 +02:00
json_add_node_id(ret, "destination", p->destination);
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
json_add_sha256(ret, "payment_hash", p->payment_hash);
paymod: Add attempts array to `pay` return value
2020-06-09 15:41:05 +02:00
json_add_timeabs(ret, "created_at", p->start_time);
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
json_add_num(ret, "parts", result.attempts);
json_add_amount_msat_compat(ret, p->amount, "msatoshi",
"amount_msat");
json_add_amount_msat_compat(ret, result.sent,
"msatoshi_sent",
"amount_sent_msat");
if (result.leafstates != PAYMENT_STEP_SUCCESS)
json_add_string(
JSON API: fix up two existing warnings to be conformant. Technically an API break, but nobody relies on these I hope! Note that the feerates warning was buried inside the style object: it should be top-level. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-14 08:27:50 +02:00
ret, "warning_partial_completion",
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
"Some parts of the payment are not yet "
"completed, but we have the confirmation "
"from the recipient.");
json_add_preimage(ret, "payment_preimage", result.preimage);
json_add_string(ret, "status", "complete");
if (command_finished(cmd, ret)) {/* Ignore result. */}
return;
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
} else if (result.failure == NULL || result.failure->failcode < NODE) {
/* This is failing because we have no more routes to try */
pay: Do not duplicate "message" field in response to `pay`
2020-07-10 11:17:41 +02:00
msg = tal_fmt(cmd,
"Ran out of routes to try after "
"%d attempt%s: see `paystatus`",
result.attempts,
result.attempts == 1 ? "" : "s");
paymod: Activate paymod and move legacy pay to `legacypay` command As suggested during the paymod-03 review it is better to activate the new code right away, and give users an escape hatch to use the legacy code instead. The way I implemented it allows using either `legacypay` or `pay` and then set `legacy` to switch to the other implementation. Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required. Suggested-by: Rusty Russell <@rustyrussell> Suggested-by: ZmnSCPxj <@zmnscpxj>
2020-07-02 14:52:16 +02:00
ret = jsonrpc_stream_fail(cmd, PAY_STOPPED_RETRYING,
pay: Do not duplicate "message" field in response to `pay`
2020-07-10 11:17:41 +02:00
msg);
paymod: Add attempts array to `pay` return value
2020-06-09 15:41:05 +02:00
payment_json_add_attempts(ret, "attempts", p);
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
if (command_finished(cmd, ret)) {/* Ignore result. */}
return;
} else {
struct payment_result *failure = result.failure;
assert(failure!= NULL);
paymod: Don't assume that the first payment was executed at all With the `presplit`-modifier we actually skip execution of the root altogether which results in the root not having a result at all. Instead we should use the result returned by `payment_collect_result`.
2020-07-03 17:22:58 +02:00
ret = jsonrpc_stream_fail(cmd, failure->code,
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
failure->message);
paymod: Don't assume that the first payment was executed at all With the `presplit`-modifier we actually skip execution of the root altogether which results in the root not having a result at all. Instead we should use the result returned by `payment_collect_result`.
2020-07-03 17:22:58 +02:00
json_add_u64(ret, "id", failure->id);
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
paymod: Don't assume that the first payment was executed at all With the `presplit`-modifier we actually skip execution of the root altogether which results in the root not having a result at all. Instead we should use the result returned by `payment_collect_result`.
2020-07-03 17:22:58 +02:00
json_add_u32(ret, "failcode", failure->failcode);
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
json_add_string(ret, "failcodename",
paymod: Don't assume that the first payment was executed at all With the `presplit`-modifier we actually skip execution of the root altogether which results in the root not having a result at all. Instead we should use the result returned by `payment_collect_result`.
2020-07-03 17:22:58 +02:00
failure->failcodename);
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
if (p->bolt11)
json_add_string(ret, "bolt11", p->bolt11);
json_add_hex_talarr(ret, "raw_message",
paymod: Don't assume that the first payment was executed at all With the `presplit`-modifier we actually skip execution of the root altogether which results in the root not having a result at all. Instead we should use the result returned by `payment_collect_result`.
2020-07-03 17:22:58 +02:00
result.failure->raw_message);
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
json_add_num(ret, "created_at", p->start_time.ts.tv_sec);
json_add_node_id(ret, "destination", p->destination);
json_add_sha256(ret, "payment_hash", p->payment_hash);
if (result.leafstates & PAYMENT_STEP_SUCCESS) {
/* If one sub-payment succeeded then we have
* proof of payment, and the payment is a
* success. */
json_add_string(ret, "status", "complete");
} else if (result.leafstates & ~PAYMENT_FAILED) {
/* If there are non-failed leafs we are still trying. */
json_add_string(ret, "status", "pending");
} else {
json_add_string(ret, "status", "failed");
}
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
paymod: Fill in the `pay` return value
2020-05-28 15:13:43 +02:00
json_add_amount_msat_compat(ret, p->amount, "msatoshi",
"amount_msat");
json_add_amount_msat_compat(ret, result.sent,
"msatoshi_sent",
"amount_sent_msat");
if (failure != NULL) {
if (failure->erring_index)
json_add_num(ret, "erring_index",
*failure->erring_index);
if (failure->erring_node)
json_add_node_id(ret, "erring_node",
failure->erring_node);
if (failure->erring_channel)
json_add_short_channel_id(
ret, "erring_channel",
failure->erring_channel);
if (failure->erring_direction)
json_add_num(
ret, "erring_direction",
*failure->erring_direction);
}
if (command_finished(cmd, ret)) {/* Ignore result. */}
paymod: Collect and return results of a tree of partial payments The status of what started as a simple JSON-RPC call is now spread across an entire tree of partial payments and payment attempts. So we collect the status in a single struct in order to report back success of failure.
2020-05-18 14:44:45 +02:00
return;
}
} else {
payment_child_finished(p->parent, p);
return;
}
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
}
paymod: Add function to set the current step This is necessary otherwise we would not be calling modifiers for the newly set state which can lead to unexpected results.
2020-06-09 15:52:50 +02:00
void payment_set_step(struct payment *p, enum payment_step newstep)
{
p->current_modifier = -1;
p->step = newstep;
plugin: Always set an end_time for payments in a final state Reported-by: @thestick613 Fixes #3848
2020-07-18 15:35:57 +02:00
/* Any final state needs an end_time */
if (p->step >= PAYMENT_STEP_SPLIT)
p->end_time = time_now();
paymod: Add function to set the current step This is necessary otherwise we would not be calling modifiers for the newly set state which can lead to unexpected results.
2020-06-09 15:52:50 +02:00
}
paymod: Implement base state-machine transitions The actual steps are mocked out, but we can already run through the various phases of a payment, and the modifiers should be called for each state.
2020-05-08 14:48:40 +02:00
void payment_continue(struct payment *p)
{
struct payment_modifier *mod;
void *moddata;
/* If we are in the middle of calling the modifiers, continue calling
* them, otherwise we can continue with the payment state-machine. */
p->current_modifier++;
mod = p->modifiers[p->current_modifier];
if (mod != NULL) {
/* There is another modifier, so call it. */
moddata = p->modifier_data[p->current_modifier];
return mod->post_step_cb(moddata, p);
} else {
/* There are no more modifiers, so reset the call chain and
* proceed to the next state. */
p->current_modifier = -1;
switch (p->step) {
case PAYMENT_STEP_INITIALIZED:
payment_getroute(p);
return;
case PAYMENT_STEP_GOT_ROUTE:
payment_compute_onion_payloads(p);
return;
case PAYMENT_STEP_ONION_PAYLOAD:
payment_sendonion(p);
return;
case PAYMENT_STEP_SUCCESS:
case PAYMENT_STEP_FAILED:
payment_finished(p);
return;
case PAYMENT_STEP_RETRY:
case PAYMENT_STEP_SPLIT:
/* Do nothing, we'll get pinged by a child succeeding
* or failing. */
return;
}
}
/* We should never get here, it'd mean one of the state machine called
* `payment_continue` after the final state. */
abort();
}
paymod: Add a simple test-command to test the paymod state-machine This commit can be reverted/skipped once we have implemented all the logic and have feature parity with the normal `pay`. It's main purpose is to expose the unfinished functionality to test it, without completely breaking the existing `pay` command.
2020-05-08 15:03:48 +02:00
paymod: Add a human readable failreason to payments This makes it easier to stash a human readable failure message in an attempt.
2020-06-09 15:38:16 +02:00
void payment_fail(struct payment *p, const char *fmt, ...)
paymod: Make the transition to FAILED a function call We need to set some variables as well, and the transition to FAILED is a bit special in that quite some modifiers will skip into FAILED.
2020-05-29 15:16:40 +02:00
{
paymod: Add function to set the current step This is necessary otherwise we would not be calling modifiers for the newly set state which can lead to unexpected results.
2020-06-09 15:52:50 +02:00
va_list ap;
paymod: Make the transition to FAILED a function call We need to set some variables as well, and the transition to FAILED is a bit special in that quite some modifiers will skip into FAILED.
2020-05-29 15:16:40 +02:00
p->end_time = time_now();
paymod: Add function to set the current step This is necessary otherwise we would not be calling modifiers for the newly set state which can lead to unexpected results.
2020-06-09 15:52:50 +02:00
payment_set_step(p, PAYMENT_STEP_FAILED);
va_start(ap, fmt);
p->failreason = tal_vfmt(p, fmt, ap);
va_end(ap);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_INFORM, "%s", p->failreason);
paymod: Add function to set the current step This is necessary otherwise we would not be calling modifiers for the newly set state which can lead to unexpected results.
2020-06-09 15:52:50 +02:00
paymod: Make the transition to FAILED a function call We need to set some variables as well, and the transition to FAILED is a bit special in that quite some modifiers will skip into FAILED.
2020-05-29 15:16:40 +02:00
payment_continue(p);
}
paymod: Generate type-safe accessor functions for modifier data This should make it easy for JSON-RPC functions and modifiers to get the associated data for a given modifier name. Useful if a modifier needs to access its parent's modifier data, or in other functions that need to access modifier data, e.g., when passing destination pointers into the `param()` call.
2020-05-08 16:46:56 +02:00
void *payment_mod_get_data(const struct payment *p,
const struct payment_modifier *mod)
{
for (size_t i = 0; p->modifiers[i] != NULL; i++)
if (p->modifiers[i] == mod)
return p->modifier_data[i];
/* If we ever get here it means that we asked for the data for a
* non-existent modifier. This is a compile-time/wiring issue, so we
* better check that modifiers match the data we ask for. */
abort();
}
paymod: Add a retry modifier that retries payments on failure This is likely a bit of overkill for this type of functionality, but it is a nice first use-case of how functionality can be compartmentalized into modifiers. If makes swapping retry mechanisms in and out really simple.
2020-05-08 16:51:43 +02:00
static struct retry_mod_data *retry_data_init(struct payment *p);
static inline void retry_step_cb(struct retry_mod_data *rd,
struct payment *p);
static struct retry_mod_data *
retry_data_init(struct payment *p)
{
struct retry_mod_data *rdata = tal(p, struct retry_mod_data);
struct retry_mod_data *parent_rdata;
retrymod: Reset retry counter if parent is a split If the parent is a split we have new payment parameters, and want to perform a number of attempts with those.
2020-07-06 20:27:30 +02:00
/* We start the retry counter from scratch for the root payment, or if
* the parent was split, meaning this is a new attempt with new
* amounts. */
if (p->parent == NULL || p->parent->step == PAYMENT_STEP_SPLIT) {
rdata->retries = 10;
} else {
paymod: Add a retry modifier that retries payments on failure This is likely a bit of overkill for this type of functionality, but it is a nice first use-case of how functionality can be compartmentalized into modifiers. If makes swapping retry mechanisms in and out really simple.
2020-05-08 16:51:43 +02:00
parent_rdata = payment_mod_retry_get_data(p->parent);
rdata->retries = parent_rdata->retries - 1;
}
return rdata;
}
paymod: Simplify retry mod logic and add abort logic
2020-05-26 18:59:41 +02:00
/* Determine whether retrying could possibly succeed. Retrying in this case
* means that we repeat the entire flow, including computing a new route, new
* payload and a new sendonion call. It does not mean we retry the exact same
* attempt that just failed. */
static bool payment_can_retry(struct payment *p)
{
struct payment_result *res = p->result;
u32 idx;
bool is_final;
if (p->result == NULL)
paymod: Routehintmod signals that we can retry if getroute fails The shortcut in the retry_mod that we can skip retrying if getroute fails or we have no result is only valid if the parameters don't change. As we iterate through the routehints the parameters change, and so we must signal to the retry_mod that it can retry even in those cases.
2020-07-23 16:07:39 +02:00
return p->failroute_retry;
paymod: Simplify retry mod logic and add abort logic
2020-05-26 18:59:41 +02:00
idx = res->erring_index != NULL ? *res->erring_index : 0;
is_final = (idx == tal_count(p->route));
/* Full matrix of failure code x is_final. Prefer to retry once too
* often over eagerly failing. */
switch (res->failcode) {
case WIRE_EXPIRY_TOO_FAR:
case WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS:
case WIRE_INVALID_ONION_PAYLOAD:
case WIRE_INVALID_ONION_VERSION:
case WIRE_INVALID_REALM:
case WIRE_MPP_TIMEOUT:
case WIRE_PERMANENT_NODE_FAILURE:
case WIRE_REQUIRED_NODE_FEATURE_MISSING:
case WIRE_TEMPORARY_NODE_FAILURE:
case WIRE_UNKNOWN_NEXT_PEER:
return !is_final;
case WIRE_AMOUNT_BELOW_MINIMUM:
case WIRE_CHANNEL_DISABLED:
case WIRE_EXPIRY_TOO_SOON:
case WIRE_FEE_INSUFFICIENT:
case WIRE_FINAL_INCORRECT_CLTV_EXPIRY:
case WIRE_FINAL_INCORRECT_HTLC_AMOUNT:
case WIRE_INCORRECT_CLTV_EXPIRY:
case WIRE_INVALID_ONION_HMAC:
case WIRE_INVALID_ONION_KEY:
case WIRE_PERMANENT_CHANNEL_FAILURE:
case WIRE_REQUIRED_CHANNEL_FEATURE_MISSING:
case WIRE_TEMPORARY_CHANNEL_FAILURE:
#if EXPERIMENTAL_FEATURES
case WIRE_INVALID_ONION_BLINDING:
#endif
return true;
}
/* We should never get here, otherwise the above `switch` isn't
* exhaustive. Nevertheless the failcode is provided by the erring
* node, so retry anyway. `abort()`ing on externally supplied info is
* not a good idea. */
return true;
}
paymod: Add a retry modifier that retries payments on failure This is likely a bit of overkill for this type of functionality, but it is a nice first use-case of how functionality can be compartmentalized into modifiers. If makes swapping retry mechanisms in and out really simple.
2020-05-08 16:51:43 +02:00
static inline void retry_step_cb(struct retry_mod_data *rd,
struct payment *p)
{
paymod: Set the STOPPED_RETRYING status code if we stopped retrying
2020-07-06 22:39:04 +02:00
struct payment *subpayment, *root = payment_root(p);
paymod: Add a retry modifier that retries payments on failure This is likely a bit of overkill for this type of functionality, but it is a nice first use-case of how functionality can be compartmentalized into modifiers. If makes swapping retry mechanisms in and out really simple.
2020-05-08 16:51:43 +02:00
struct retry_mod_data *rdata = payment_mod_retry_get_data(p);
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
struct timeabs now = time_now();
paymod: Teach the retry_mod not to insist when it's futile An important life lesson: if there is no path to happiness, then trying harder will still not get you there.
2020-05-22 17:44:47 +02:00
paymod: Simplify retry mod logic and add abort logic
2020-05-26 18:59:41 +02:00
if (p->step != PAYMENT_STEP_FAILED)
return payment_continue(p);
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
if (time_after(now, p->deadline)) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_INFORM,
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
"Payment deadline expired, not retrying (partial-)payment "
"%s/%d",
type_to_string(tmpctx, struct sha256, p->payment_hash),
p->partid);
paymod: Set the STOPPED_RETRYING status code if we stopped retrying
2020-07-06 22:39:04 +02:00
root->abort = true;
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
return payment_continue(p);
}
paymod: Teach the retry_mod not to insist when it's futile An important life lesson: if there is no path to happiness, then trying harder will still not get you there.
2020-05-22 17:44:47 +02:00
/* If we failed to find a route, it's unlikely we can suddenly find a
* new one without any other changes, so it's time to give up. */
paymod: Routehintmod signals that we can retry if getroute fails The shortcut in the retry_mod that we can skip retrying if getroute fails or we have no result is only valid if the parameters don't change. As we iterate through the routehints the parameters change, and so we must signal to the retry_mod that it can retry even in those cases.
2020-07-23 16:07:39 +02:00
if (p->route == NULL && !p->failroute_retry)
paymod: Simplify retry mod logic and add abort logic
2020-05-26 18:59:41 +02:00
return payment_continue(p);
paymod: Teach the retry_mod not to insist when it's futile An important life lesson: if there is no path to happiness, then trying harder will still not get you there.
2020-05-22 17:44:47 +02:00
paymod: Simplify retry mod logic and add abort logic
2020-05-26 18:59:41 +02:00
/* If the root is marked as abort, we do not retry anymore */
if (payment_root(p)->abort)
return payment_continue(p);
if (!payment_can_retry(p))
return payment_continue(p);
/* If the failure was not final, and we tried a route, try again. */
if (rdata->retries > 0) {
paymod: Update step before creating child payments The child payments will sometimes depend on the step of the parent, and making sure that the parent state is correct before we create the children is therefore important.
2020-07-23 16:02:02 +02:00
payment_set_step(p, PAYMENT_STEP_RETRY);
paymod: Do not pass cmd to sub-payments, plugin suffices We were passing the `cmd` instance around where `plugin` suffices (used to start RPC calls and logs).
2020-05-18 20:44:21 +02:00
subpayment = payment_new(p, NULL, p, p->modifiers);
paymod: Add a retry modifier that retries payments on failure This is likely a bit of overkill for this type of functionality, but it is a nice first use-case of how functionality can be compartmentalized into modifiers. If makes swapping retry mechanisms in and out really simple.
2020-05-08 16:51:43 +02:00
payment_start(subpayment);
paymod: Add reason why a payment was attempted This is a slight change in interface in that the string no longer spells out that a specific node was excluded.
2020-05-29 17:08:20 +02:00
subpayment->why =
tal_fmt(subpayment, "Still have %d attempts left",
rdata->retries - 1);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_DBG,
retrymod: Make retry modifier slightly more verbose I found it rather useful to trace how a payment is getting retried in the logs.
2020-07-13 17:30:51 +02:00
"Retrying %s/%d (%s), new partid %d. %d attempts left\n",
type_to_string(tmpctx, struct sha256, p->payment_hash),
p->partid,
type_to_string(tmpctx, struct amount_msat, &p->amount),
subpayment->partid,
rdata->retries - 1);
paymod: Add a retry modifier that retries payments on failure This is likely a bit of overkill for this type of functionality, but it is a nice first use-case of how functionality can be compartmentalized into modifiers. If makes swapping retry mechanisms in and out really simple.
2020-05-08 16:51:43 +02:00
}
payment_continue(p);
}
paymod: Add a local_channel_hints modifier to collect local channels We can have quite detailed information about our local channels, so call `listpeers` before the `getroute` call on the root payment, to seed that information in the channel_hints.
2020-05-22 17:20:36 +02:00
paymod: Simplify retry mod logic and add abort logic
2020-05-26 18:59:41 +02:00
REGISTER_PAYMENT_MODIFIER(retry, struct retry_mod_data *, retry_data_init,
retry_step_cb);
paymod: Add a local_channel_hints modifier to collect local channels We can have quite detailed information about our local channels, so call `listpeers` before the `getroute` call on the root payment, to seed that information in the channel_hints.
2020-05-22 17:20:36 +02:00
static struct command_result *
local_channel_hints_listpeers(struct command *cmd, const char *buffer,
const jsmntok_t *toks, struct payment *p)
{
paymod: Track how many HTLCs each channel can still add It turns out that by aggressively splitting payments we may end up exhausting the number of HTLCs we can add to a channel quickly. By tracking the number of HTLCs we can still add, and excluding the channels to which we cannot add any more we increase the route diversity, and avoid quickly exhausting the HTLC budget. In the next commit we'll also implement an early abort if we've exhausted all channels, so we don't end up splitting indefinitely and we can also optimize the initial split to not run afoul of that limit.
2020-07-20 20:28:31 +02:00
const jsmntok_t *peers, *peer, *channels, *channel, *spendsats, *scid,
*dir, *connected, *max_htlc, *htlcs;
paymod: Add a local_channel_hints modifier to collect local channels We can have quite detailed information about our local channels, so call `listpeers` before the `getroute` call on the root payment, to seed that information in the channel_hints.
2020-05-22 17:20:36 +02:00
size_t i, j;
peers = json_get_member(buffer, toks, "peers");
if (peers == NULL)
goto done;
/* cppcheck-suppress uninitvar - cppcheck can't undestand these macros. */
json_for_each_arr(i, peer, peers) {
channels = json_get_member(buffer, peer, "channels");
if (channels == NULL)
continue;
connected = json_get_member(buffer, peer, "connected");
json_for_each_arr(j, channel, channels) {
struct channel_hint h;
spendsats = json_get_member(buffer, channel, "spendable_msat");
scid = json_get_member(buffer, channel, "short_channel_id");
dir = json_get_member(buffer, channel, "direction");
paymod: Track how many HTLCs each channel can still add It turns out that by aggressively splitting payments we may end up exhausting the number of HTLCs we can add to a channel quickly. By tracking the number of HTLCs we can still add, and excluding the channels to which we cannot add any more we increase the route diversity, and avoid quickly exhausting the HTLC budget. In the next commit we'll also implement an early abort if we've exhausted all channels, so we don't end up splitting indefinitely and we can also optimize the initial split to not run afoul of that limit.
2020-07-20 20:28:31 +02:00
max_htlc = json_get_member(buffer, channel, "max_accepted_htlcs");
htlcs = json_get_member(buffer, channel, "htlcs");
if (spendsats == NULL || scid == NULL || dir == NULL ||
max_htlc == NULL ||
max_htlc->type != JSMN_PRIMITIVE || htlcs == NULL ||
htlcs->type != JSMN_ARRAY)
Fixed assertion in pay plugin This PR includes the fix discussed on PR #3855. This fix was tested with the use case described inside the issue and worked. Fixes: #3855 Changelog-None
2020-07-19 17:09:24 +02:00
continue;
paymod: Add a local_channel_hints modifier to collect local channels We can have quite detailed information about our local channels, so call `listpeers` before the `getroute` call on the root payment, to seed that information in the channel_hints.
2020-05-22 17:20:36 +02:00
json_to_bool(buffer, connected, &h.enabled);
json_to_short_channel_id(buffer, scid, &h.scid.scid);
json_to_int(buffer, dir, &h.scid.dir);
json_to_msat(buffer, spendsats, &h.estimated_capacity);
paymod: Track how many HTLCs each channel can still add It turns out that by aggressively splitting payments we may end up exhausting the number of HTLCs we can add to a channel quickly. By tracking the number of HTLCs we can still add, and excluding the channels to which we cannot add any more we increase the route diversity, and avoid quickly exhausting the HTLC budget. In the next commit we'll also implement an early abort if we've exhausted all channels, so we don't end up splitting indefinitely and we can also optimize the initial split to not run afoul of that limit.
2020-07-20 20:28:31 +02:00
/* Take the configured number of max_htlcs and
* subtract any HTLCs that might already be added to
* the channel. This is a best effort estimate and
* mostly considers stuck htlcs, concurrent payments
* may throw us off a bit. */
json_to_u16(buffer, max_htlc, &h.htlc_budget);
h.htlc_budget -= htlcs->size;
h.local = true;
paymod: Consolidate channel_hint creation in channel_hints_update As the hints get new fields added it is easy to forget to amend one of the places we create them, since we already have an update method let's use that to handle all additions to the array of known channel hints.
2020-07-22 11:36:55 +02:00
channel_hints_update(p, h.scid.scid, h.scid.dir,
h.enabled, true, &h.estimated_capacity, &h.htlc_budget);
paymod: Add a local_channel_hints modifier to collect local channels We can have quite detailed information about our local channels, so call `listpeers` before the `getroute` call on the root payment, to seed that information in the channel_hints.
2020-05-22 17:20:36 +02:00
}
}
done:
payment_continue(p);
return command_still_pending(cmd);
}
static void local_channel_hints_cb(void *d UNUSED, struct payment *p)
{
struct out_req *req;
/* If we are not the root we don't look up the channel balances since
* it is unlikely that the capacities have changed much since the root
* payment looked at them. We also only call `listpeers` when the
* payment is in state PAYMENT_STEP_INITIALIZED, right before calling
* `getroute`. */
if (p->parent != NULL || p->step != PAYMENT_STEP_INITIALIZED)
return payment_continue(p);
req = jsonrpc_request_start(p->plugin, NULL, "listpeers",
local_channel_hints_listpeers,
local_channel_hints_listpeers, p);
send_outreq(p->plugin, req);
}
REGISTER_PAYMENT_MODIFIER(local_channel_hints, void *, NULL, local_channel_hints_cb);
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
/* Trim route to this length by taking from the *front* of route
* (end points to destination, so we need that bit!) */
static void trim_route(struct route_info **route, size_t n)
{
size_t remove = tal_count(*route) - n;
memmove(*route, *route + remove, sizeof(**route) * n);
tal_resize(route, n);
}
/* Make sure routehints are reasonable length, and (since we assume we
* can append), not directly to us. Note: untrusted data! */
static struct route_info **filter_routehints(struct routehints_data *d,
struct node_id *myid,
struct route_info **hints)
{
char *mods = tal_strdup(tmpctx, "");
for (size_t i = 0; i < tal_count(hints); i++) {
/* Trim any routehint > 10 hops */
size_t max_hops = ROUTING_MAX_HOPS / 2;
if (tal_count(hints[i]) > max_hops) {
tal_append_fmt(&mods,
"Trimmed routehint %zu (%zu hops) to %zu. ",
i, tal_count(hints[i]), max_hops);
trim_route(&hints[i], max_hops);
}
/* If we are first hop, trim. */
if (tal_count(hints[i]) > 0
&& node_id_eq(&hints[i][0].pubkey, myid)) {
tal_append_fmt(&mods,
"Removed ourselves from routehint %zu. ",
i);
trim_route(&hints[i], tal_count(hints[i])-1);
}
/* If route is empty, remove altogether. */
if (tal_count(hints[i]) == 0) {
tal_append_fmt(&mods,
"Removed empty routehint %zu. ", i);
tal_arr_remove(&hints, i);
i--;
}
}
if (!streq(mods, ""))
d->routehint_modifications = tal_steal(d, mods);
return tal_steal(d, hints);
}
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
static bool route_msatoshi(struct amount_msat *total,
const struct amount_msat msat,
const struct route_info *route, size_t num_route);
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
static bool routehint_excluded(struct payment *p,
const struct route_info *routehint)
{
const struct node_id *nodes = payment_get_excluded_nodes(tmpctx, p);
const struct short_channel_id_dir *chans =
payment_get_excluded_channels(tmpctx, p);
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
const struct channel_hint *hints = payment_root(p)->channel_hints;
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
/* Note that we ignore direction here: in theory, we could have
* found that one direction of a channel is unavailable, but they
* are suggesting we use it the other way. Very unlikely though! */
for (size_t i = 0; i < tal_count(routehint); i++) {
const struct route_info *r = &routehint[i];
paymod: fix typo which can cause memory overrun. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-23 06:50:10 +02:00
for (size_t j = 0; j < tal_count(nodes); j++)
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
if (node_id_eq(&r->pubkey, &nodes[j]))
return true;
for (size_t j = 0; j < tal_count(chans); j++)
if (short_channel_id_eq(&chans[j].scid, &r->short_channel_id))
return true;
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
/* Skip the capacity check if this is the last hop
* in the routehint.
* The last hop in the routehint delivers the exact
* final amount to the destination, which
* payment_get_excluded_channels uses for excluding
* already.
* Thus, the capacity check below only really matters
* for multi-hop routehints.
*/
if (i == tal_count(routehint) - 1)
continue;
/* Check our capacity fits. */
struct amount_msat needed_capacity;
if (!route_msatoshi(&needed_capacity, p->amount,
r + 1, tal_count(routehint) - i - 1))
return true;
/* Why do we scan the hints again if
* payment_get_excluded_channels already does?
* Because payment_get_excluded_channels checks the
* amount at destination, but we know that we are
* a specific distance from the destination and we
* know the exact capacity we need to send via this
* channel, which is greater than the destination.
*/
for (size_t j = 0; j < tal_count(hints); j++) {
if (!short_channel_id_eq(&hints[j].scid.scid, &r->short_channel_id))
continue;
/* We exclude on equality because we set the estimate
* to the smallest failed attempt. */
if (amount_msat_greater_eq(needed_capacity,
hints[j].estimated_capacity))
return true;
}
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
}
return false;
}
static struct route_info *next_routehint(struct routehints_data *d,
struct payment *p)
{
paymod: Randomly select a routehint, or none at random The adaptive MPP test was showing an issue with always using a routehint, even when it wasn't necessary: we would insist on routhing to the entrypoint of the routehint, even through the actual destination. If a channel on that loop would result being over capacity we'd slam below 0, and then increase again by unapplying the route. The solution really is not to insist on routing through a routehint, so we implement random skipping of routehints, and we rotate them if we have multiples.
2020-07-22 16:06:39 +02:00
size_t numhints = tal_count(d->routehints);
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
struct route_info *curr;
paymod: Randomly select a routehint, or none at random The adaptive MPP test was showing an issue with always using a routehint, even when it wasn't necessary: we would insist on routhing to the entrypoint of the routehint, even through the actual destination. If a channel on that loop would result being over capacity we'd slam below 0, and then increase again by unapplying the route. The solution really is not to insist on routing through a routehint, so we implement random skipping of routehints, and we rotate them if we have multiples.
2020-07-22 16:06:39 +02:00
paymod: Simplify routehint data initialization It was spread over the step callback, but we only need to initialize the routehints array there, child-payments can just inherit most of the information.
2020-07-23 11:07:08 +02:00
if (d->routehints == NULL || numhints == 0)
return NULL;
plugins/pay: Exclude the entrypoint to a routehint to avoid cycles This uses @cdecker's idea of excluding the routehinted channel from the route, and also consumes the route hints as it goes so that it makes progress. I don't know if this is correct, but it reliably passes tests/test_pay.py::test_tlv_or_legacy now.
2020-07-23 06:50:10 +02:00
/* BOLT #11:
*
* - if a writer offers more than one of any field type, it:
* - MUST specify the most-preferred field first, followed
* by less-preferred fields, in order.
*/
plugins/libplugin-pay.c: Round-robin routehints when splitting. This improves the success rate of `test_mpp_interference_2`, though still not quite up to the level that we can remove `@flaky` from it.
2020-08-11 11:29:08 +02:00
for (; d->offset < numhints; d->offset++) {
curr = d->routehints[(d->base + d->offset) % numhints];
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
if (curr == NULL || !routehint_excluded(p, curr))
return curr;
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
}
return NULL;
}
/* Calculate how many millisatoshi we need at the start of this route
* to get msatoshi to the end. */
static bool route_msatoshi(struct amount_msat *total,
const struct amount_msat msat,
const struct route_info *route, size_t num_route)
{
*total = msat;
for (ssize_t i = num_route - 1; i >= 0; i--) {
if (!amount_msat_add_fee(total,
route[i].fee_base_msat,
route[i].fee_proportional_millionths))
return false;
}
return true;
}
/* The pubkey to use is the destination of this routehint. */
static const struct node_id *route_pubkey(const struct payment *p,
const struct route_info *routehint,
size_t n)
{
if (n == tal_count(routehint))
return p->destination;
return &routehint[n].pubkey;
}
static u32 route_cltv(u32 cltv,
const struct route_info *route, size_t num_route)
{
for (size_t i = 0; i < num_route; i++)
cltv += route[i].cltv_expiry_delta;
return cltv;
}
plugins/libplugin-pay.c: Properly handle exclusions for routehints with two hops or more. Arguably a low-priority bug since no current node ever generates routehints longer than one hop. However, it is possible as an edge case, if the destination is directly accessible *and* supports multiple channels, that we route through the destination, one of the *other* channels it has not in the routehint, to the entry point, and then through the routehint. This change removes the risk of the above edge case. Changelog-None: arguably a low-priority bug.
2020-07-31 09:59:20 +02:00
/** routehint_generate_exclusion_list
*
* @brief generate a list of items to append to `excludes`
* parameter of `getroute`.
*
* @param ctx - the context to allocate off of.
* @param routehint - the actual routehint, a `tal` array.
* @param payment - the payment that we will create an
* exclusion list for.
*
* @return an array of strings that will be appended to the
* `excludes` parameter of `getroute`.
*/
static
const char **routehint_generate_exclusion_list(const tal_t *ctx,
struct route_info *routehint,
struct payment *payment)
{
const char **exc;
size_t i;
if (!routehint || tal_count(routehint) == 0)
/* Nothing to exclude. */
return NULL;
exc = tal_arr(ctx, const char *, 0);
/* Exclude every node except the first, because the first is
* the entry point to the routehint. */
for (i = 1 /* Skip the first! */; i < tal_count(routehint); ++i)
tal_arr_expand(&exc,
type_to_string(exc, struct node_id,
&routehint[i].pubkey));
/* Also exclude the destination, because it would be foolish to
* pass through it and *then* go to the routehint entry point. */
tal_arr_expand(&exc,
type_to_string(exc, struct node_id,
payment->destination));
return exc;
}
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
/* Change the destination and compute the final msatoshi amount to send to the
* routehint entry point. */
static void routehint_pre_getroute(struct routehints_data *d, struct payment *p)
{
paymod: Routehintmod signals that we can retry if getroute fails The shortcut in the retry_mod that we can skip retrying if getroute fails or we have no result is only valid if the parameters don't change. As we iterate through the routehints the parameters change, and so we must signal to the retry_mod that it can retry even in those cases.
2020-07-23 16:07:39 +02:00
bool have_more;
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
d->current_routehint = next_routehint(d, p);
paymod: Routehintmod signals that we can retry if getroute fails The shortcut in the retry_mod that we can skip retrying if getroute fails or we have no result is only valid if the parameters don't change. As we iterate through the routehints the parameters change, and so we must signal to the retry_mod that it can retry even in those cases.
2020-07-23 16:07:39 +02:00
/* Signal that we could retry with another routehint even if getroute
* fails. */
have_more = (d->offset < tal_count(d->routehints) - 1);
p->failroute_retry = have_more;
plugins/libplugin-pay.c: Properly handle exclusions for routehints with two hops or more. Arguably a low-priority bug since no current node ever generates routehints longer than one hop. However, it is possible as an edge case, if the destination is directly accessible *and* supports multiple channels, that we route through the destination, one of the *other* channels it has not in the routehint, to the entry point, and then through the routehint. This change removes the risk of the above edge case. Changelog-None: arguably a low-priority bug.
2020-07-31 09:59:20 +02:00
p->temp_exclusion = tal_free(p->temp_exclusion);
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
if (d->current_routehint != NULL) {
if (!route_msatoshi(&p->getroute->amount, p->amount,
d->current_routehint,
tal_count(d->current_routehint))) {
}
d->final_cltv = p->getroute->cltv;
p->getroute->destination = &d->current_routehint[0].pubkey;
p->getroute->cltv =
route_cltv(p->getroute->cltv, d->current_routehint,
tal_count(d->current_routehint));
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_DBG, "Using routehint %s (%s) cltv_delta=%d",
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
type_to_string(tmpctx, struct node_id,
&d->current_routehint->pubkey),
type_to_string(tmpctx, struct short_channel_id,
&d->current_routehint->short_channel_id),
d->current_routehint->cltv_expiry_delta);
plugins/pay: Exclude the entrypoint to a routehint to avoid cycles This uses @cdecker's idea of excluding the routehinted channel from the route, and also consumes the route hints as it goes so that it makes progress. I don't know if this is correct, but it reliably passes tests/test_pay.py::test_tlv_or_legacy now.
2020-07-23 06:50:10 +02:00
/* Exclude the entrypoint to the routehint, so we don't end up
* going through the destination to the entrypoint. */
plugins/libplugin-pay.c: Properly handle exclusions for routehints with two hops or more. Arguably a low-priority bug since no current node ever generates routehints longer than one hop. However, it is possible as an edge case, if the destination is directly accessible *and* supports multiple channels, that we route through the destination, one of the *other* channels it has not in the routehint, to the entry point, and then through the routehint. This change removes the risk of the above edge case. Changelog-None: arguably a low-priority bug.
2020-07-31 09:59:20 +02:00
p->temp_exclusion = routehint_generate_exclusion_list(p, d->current_routehint, p);
} else
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_DBG, "Not using a routehint");
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
}
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
static struct command_result *routehint_getroute_result(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
struct payment *root = payment_root(p);
const jsmntok_t *rtok = json_get_member(buffer, toks, "route");
struct routehints_data *d = payment_mod_routehints_get_data(root);
/* If there was a route the destination is reachable without
* routehints. */
d->destination_reachable = (rtok != NULL);
if (d->destination_reachable)
tal_arr_expand(&d->routehints, NULL);
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
routehint_pre_getroute(d, p);
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_DBG,
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
"The destination is%s directly reachable %s attempts "
"without routehints",
d->destination_reachable ? "" : " not",
d->destination_reachable ? "including" : "excluding");
/* Now we can continue on our merry way. */
payment_continue(p);
/* Let payment_finished_ handle this, so we mark it as pending */
return command_still_pending(cmd);
}
static void routehint_check_reachable(struct payment *p)
{
struct out_req *req;
/* Start a tiny exploratory getroute request, so we
* know whether we stand any chance of reaching the
* destination without routehints. This will later be
* used to mix in attempts without routehints. */
req = jsonrpc_request_start(p->plugin, NULL, "getroute",
routehint_getroute_result,
routehint_getroute_result, p);
json_add_node_id(req->js, "id", p->destination);
json_add_amount_msat_only(req->js, "msatoshi", AMOUNT_MSAT(1000));
json_add_num(req->js, "maxhops", 20);
json_add_num(req->js, "riskfactor", 10);
send_outreq(p->plugin, req);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_DBG,
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
"Asking gossipd whether %s is reachable "
"without routehints.",
type_to_string(tmpctx, struct node_id, p->destination));
}
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
static void routehint_step_cb(struct routehints_data *d, struct payment *p)
{
struct route_hop hop;
const struct payment *root = payment_root(p);
if (p->step == PAYMENT_STEP_INITIALIZED) {
if (root->invoice == NULL || root->invoice->routes == NULL)
return payment_continue(p);
paymod: Fix the routehints being lost when retrying We were removing the current hint from the list and not inheriting the current routehint, so we'd be forgetting a hint at each retry. Now we keep the array unchanged in the root, and simply skip the ones that are not usable given the current information we have about the channels (in the form of channel_hints). Fixes #3861
2020-07-21 22:09:54 +02:00
/* We filter out non-functional routehints once at the
* beginning, and every other payment will filter out the
* exluded ones on the fly. */
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
if (p->parent == NULL) {
d->routehints = filter_routehints(d, p->local_id,
p->invoice->routes);
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_DBG,
paymod: Check if destination is reachable at all directly at startup This does two things: it checks if the destination of the payment is at all reachable without routehints, and if it is it adds a direct attempt as option to the routehints in the form of a NULL routehint. It also simplifies the selection of the routehint since the direct case is no longer special, instead we just return a NULL routehint as if it were a normal routehint.
2020-07-23 10:52:08 +02:00
"After filtering routehints we're left with "
"%zu usable hints",
tal_count(d->routehints));
/* Do not continue normally, instead go and check if
* we can reach the destination directly. */
return routehint_check_reachable(p);
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
}
paymod: Move application of routehints into its own function We have two places we need to do that now: in the root payment after we checked if the destination is reachable, and in any other payment directly in the initialization-step callback.
2020-07-23 14:14:40 +02:00
routehint_pre_getroute(d, p);
paymod: Simplify routehint data initialization It was spread over the step callback, but we only need to initialize the routehints array there, child-payments can just inherit most of the information.
2020-07-23 11:07:08 +02:00
} else if (p->step == PAYMENT_STEP_GOT_ROUTE && d->current_routehint != NULL) {
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
/* Now it's time to stitch the two partial routes together. */
struct amount_msat dest_amount;
struct route_info *routehint = d->current_routehint;
struct route_hop *prev_hop;
for (ssize_t i = 0; i < tal_count(routehint); i++) {
prev_hop = &p->route[tal_count(p->route)-1];
if (!route_msatoshi(&dest_amount, p->amount,
routehint + i + 1,
tal_count(routehint) - i - 1)) {
/* Just let it fail, since we couldn't stitch
* the routes together. */
return payment_continue(p);
}
hop.nodeid = *route_pubkey(p, routehint, i + 1);
hop.style = ROUTE_HOP_TLV;
hop.channel_id = routehint[i].short_channel_id;
hop.amount = dest_amount;
hop.delay = route_cltv(d->final_cltv, routehint + i + 1,
tal_count(routehint) - i - 1);
/* Should we get a failure inside the routehint we'll
* need the direction so we can exclude it. Luckily
* it's rather easy to compute given the two
* subsequent hops. */
hop.direction =
node_id_cmp(&prev_hop->nodeid, &hop.nodeid) > 0 ? 1
: 0;
tal_arr_expand(&p->route, hop);
}
}
payment_continue(p);
}
static struct routehints_data *routehint_data_init(struct payment *p)
{
paymod: Simplify routehint data initialization It was spread over the step callback, but we only need to initialize the routehints array there, child-payments can just inherit most of the information.
2020-07-23 11:07:08 +02:00
struct routehints_data *pd, *d = tal(p, struct routehints_data);
/* If for some reason we skipped the getroute call (directpay) we'll
* need this to be initialized. */
d->current_routehint = NULL;
if (p->parent != NULL) {
pd = payment_mod_routehints_get_data(payment_root(p));
d->destination_reachable = pd->destination_reachable;
d->routehints = pd->routehints;
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
pd = payment_mod_routehints_get_data(p->parent);
if (p->parent->step == PAYMENT_STEP_RETRY) {
plugins/libplugin-pay.c: Round-robin routehints when splitting. This improves the success rate of `test_mpp_interference_2`, though still not quite up to the level that we can remove `@flaky` from it.
2020-08-11 11:29:08 +02:00
d->base = pd->base;
plugins/libplugin-pay.c: Be less aggressive with advancing through routehints. Only advance through routehints if no route was found at all, or if the estimated capacity at the routehint is lower than the amount that we have to send through the routehint. Changelog-Fixed: pay: Be less aggressive with forgetting routehints.
2020-08-09 09:19:21 +02:00
d->offset = pd->offset;
/* If the previous try failed to route, advance
* to the next routehint. */
if (!p->parent->route)
++d->offset;
plugins/libplugin-pay.c: Round-robin routehints when splitting. This improves the success rate of `test_mpp_interference_2`, though still not quite up to the level that we can remove `@flaky` from it.
2020-08-11 11:29:08 +02:00
} else {
size_t num_routehints = tal_count(d->routehints);
paymod: Iterate through the routehints in order We store an offset of the current routehint in the modifier data. It gets incremented on retry, and it gets reset to 0 on split. This is because once we split we have a different amount and a previously unusable routehint becomes usable again.
2020-07-23 11:43:11 +02:00
d->offset = 0;
plugins/libplugin-pay.c: Round-robin routehints when splitting. This improves the success rate of `test_mpp_interference_2`, though still not quite up to the level that we can remove `@flaky` from it.
2020-08-11 11:29:08 +02:00
/* This used to be pseudorand.
*
* However, it turns out that using the partid for
* this payment has some nice properties.
* The partid is in general quite random, due to
* getting entropy from the network on the timing
* of when payments complete/fail, and the routehint
* randomization is not a privacy or security feature,
* only a reliability one, thus does not need a lot
* of entropy.
*
* But the most important bit is that *splits get
* contiguous partids*, e.g. a presplit into 4 will
* usually be numbered 2,3,4,5, and an adaptive split
* will get two consecutive partid.
* Because of the contiguity, using the partid for
* the base will cause the split-up payments to
* have fairly diverse initial routehints.
*
* The special-casing for <= 2 and the - 2 is due
* to the presplitter skipping over partid 1, we want
* the starting splits to have partid 2 start at
* base 0.
*/
if (p->partid <= 2 || num_routehints <= 1)
d->base = 0;
else
d->base = (p->partid - 2) % num_routehints;
}
paymod: Iterate through the routehints in order We store an offset of the current routehint in the modifier data. It gets incremented on retry, and it gets reset to 0 on split. This is because once we split we have a different amount and a previously unusable routehint becomes usable again.
2020-07-23 11:43:11 +02:00
return d;
paymod: Simplify routehint data initialization It was spread over the step callback, but we only need to initialize the routehints array there, child-payments can just inherit most of the information.
2020-07-23 11:07:08 +02:00
} else {
/* We defer the actual initialization of the routehints array to
* the step callback when we have the invoice attached. */
d->routehints = NULL;
plugins/libplugin-pay.c: Round-robin routehints when splitting. This improves the success rate of `test_mpp_interference_2`, though still not quite up to the level that we can remove `@flaky` from it.
2020-08-11 11:29:08 +02:00
d->base = 0;
paymod: Iterate through the routehints in order We store an offset of the current routehint in the modifier data. It gets incremented on retry, and it gets reset to 0 on split. This is because once we split we have a different amount and a previously unusable routehint becomes usable again.
2020-07-23 11:43:11 +02:00
d->offset = 0;
return d;
paymod: Simplify routehint data initialization It was spread over the step callback, but we only need to initialize the routehints array there, child-payments can just inherit most of the information.
2020-07-23 11:07:08 +02:00
}
return d;
paymod: Exclude nodes that we found to be faulty
2020-05-29 17:10:47 +02:00
}
REGISTER_PAYMENT_MODIFIER(routehints, struct routehints_data *,
routehint_data_init, routehint_step_cb);
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
/* For tiny payments the fees incurred due to the fixed base_fee may dominate
* the overall cost of the payment. Since these payments are often used as a
* way to signal, rather than actually transfer the amount, we add an
* exemption that allows tiny payments to exceed the fee allowance. This is
* implemented by setting a larger allowance than we would normally do if the
* payment is below the threshold. */
static struct exemptfee_data *exemptfee_data_init(struct payment *p)
{
paymod: Inherit exemptfee modifier data on retry
2020-07-10 16:16:42 +02:00
if (p->parent == NULL) {
struct exemptfee_data *d = tal(p, struct exemptfee_data);
d->amount = AMOUNT_MSAT(5000);
return d;
} else {
return payment_mod_exemptfee_get_data(p->parent);
}
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
}
static void exemptfee_cb(struct exemptfee_data *d, struct payment *p)
{
exemptfee: Only apply to the root payment and fix logs We were applying the fee exemption to all payments individually, which is ok until we switch to MPP, where amounts change. Also the log entry was referring to the total amount, and not the fee of the payment.
2020-07-12 16:05:23 +02:00
if (p->step != PAYMENT_STEP_INITIALIZED || p->parent != NULL)
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
return payment_continue(p);
paymod: Update the route constraints as we apply routes and mods These are primarily the fee and cltv constraints that we need to keep up to date in order to give modifiers a correct view of what is and what isn't allowed.
2020-06-04 15:44:19 +02:00
if (amount_msat_greater_eq(d->amount, p->constraints.fee_budget)) {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_INFORM,
exemptfee: Only apply to the root payment and fix logs We were applying the fee exemption to all payments individually, which is ok until we switch to MPP, where amounts change. Also the log entry was referring to the total amount, and not the fee of the payment.
2020-07-12 16:05:23 +02:00
"Payment fee constraint %s is below exemption threshold, "
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
"allowing a maximum fee of %s",
exemptfee: Only apply to the root payment and fix logs We were applying the fee exemption to all payments individually, which is ok until we switch to MPP, where amounts change. Also the log entry was referring to the total amount, and not the fee of the payment.
2020-07-12 16:05:23 +02:00
type_to_string(tmpctx, struct amount_msat, &p->constraints.fee_budget),
type_to_string(tmpctx, struct amount_msat, &d->amount));
p->constraints.fee_budget = d->amount;
p->start_constraints->fee_budget = d->amount;
paymod: Exclude most expensive/slowest chan if limits are exceeded
2020-06-03 17:58:20 +02:00
}
return payment_continue(p);
}
REGISTER_PAYMENT_MODIFIER(exemptfee, struct exemptfee_data *,
exemptfee_data_init, exemptfee_cb);
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
/* BOLT #7:
*
* If a route is computed by simply routing to the intended recipient and
* summing the `cltv_expiry_delta`s, then it's possible for intermediate nodes
* to guess their position in the route. Knowing the CLTV of the HTLC, the
* surrounding network topology, and the `cltv_expiry_delta`s gives an
* attacker a way to guess the intended recipient. Therefore, it's highly
* desirable to add a random offset to the CLTV that the intended recipient
* will receive, which bumps all CLTVs along the route.
*
* In order to create a plausible offset, the origin node MAY start a limited
* random walk on the graph, starting from the intended recipient and summing
* the `cltv_expiry_delta`s, and use the resulting sum as the offset. This
* effectively creates a _shadow route extension_ to the actual route and
* provides better protection against this attack vector than simply picking a
* random offset would.
*/
static struct shadow_route_data *shadow_route_init(struct payment *p)
{
pay: fix problematic sharing of shadow route data. Because listchannels responses are async, with mpp we can end up fighting over use of the parent's data giving results from incorrect final CLTVs to assertion failures like below: ``` pay: plugins/libplugin-pay.c:1964: shadow_route_listchannels: Assertion `amount_msat_greater_eq(p->constraints.fee_budget, d->constraints.fee_budget)' failed. pay: FATAL SIGNAL 6 (version v0.9.0rc2-11-g29d32e0-modded) 0x563129eb6a15 send_backtrace common/daemon.c:38 0x563129eb6abf crashdump common/daemon.c:51 0x7fe37c8b920f ??? ???:0 0x7fe37c8b918b ??? ???:0 0x7fe37c898858 ??? ???:0 0x7fe37c898728 ??? ???:0 0x7fe37c8a9f35 ??? ???:0 0x563129ea5c63 shadow_route_listchannels plugins/libplugin-pay.c:1964 0x563129e9c56a handle_rpc_reply plugins/libplugin.c:547 0x563129e9cbfa rpc_read_response_one plugins/libplugin.c:662 0x563129e9ccf0 rpc_conn_read_response plugins/libplugin.c:681 0x563129ece660 next_plan ccan/ccan/io/io.c:59 0x563129ecf245 do_plan ccan/ccan/io/io.c:407 0x563129ecf287 io_ready ccan/ccan/io/io.c:417 0x563129ed151f io_loop ccan/ccan/io/poll.c:445 0x563129e9ef03 plugin_main plugins/libplugin.c:1284 0x563129e9b099 main plugins/pay.c:2010 0x7fe37c89a0b2 ??? ???:0 0x563129e9452d ??? ???:0 0xffffffffffffffff ??? ???:0 ```
2020-07-20 06:48:56 +02:00
struct shadow_route_data *d = tal(p, struct shadow_route_data), *pd;
/* If we're not the root we need to inherit the flags set only on the
* root payment. Since we inherit them at each step it's sufficient to
* do so from our direct parent. */
paymod: Allow callers to opt out of shadow routing amount fuzzing With MPP we require that the sum of parts is equal to the `total_msat` amount declared in the onion. Since that can't be changed once the first part arrives we need a way to disable amount fuzzing for MPP.
2020-07-06 19:21:33 +02:00
if (p->parent != NULL) {
pay: fix problematic sharing of shadow route data. Because listchannels responses are async, with mpp we can end up fighting over use of the parent's data giving results from incorrect final CLTVs to assertion failures like below: ``` pay: plugins/libplugin-pay.c:1964: shadow_route_listchannels: Assertion `amount_msat_greater_eq(p->constraints.fee_budget, d->constraints.fee_budget)' failed. pay: FATAL SIGNAL 6 (version v0.9.0rc2-11-g29d32e0-modded) 0x563129eb6a15 send_backtrace common/daemon.c:38 0x563129eb6abf crashdump common/daemon.c:51 0x7fe37c8b920f ??? ???:0 0x7fe37c8b918b ??? ???:0 0x7fe37c898858 ??? ???:0 0x7fe37c898728 ??? ???:0 0x7fe37c8a9f35 ??? ???:0 0x563129ea5c63 shadow_route_listchannels plugins/libplugin-pay.c:1964 0x563129e9c56a handle_rpc_reply plugins/libplugin.c:547 0x563129e9cbfa rpc_read_response_one plugins/libplugin.c:662 0x563129e9ccf0 rpc_conn_read_response plugins/libplugin.c:681 0x563129ece660 next_plan ccan/ccan/io/io.c:59 0x563129ecf245 do_plan ccan/ccan/io/io.c:407 0x563129ecf287 io_ready ccan/ccan/io/io.c:417 0x563129ed151f io_loop ccan/ccan/io/poll.c:445 0x563129e9ef03 plugin_main plugins/libplugin.c:1284 0x563129e9b099 main plugins/pay.c:2010 0x7fe37c89a0b2 ??? ???:0 0x563129e9452d ??? ???:0 0xffffffffffffffff ??? ???:0 ```
2020-07-20 06:48:56 +02:00
pd = payment_mod_shadowroute_get_data(p->parent);
d->fuzz_amount = pd->fuzz_amount;
#if DEVELOPER
d->use_shadow = pd->use_shadow;
#endif
paymod: Allow callers to opt out of shadow routing amount fuzzing With MPP we require that the sum of parts is equal to the `total_msat` amount declared in the onion. Since that can't be changed once the first part arrives we need a way to disable amount fuzzing for MPP.
2020-07-06 19:21:33 +02:00
} else {
d->fuzz_amount = true;
}
pay: fix problematic sharing of shadow route data. Because listchannels responses are async, with mpp we can end up fighting over use of the parent's data giving results from incorrect final CLTVs to assertion failures like below: ``` pay: plugins/libplugin-pay.c:1964: shadow_route_listchannels: Assertion `amount_msat_greater_eq(p->constraints.fee_budget, d->constraints.fee_budget)' failed. pay: FATAL SIGNAL 6 (version v0.9.0rc2-11-g29d32e0-modded) 0x563129eb6a15 send_backtrace common/daemon.c:38 0x563129eb6abf crashdump common/daemon.c:51 0x7fe37c8b920f ??? ???:0 0x7fe37c8b918b ??? ???:0 0x7fe37c898858 ??? ???:0 0x7fe37c898728 ??? ???:0 0x7fe37c8a9f35 ??? ???:0 0x563129ea5c63 shadow_route_listchannels plugins/libplugin-pay.c:1964 0x563129e9c56a handle_rpc_reply plugins/libplugin.c:547 0x563129e9cbfa rpc_read_response_one plugins/libplugin.c:662 0x563129e9ccf0 rpc_conn_read_response plugins/libplugin.c:681 0x563129ece660 next_plan ccan/ccan/io/io.c:59 0x563129ecf245 do_plan ccan/ccan/io/io.c:407 0x563129ecf287 io_ready ccan/ccan/io/io.c:417 0x563129ed151f io_loop ccan/ccan/io/poll.c:445 0x563129e9ef03 plugin_main plugins/libplugin.c:1284 0x563129e9b099 main plugins/pay.c:2010 0x7fe37c89a0b2 ??? ???:0 0x563129e9452d ??? ???:0 0xffffffffffffffff ??? ???:0 ```
2020-07-20 06:48:56 +02:00
return d;
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
}
/* Mutual recursion */
static struct command_result *shadow_route_listchannels(struct command *cmd,
const char *buf,
const jsmntok_t *result,
struct payment *p);
static struct command_result *shadow_route_extend(struct shadow_route_data *d,
struct payment *p)
{
struct out_req *req;
req = jsonrpc_request_start(p->plugin, NULL, "listchannels",
shadow_route_listchannels,
payment_rpc_failure, p);
json_add_string(req->js, "source",
type_to_string(req, struct node_id, &d->destination));
return send_outreq(p->plugin, req);
}
static struct command_result *shadow_route_listchannels(struct command *cmd,
const char *buf,
const jsmntok_t *result,
struct payment *p)
{
/* Use reservoir sampling across the capable channels. */
struct shadow_route_data *d = payment_mod_shadowroute_get_data(p);
struct payment_constraints *cons = &d->constraints;
struct route_info *best = NULL;
size_t i;
u64 sample = 0;
struct amount_msat best_fee;
const jsmntok_t *sattok, *delaytok, *basefeetok, *propfeetok, *desttok,
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
*channelstok, *chan, *scidtok;
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
paymod: Add invariant verification for constraints on shadowroute This was highlighted in #3851, so I added an assertion. After the rewrite in the next commit we would simply skip if any of the constraints were not maintained, but this serves as the canary in the coalmine, so we don't paper over.
2020-07-19 21:32:52 +02:00
/* Check the invariants on the constraints between payment and modifier. */
assert(d->constraints.cltv_budget <= p->constraints.cltv_budget / 4);
assert(amount_msat_greater_eq(p->constraints.fee_budget,
d->constraints.fee_budget));
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
channelstok = json_get_member(buf, result, "channels");
json_for_each_arr(i, chan, channelstok) {
u64 v = pseudorand(UINT64_MAX);
struct route_info curr;
struct amount_sat capacity;
struct amount_msat fee;
sattok = json_get_member(buf, chan, "satoshis");
delaytok = json_get_member(buf, chan, "delay");
basefeetok = json_get_member(buf, chan, "base_fee_millisatoshi");
propfeetok = json_get_member(buf, chan, "fee_per_millionth");
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
scidtok = json_get_member(buf, chan, "short_channel_id");
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
desttok = json_get_member(buf, chan, "destination");
if (sattok == NULL || delaytok == NULL ||
delaytok->type != JSMN_PRIMITIVE || basefeetok == NULL ||
basefeetok->type != JSMN_PRIMITIVE || propfeetok == NULL ||
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
propfeetok->type != JSMN_PRIMITIVE || desttok == NULL ||
scidtok == NULL)
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
continue;
json_to_u16(buf, delaytok, &curr.cltv_expiry_delta);
json_to_number(buf, basefeetok, &curr.fee_base_msat);
json_to_number(buf, propfeetok,
&curr.fee_proportional_millionths);
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
json_to_short_channel_id(buf, scidtok, &curr.short_channel_id);
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
json_to_sat(buf, sattok, &capacity);
json_to_node_id(buf, desttok, &curr.pubkey);
if (!best || v > sample) {
/* If the capacity is insufficient to pass the amount
* it's not a plausible extension. */
if (amount_msat_greater_sat(p->amount, capacity))
continue;
if (curr.cltv_expiry_delta > cons->cltv_budget)
continue;
if (!amount_msat_fee(
&fee, p->amount, curr.fee_base_msat,
curr.fee_proportional_millionths)) {
/* Fee computation failed... */
continue;
}
if (amount_msat_greater_eq(fee, cons->fee_budget))
continue;
best = tal_dup(tmpctx, struct route_info, &curr);
best_fee = fee;
sample = v;
}
}
if (best != NULL) {
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
/* Check that we could apply the shadow route extension. Check
* against both the shadow route budget as well as the
* original payment's budget. */
if (best->cltv_expiry_delta > d->constraints.cltv_budget ||
best->cltv_expiry_delta > p->constraints.cltv_budget) {
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
best = NULL;
goto next;
}
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
/* Check the fee budget only if we didn't opt out, since
* testing against a virtual budget is not useful if we do not
* actually use it (it could give false positives and fail
* attempts that might have gone through, */
if (d->fuzz_amount &&
(amount_msat_greater(best_fee, d->constraints.fee_budget) ||
(amount_msat_greater(best_fee,
p->constraints.fee_budget)))) {
best = NULL;
goto next;
paymod: Allow callers to opt out of shadow routing amount fuzzing With MPP we require that the sum of parts is equal to the `total_msat` amount declared in the onion. Since that can't be changed once the first part arrives we need a way to disable amount fuzzing for MPP.
2020-07-06 19:21:33 +02:00
}
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
/* Now we can be sure that adding the shadow route will succeed */
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(
p, LOG_DBG,
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
"Adding shadow_route hop over channel %s: adding %s "
"in fees and %d CLTV delta",
type_to_string(tmpctx, struct short_channel_id,
&best->short_channel_id),
type_to_string(tmpctx, struct amount_msat, &best_fee),
best->cltv_expiry_delta);
d->destination = best->pubkey;
d->constraints.cltv_budget -= best->cltv_expiry_delta;
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
p->getroute->cltv += best->cltv_expiry_delta;
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
if (!d->fuzz_amount)
goto next;
/* Only try to apply the fee budget changes if we want to fuzz
* the amount. Virtual fees that we then don't deliver to the
* destination could otherwise cause the route to be too
* expensive, while really being ok. If any of these fail then
* the above checks are insufficient. */
if (!amount_msat_sub(&d->constraints.fee_budget,
d->constraints.fee_budget, best_fee) ||
!amount_msat_sub(&p->constraints.fee_budget,
p->constraints.fee_budget, best_fee))
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_err(p,
paymod: Rewrite the shadow-route constraint enforcement We now check against both constraints on the modifier and the payment before applying either. This "fixes" the assert that was causing the crash in #3851, but we are still looking for the source of the inconsistency where the modifier constraints, initialized to 1/4th of the payment, suddenly get more permissive than the payment itself.
2020-07-19 21:34:50 +02:00
"Could not update fee constraints "
"for shadow route extension. "
"payment fee budget %s, modifier "
"fee budget %s, shadow fee to add %s",
type_to_string(tmpctx, struct amount_msat,
&p->constraints.fee_budget),
type_to_string(tmpctx, struct amount_msat,
&d->constraints.fee_budget),
type_to_string(tmpctx, struct amount_msat,
&best_fee));
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
}
next:
/* Now it's time to decide whether we want to extend or continue. */
if (best == NULL || pseudorand(2) == 0) {
payment_continue(p);
return command_still_pending(cmd);
} else {
return shadow_route_extend(d, p);
}
}
static void shadow_route_cb(struct shadow_route_data *d,
struct payment *p)
{
#if DEVELOPER
if (!d->use_shadow)
return payment_continue(p);
#endif
if (p->step != PAYMENT_STEP_INITIALIZED)
return payment_continue(p);
d->destination = *p->destination;
/* Allow shadowroutes to consume up to 1/4th of our budget. */
d->constraints.cltv_budget = p->constraints.cltv_budget / 4;
amount: add amount_msat_scale, amount_msat_ratio, amount_{msat,sat}_div It's not all that rare to do these operations, and requiring annotations for it is a little painful. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-05 05:56:32 +02:00
d->constraints.fee_budget
= amount_msat_div(p->constraints.fee_budget, 4);
paymod: Add user-provided label back into the paystatus result
2020-07-02 15:33:55 +02:00
if (pseudorand(2) == 0) {
return payment_continue(p);
} else {
shadow_route_extend(d, p);
}
}
REGISTER_PAYMENT_MODIFIER(shadowroute, struct shadow_route_data *,
shadow_route_init, shadow_route_cb);
paymod: Consolidate parsing RPC results in libplugin We handle these in a number of different ways, and regularly get the parsing and logic for optional fields wrong, so let's consolidate them here.
2020-06-09 18:30:14 +02:00
static void direct_pay_override(struct payment *p) {
/* The root has performed the search for a direct channel. */
struct payment *root = payment_root(p);
struct direct_pay_data *d;
struct channel_hint *hint = NULL;
/* If we were unable to find a direct channel we don't need to do
* anything. */
d = payment_mod_directpay_get_data(root);
if (d->chan == NULL)
return payment_continue(p);
/* If we have a channel we need to make sure that it still has
* sufficient capacity. Look it up in the channel_hints. */
for (size_t i=0; i<tal_count(root->channel_hints); i++) {
struct short_channel_id_dir *cur = &root->channel_hints[i].scid;
if (short_channel_id_eq(&cur->scid, &d->chan->scid) &&
cur->dir == d->chan->dir) {
hint = &root->channel_hints[i];
break;
}
}
if (hint && hint->enabled &&
amount_msat_greater(hint->estimated_capacity, p->amount)) {
/* Now build a route that consists only of this single hop */
p->route = tal_arr(p, struct route_hop, 1);
p->route[0].amount = p->amount;
p->route[0].delay = p->getroute->cltv;
p->route[0].channel_id = hint->scid.scid;
p->route[0].direction = hint->scid.dir;
p->route[0].nodeid = *p->destination;
p->route[0].style = ROUTE_HOP_TLV;
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_DBG,
paymod: Consolidate parsing RPC results in libplugin We handle these in a number of different ways, and regularly get the parsing and logic for optional fields wrong, so let's consolidate them here.
2020-06-09 18:30:14 +02:00
"Found a direct channel (%s) with sufficient "
"capacity, skipping route computation.",
type_to_string(tmpctx, struct short_channel_id_dir,
&hint->scid));
payment_set_step(p, PAYMENT_STEP_GOT_ROUTE);
}
payment_continue(p);
}
/* Now that we have the listpeers result for the root payment, let's search
* for a direct channel that is a) connected and b) in state normal. We will
* check the capacity based on the channel_hints in the override. */
static struct command_result *direct_pay_listpeers(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
struct listpeers_result *r =
json_to_listpeers_result(tmpctx, buffer, toks);
struct direct_pay_data *d = payment_mod_directpay_get_data(p);
if (tal_count(r->peers) == 1) {
struct listpeers_peer *peer = r->peers[0];
if (!peer->connected)
goto cont;
for (size_t i=0; i<tal_count(peer->channels); i++) {
struct listpeers_channel *chan = r->peers[0]->channels[i];
if (!streq(chan->state, "CHANNELD_NORMAL"))
continue;
d->chan = tal(d, struct short_channel_id_dir);
d->chan->scid = *chan->scid;
d->chan->dir = *chan->direction;
}
}
cont:
direct_pay_override(p);
return command_still_pending(cmd);
}
static void direct_pay_cb(struct direct_pay_data *d, struct payment *p)
{
struct out_req *req;
/* Look up the direct channel only on root. */
if (p->step != PAYMENT_STEP_INITIALIZED)
return payment_continue(p);
req = jsonrpc_request_start(p->plugin, NULL, "listpeers",
direct_pay_listpeers, direct_pay_listpeers,
p);
json_add_node_id(req->js, "id", p->destination);
send_outreq(p->plugin, req);
}
static struct direct_pay_data *direct_pay_init(struct payment *p)
{
struct direct_pay_data *d = tal(p, struct direct_pay_data);
d->chan = NULL;
return d;
}
REGISTER_PAYMENT_MODIFIER(directpay, struct direct_pay_data *, direct_pay_init,
direct_pay_cb);
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
static struct command_result *waitblockheight_rpc_cb(struct command *cmd,
const char *buffer,
const jsmntok_t *toks,
struct payment *p)
{
struct payment *subpayment;
subpayment = payment_new(p, NULL, p, p->modifiers);
payment_start(subpayment);
payment_set_step(p, PAYMENT_STEP_RETRY);
subpayment->why =
tal_fmt(subpayment, "Retrying after waiting for blockchain sync.");
plugins/libplugin-pay.c: Also print events that create new sub-payments.
2020-08-10 08:16:50 +02:00
paymod_log(p, LOG_DBG,
"Retrying after waitblockheight, new partid %"PRIu32,
subpayment->partid);
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
payment_continue(p);
return command_still_pending(cmd);
}
static void waitblockheight_cb(void *d, struct payment *p)
{
struct out_req *req;
struct timeabs now = time_now();
struct timerel remaining;
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
u32 blockheight;
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
if (p->step != PAYMENT_STEP_FAILED)
return payment_continue(p);
/* If we don't have an error message to parse we can't wait for blockheight. */
if (p->result == NULL)
return payment_continue(p);
if (time_after(now, p->deadline))
return payment_continue(p);
remaining = time_between(p->deadline, now);
plugins/libplugin-pay.c: Make sure blockheight disagreement does not prevent all future progress. Blockheight disagreement is signalled with a permanent failure at the end node, but is actually a transient failure.
2020-08-10 09:25:41 +02:00
/* *Was* it a blockheight disagreement that caused the failure? */
if (!failure_is_blockheight_disagreement(p, &blockheight))
paymod: Do not wait for a blockheight if we're already there We want to differentiate a wrong block-height from other failure reasons, such as an unknown `payment_hash`, so we skip the `waitblockheight` if we're already at the correct height.
2020-06-17 13:51:00 +02:00
return payment_continue(p);
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_INFORM,
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
"Remote node appears to be on a longer chain, which causes "
"CLTV timeouts to be incorrect. Waiting up to %" PRIu64
" seconds to catch up to block %d before retrying.",
time_to_sec(remaining), blockheight);
paymod: Reset the step if we're waiting for a block This was causing the state flapping test to fail, since we were yielding control of the io_loop, waiting for the blockheight to be reached, and not setting the status beforehand. An interim `paystatus` call would then find a failed leaf and deduce the entire payment failed. Setting it back to the previous state keeps the overall payment pending while we wait.
2020-07-08 21:10:45 +02:00
/* Set temporarily set the state of the payment to not failed, so
* interim status queries don't show this as terminally failed. We're
* in control for this payment so nobody else could be fooled by
* this. The callback will set it to retry anyway. */
payment_set_step(p, PAYMENT_STEP_RETRY);
paymod: Add a deadline to the pay command and retry modifier
2020-06-11 11:38:41 +02:00
req = jsonrpc_request_start(p->plugin, NULL, "waitblockheight",
waitblockheight_rpc_cb,
waitblockheight_rpc_cb, p);
json_add_u32(req->js, "blockheight", blockheight);
json_add_u32(req->js, "timeout", time_to_sec(remaining));
send_outreq(p->plugin, req);
}
REGISTER_PAYMENT_MODIFIER(waitblockheight, void *, NULL, waitblockheight_cb);
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
/*****************************************************************************
* presplit -- Early MPP splitter modifier.
*
* This splitter modifier is applied to the root payment, and splits the
* payment into parts that are more likely to succeed right away. The
* parameters are derived from probing the network for channel capacities, and
* may be adjusted in future.
*/
/*By probing the capacity from a well-connected vantage point in the network
* we found that the 80th percentile of capacities is >= 9765 sats.
*
* Rounding to 10e6 msats per part there is a ~80% chance that the payment
* will go through without requiring further splitting. The fuzzing is
* symmetric and uniformy distributed around this value, so this should not
* change the success rate much. For the remaining 20% of payments we might
* require a split to make the parts succeed, so we try only a limited number
* of times before we split adaptively.
*
* Notice that these numbers are based on a worst case assumption that
* payments from any node to any other node are equally likely, which isn't
* really the case, so this is likely a lower bound on the success rate.
*
* As the network evolves these numbers are also likely to change.
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
*
* Finally, if applied trivially this splitter may end up creating more splits
* than the sum of all channels can support, i.e., each split results in an
* HTLC, and each channel has an upper limit on the number of HTLCs it'll
* allow us to add. If the initial split would result in more than 1/3rd of
* the total available HTLCs we clamp the number of splits to 1/3rd. We don't
* use 3/3rds in order to retain flexibility in the adaptive splitter.
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
*/
#define MPP_TARGET_SIZE (10 * 1000 * 1000)
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
#define PRESPLIT_MAX_HTLC_SHARE 3
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
pay: Implement simple presplit fix for ludicrous amounts This is the simplest possible fix: increase the target amount until we get the desired number of parts, while still bucketizing payments together that are in approximately the same size. The current logic puts all payments that are in the range x < amount <= 16*x in the same bucket, making them harder to distinguish. Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-26 20:40:36 +02:00
/* How many parts do we split into before we increase the bucket size. This is
* a tradeoff between the number of payments whose parts are identical and the
* number of concurrent HTLCs. The larger this amount the more HTLCs we may
* end up starting, but the more payments result in the same part sizes.*/
#define PRESPLIT_MAX_SPLITS 16
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
static struct presplit_mod_data *presplit_mod_data_init(struct payment *p)
{
struct presplit_mod_data *d;
if (p->parent == NULL) {
d = tal(p, struct presplit_mod_data);
d->disable = false;
return d;
} else {
return payment_mod_presplit_get_data(p->parent);
}
}
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
static u32 payment_max_htlcs(const struct payment *p)
{
struct channel_hint *h;
u32 res = 0;
for (size_t i = 0; i < tal_count(p->channel_hints); i++) {
h = &p->channel_hints[i];
if (h->local && h->enabled)
res += h->htlc_budget;
}
return res;
}
mpp: Detect if destination supports MPP from invoice and abort early We abort on the root since that is the coordination point for all parts of the payment.
2020-07-10 15:21:47 +02:00
static bool payment_supports_mpp(struct payment *p)
{
if (p->invoice == NULL || p->invoice->features == NULL)
return false;
return feature_offered(p->invoice->features, OPT_BASIC_MPP);
}
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
/* Return fuzzed amount ~= target, but never exceeding max */
static struct amount_msat fuzzed_near(struct amount_msat target,
struct amount_msat max)
{
s64 fuzz;
struct amount_msat res = target;
/* Somewhere within 25% of target please. */
fuzz = pseudorand(target.millisatoshis / 2) /* Raw: fuzz */
- target.millisatoshis / 4; /* Raw: fuzz */
res.millisatoshis = target.millisatoshis + fuzz; /* Raw: fuzz < msat */
if (amount_msat_greater(res, max))
res = max;
return res;
}
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
static void presplit_cb(struct presplit_mod_data *d, struct payment *p)
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
{
struct payment *root = payment_root(p);
paymod: Consolidate step selection and changes in presplit modifier We skip most payment steps and all sub-payments, so consolidate the skip conditions in one if-statement. We also not use `payment_set_step` to skip any modifiers after us after the step change.
2020-07-19 21:39:53 +02:00
if (d->disable || p->parent != NULL || !payment_supports_mpp(p))
mpp: Detect if destination supports MPP from invoice and abort early We abort on the root since that is the coordination point for all parts of the payment.
2020-07-10 15:21:47 +02:00
return payment_continue(p);
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
if (p->step == PAYMENT_STEP_ONION_PAYLOAD) {
/* We need to tell the last hop the total we're going to
* send. Presplit disables amount fuzzing, so we should always
* get the exact value through. */
size_t lastidx = tal_count(p->createonion_request->hops) - 1;
struct createonion_hop *hop = &p->createonion_request->hops[lastidx];
if (hop->style == ROUTE_HOP_TLV) {
struct tlv_field **fields = &hop->tlv_payload->fields;
tlvstream_set_tlv_payload_data(
fields, root->payment_secret,
root->amount.millisatoshis); /* Raw: onion payload */
}
paymod: Consolidate step selection and changes in presplit modifier We skip most payment steps and all sub-payments, so consolidate the skip conditions in one if-statement. We also not use `payment_set_step` to skip any modifiers after us after the step change.
2020-07-19 21:39:53 +02:00
} else if (p->step == PAYMENT_STEP_INITIALIZED) {
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
/* The presplitter only acts on the root and only in the first
* step. */
paymod: Not having a result doesn't mean we failed at getroute Specifically if we split, there is no result, but we shouldn't add a failure message.
2020-07-03 20:55:20 +02:00
size_t count = 0;
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
u32 htlcs = payment_max_htlcs(p) / PRESPLIT_MAX_HTLC_SHARE;
struct amount_msat target, amt = p->amount;
plugins/libplugin-pay.c: Also print events that create new sub-payments.
2020-08-10 08:16:50 +02:00
char *partids = tal_strdup(tmpctx, "");
pay: Implement simple presplit fix for ludicrous amounts This is the simplest possible fix: increase the target amount until we get the desired number of parts, while still bucketizing payments together that are in approximately the same size. The current logic puts all payments that are in the range x < amount <= 16*x in the same bucket, making them harder to distinguish. Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-26 20:40:36 +02:00
u64 target_amount = MPP_TARGET_SIZE;
/* We aim for at most PRESPLIT_MAX_SPLITS parts, even for
* large values. To achieve this we take the base amount and
* multiply it by the number of targetted parts until the
* total amount divided by part amount gives us at most that
* number of parts. */
while (target_amount * PRESPLIT_MAX_SPLITS < p->amount.millisatoshis) /* Raw: Multiplication comparison */
target_amount *= PRESPLIT_MAX_SPLITS;
paymod: Not having a result doesn't mean we failed at getroute Specifically if we split, there is no result, but we shouldn't add a failure message.
2020-07-03 20:55:20 +02:00
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
/* We need to opt-in to the MPP sending facility no matter
* what we do. That means setting all partids to a non-zero
* value. */
root->partid++;
paymod: Do not duplicate partids When using mpp we need to always have partids>0, since we bumped the partid for the root, but not the next_id we'd end up with partid=1 being duplicated. Not a big problem since we never ended up sending the root to lightningd, instead skipping it, but it was confusing me while trying to trace sub-payment's ancestry.
2020-07-19 21:41:39 +02:00
/* Bump the next_partid as well so we don't have duplicate
* partids. Not really necessary since the root payment whose
* id could be reused will never reach the `sendonion` step,
* but makes debugging a bit easier. */
root->next_partid++;
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
if (htlcs == 0) {
p->abort = true;
return payment_fail(
p, "Cannot attempt payment, we have no channel to "
"which we can add an HTLC");
pay: Implement simple presplit fix for ludicrous amounts This is the simplest possible fix: increase the target amount until we get the desired number of parts, while still bucketizing payments together that are in approximately the same size. The current logic puts all payments that are in the range x < amount <= 16*x in the same bucket, making them harder to distinguish. Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-26 20:40:36 +02:00
} else if (p->amount.millisatoshis / target_amount > htlcs) /* Raw: division */
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
target.millisatoshis = p->amount.millisatoshis / htlcs; /* Raw: division */
else
pay: Implement simple presplit fix for ludicrous amounts This is the simplest possible fix: increase the target amount until we get the desired number of parts, while still bucketizing payments together that are in approximately the same size. The current logic puts all payments that are in the range x < amount <= 16*x in the same bucket, making them harder to distinguish. Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
2020-08-26 20:40:36 +02:00
target.millisatoshis = target_amount; /* Raw: init */
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
/* If we are already below the target size don't split it
* either. */
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
if (amount_msat_greater(target, p->amount))
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
return payment_continue(p);
paymod: Update step before creating child payments The child payments will sometimes depend on the step of the parent, and making sure that the parent state is correct before we create the children is therefore important.
2020-07-23 16:02:02 +02:00
payment_set_step(p, PAYMENT_STEP_SPLIT);
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
/* Ok, we know we should split, so split here and then skip this
* payment and start the children instead. */
while (!amount_msat_eq(amt, AMOUNT_MSAT(0))) {
double multiplier;
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
struct payment *c =
payment_new(p, NULL, p, p->modifiers);
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
pytest: Reproduce #3915
2020-08-11 15:23:33 +02:00
/* Annotate the subpayments with the bolt11 string,
* they'll be used when aggregating the payments
* again. */
c->bolt11 = tal_strdup(c, p->bolt11);
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
/* Get ~ target, but don't exceed amt */
c->amount = fuzzed_near(target, amt);
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
if (!amount_msat_sub(&amt, amt, c->amount))
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_err(
p,
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
"Cannot subtract %s from %s in splitter",
type_to_string(tmpctx, struct amount_msat,
&c->amount),
type_to_string(tmpctx, struct amount_msat,
&amt));
/* Now adjust the constraints so we don't multiply them
* when splitting. */
amount: add amount_msat_scale, amount_msat_ratio, amount_{msat,sat}_div It's not all that rare to do these operations, and requiring annotations for it is a little painful. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-05 05:56:32 +02:00
multiplier = amount_msat_ratio(c->amount, p->amount);
if (!amount_msat_scale(&c->constraints.fee_budget,
c->constraints.fee_budget,
multiplier))
abort(); /* multiplier < 1! */
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
payment_start(c);
plugins/libplugin-pay.c: Also print events that create new sub-payments.
2020-08-10 08:16:50 +02:00
/* Why the wordy "new partid n" that we repeat for
* each payment?
* So that you can search the logs for the
* creation of a partid by just "new partid n".
*/
if (count == 0)
tal_append_fmt(&partids, "new partid %"PRIu32, c->partid);
else
tal_append_fmt(&partids, ", new partid %"PRIu32, c->partid);
paymod: Not having a result doesn't mean we failed at getroute Specifically if we split, there is no result, but we shouldn't add a failure message.
2020-07-03 20:55:20 +02:00
count++;
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
}
paymod: Consolidate step selection and changes in presplit modifier We skip most payment steps and all sub-payments, so consolidate the skip conditions in one if-statement. We also not use `payment_set_step` to skip any modifiers after us after the step change.
2020-07-19 21:39:53 +02:00
p->result = NULL;
p->route = NULL;
paymod: Not having a result doesn't mean we failed at getroute Specifically if we split, there is no result, but we shouldn't add a failure message.
2020-07-03 20:55:20 +02:00
p->why = tal_fmt(
p,
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
"Split into %zu sub-payments due to initial size (%s > %s)",
paymod: Not having a result doesn't mean we failed at getroute Specifically if we split, there is no result, but we shouldn't add a failure message.
2020-07-03 20:55:20 +02:00
count,
type_to_string(tmpctx, struct amount_msat, &root->amount),
paymod: Teach the presplit modifier to respect the chan HTLC limit The presplit modifier could end up exceeding the maximum number of HTLCs we can add to a channel right out the gate, so we switch to a dynamic presplit if that is the case. The presplit will now at most use 1/3rd of the available HTLCs on the channels if the normal split would exceed the number of availabe HTLCs. And we also abort early if we don't have a sufficient HTLCs available.
2020-07-20 21:56:05 +02:00
type_to_string(tmpctx, struct amount_msat, &target));
plugins/libplugin-pay.c: Also print events that create new sub-payments.
2020-08-10 08:16:50 +02:00
paymod_log(p, LOG_INFORM, "%s: %s", p->why, partids);
mpp: Add the presplit-modifier that splits a root payment first
2020-07-01 18:59:42 +02:00
}
payment_continue(p);
}
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
REGISTER_PAYMENT_MODIFIER(presplit, struct presplit_mod_data *,
presplit_mod_data_init, presplit_cb);
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
/*****************************************************************************
* Adaptive splitter -- Split payment if we can't get it through.
*
* The adaptive splitter splits the amount of a failed payment in half, with
* +/- 10% randomness, and then starts two attempts, one for either side of
* the split. The goal is to find two smaller routes, that still adhere to our
* constraints, but that can complete the payment.
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
*
* This modifier also checks whether we can split and still have enough HTLCs
* available on the channels and aborts if that's no longer the case.
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
*/
#define MPP_ADAPTIVE_LOWER_LIMIT AMOUNT_MSAT(100 * 1000)
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
static struct adaptive_split_mod_data *adaptive_splitter_data_init(struct payment *p)
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
{
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
struct adaptive_split_mod_data *d;
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
if (p->parent == NULL) {
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
d = tal(p, struct adaptive_split_mod_data);
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
d->disable = false;
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
d->htlc_budget = 0;
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
return d;
} else {
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
return payment_mod_adaptive_splitter_get_data(p->parent);
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
}
}
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
static void adaptive_splitter_cb(struct adaptive_split_mod_data *d, struct payment *p)
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
{
struct payment *root = payment_root(p);
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
struct adaptive_split_mod_data *root_data =
payment_mod_adaptive_splitter_get_data(root);
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
if (d->disable)
return payment_continue(p);
mpp: Detect if destination supports MPP from invoice and abort early We abort on the root since that is the coordination point for all parts of the payment.
2020-07-10 15:21:47 +02:00
if (!payment_supports_mpp(p) || root->abort)
return payment_continue(p);
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
if (p->parent == NULL && d->htlc_budget == 0) {
/* Now that we potentially had an early splitter run, let's
* update our htlc_budget that we own exclusively from now
* on. We do this by subtracting the number of payment
* attempts an eventual presplitter has already performed. */
struct payment_tree_result res;
res = payment_collect_result(p);
d->htlc_budget = payment_max_htlcs(p);
if (res.attempts > d->htlc_budget) {
p->abort = true;
return payment_fail(
p,
"Cannot add %d HTLCs to our channels, we "
"only have %d HTLCs available.",
res.attempts, d->htlc_budget);
}
d->htlc_budget -= res.attempts;
}
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
if (p->step == PAYMENT_STEP_ONION_PAYLOAD) {
/* We need to tell the last hop the total we're going to
* send. Presplit disables amount fuzzing, so we should always
* get the exact value through. */
size_t lastidx = tal_count(p->createonion_request->hops) - 1;
struct createonion_hop *hop = &p->createonion_request->hops[lastidx];
if (hop->style == ROUTE_HOP_TLV) {
struct tlv_field **fields = &hop->tlv_payload->fields;
tlvstream_set_tlv_payload_data(
fields, root->payment_secret,
root->amount.millisatoshis); /* Raw: onion payload */
}
} else if (p->step == PAYMENT_STEP_FAILED && !p->abort) {
if (amount_msat_greater(p->amount, MPP_ADAPTIVE_LOWER_LIMIT)) {
struct payment *a, *b;
/* Random number in the range [90%, 110%] */
double rand = pseudorand_double() * 0.2 + 0.9;
u64 mid = p->amount.millisatoshis / 2 * rand; /* Raw: multiplication */
bool ok;
paymod: Fix the adaptive splitter partitioning We were using the current constraints, including any shadow route and other modifications, when computing the remainder that the second child should use. Instead we should use the `start_constraints` on the parent payment, which is a copy of `constraints` created in `payment_start` exactly for this purpose. Also added an assert for the invariant on the multiplier.
2020-07-19 21:44:03 +02:00
/* Use the start constraints, not the ones updated by routes and shadow-routes. */
struct payment_constraints *pconstraints = p->start_constraints;
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
/* First check that splitting doesn't exceed our HTLC budget */
if (root_data->htlc_budget == 0) {
root->abort = true;
return payment_fail(
p,
"Cannot split payment any further without "
"exceeding the maximum number of HTLCs "
"allowed by our channels");
}
paymod: Update step before creating child payments The child payments will sometimes depend on the step of the parent, and making sure that the parent state is correct before we create the children is therefore important.
2020-07-23 16:02:02 +02:00
p->step = PAYMENT_STEP_SPLIT;
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
a = payment_new(p, NULL, p, p->modifiers);
b = payment_new(p, NULL, p, p->modifiers);
a->amount.millisatoshis = mid; /* Raw: split. */
b->amount.millisatoshis -= mid; /* Raw: split. */
amount: add amount_msat_scale, amount_msat_ratio, amount_{msat,sat}_div It's not all that rare to do these operations, and requiring annotations for it is a little painful. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-05 05:56:32 +02:00
double multiplier = amount_msat_ratio(a->amount,
p->amount);
paymod: Fix the adaptive splitter partitioning We were using the current constraints, including any shadow route and other modifications, when computing the remainder that the second child should use. Instead we should use the `start_constraints` on the parent payment, which is a copy of `constraints` created in `payment_start` exactly for this purpose. Also added an assert for the invariant on the multiplier.
2020-07-19 21:44:03 +02:00
assert(multiplier >= 0.4 && multiplier < 0.6);
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
/* Adjust constraints since we don't want to double our
* fee allowance when we split. */
amount: add amount_msat_scale, amount_msat_ratio, amount_{msat,sat}_div It's not all that rare to do these operations, and requiring annotations for it is a little painful. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-05 05:56:32 +02:00
if (!amount_msat_scale(&a->constraints.fee_budget,
pconstraints->fee_budget,
multiplier))
abort();
paymod: Fix the adaptive splitter partitioning We were using the current constraints, including any shadow route and other modifications, when computing the remainder that the second child should use. Instead we should use the `start_constraints` on the parent payment, which is a copy of `constraints` created in `payment_start` exactly for this purpose. Also added an assert for the invariant on the multiplier.
2020-07-19 21:44:03 +02:00
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
ok = amount_msat_sub(&b->constraints.fee_budget,
paymod: Fix the adaptive splitter partitioning We were using the current constraints, including any shadow route and other modifications, when computing the remainder that the second child should use. Instead we should use the `start_constraints` on the parent payment, which is a copy of `constraints` created in `payment_start` exactly for this purpose. Also added an assert for the invariant on the multiplier.
2020-07-19 21:44:03 +02:00
pconstraints->fee_budget,
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
a->constraints.fee_budget);
/* Should not fail, mid is less than 55% of original
* amount. fee_budget_a <= 55% of fee_budget_p (parent
* of the new payments).*/
assert(ok);
payment_start(a);
payment_start(b);
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
plugins/libplugin-pay.c: Also print events that create new sub-payments.
2020-08-10 08:16:50 +02:00
paymod_log(p, LOG_DBG,
"Adaptively split into 2 sub-payments: "
"new partid %"PRIu32" (%s), "
"new partid %"PRIu32" (%s)",
a->partid,
type_to_string(tmpctx, struct amount_msat,
&a->amount),
b->partid,
type_to_string(tmpctx, struct amount_msat,
&b->amount));
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
/* Take note that we now have an additional split that
* may end up using an HTLC. */
root_data->htlc_budget--;
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
} else {
plugins/libplugin-pay.c: Give cmd id and partid for each log message. Changelog-None: internal debugging Makes it easier to debug payments with tons of splits.
2020-08-10 07:43:52 +02:00
paymod_log(p, LOG_INFORM,
paymod: Implement adaptive splitter This modifier splits a payment that has been attempted a number of times (by a modifier earlier in the mod chain) and has failed consistently. It splits the amount roughly in half, with a but if random fuzz, and then starts a new round of attempts for the two smaller amounts.
2020-07-06 20:29:00 +02:00
"Lower limit of adaptive splitter reached "
"(%s < %s), not splitting further.",
type_to_string(tmpctx, struct amount_msat,
&p->amount),
type_to_string(tmpctx, struct amount_msat,
&MPP_ADAPTIVE_LOWER_LIMIT));
}
}
payment_continue(p);
}
paymod: Teach the adaptive splitter to respect the HTLC limit There is little point in trying to split if the resulting HTLCs exceed the maximum number of HTLCs we can add to our channels. So abort if a split would result in more HTLCs than our channels can support.
2020-07-20 22:00:43 +02:00
REGISTER_PAYMENT_MODIFIER(adaptive_splitter, struct adaptive_split_mod_data *,
mpp: Add CLI option to opt-out of multi-part payments Several tests are not well-suited for mpp, so I added a CLI option to opt-out of the MPP support at startup time.
2020-07-10 14:48:00 +02:00
adaptive_splitter_data_init, adaptive_splitter_cb);
Reference in New Issue Copy Permalink