Tests that will only run when !EXPERIMENTAL_DUAL_FUND:
@pytest.marker.openchannel('v1')
def test_...()
Tests that will only run when EXPERIMENTAL_DUAL_FUND:
@pytest.marker.openchannel('v2')
def test_...()
Users are more upset recently with the cost of unilateral closes
than they are the risk of being cheated. While we complete our
anchor implementation so we can use low fees there, let's
get less aggressive (we already have 34 or 18 blocks to close
in the worst case).
The changes are:
- Commit transactions were "2 CONSERVATIVE" now "6 ECONOMICAL".
- HTLC resolution txs were "3 CONSERVATIVE" now "6 ECONOMICAL".
- Penalty txs were "3 CONSERVATIVE" now "12 ECONOMICAL".
- Normal txs were "4 ECONOMICAL" now "12 ECONOMICAL".
There can be no perfect levels, but we have had understandable
complaints recently about how high our default fee levels are.
Changelog-Changed: Protocol: channel feerates reduced to bitcoind's "6 block ECONOMICAL" rate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Channel can be inactive before it disconnects, apparently. Check
explicitly for the disconnected state so we get the expected error.
Here's what happened:
```
# Can't pay while its offline.
with pytest.raises(RpcError, match=r'failed: WIRE_TEMPORARY_CHANNEL_FAILURE \(First peer not ready\)'):
> l1.rpc.sendpay(route, rhash)
E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
```
And the logs show that the outgoing HTLC was sent to channeld before it
realized the connection was closed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There is little point in faking a self-payment, but we should also not
crash :-)
Fixes#4438
Changelog-Fixed: keysend: Keysend returns an error when a self-payment is requested
We were not aborting if we had routehints, even though all routehints
may have been filtered out.
Changelog-Fixed: pay: `pay` will now abort early if the destination is not reachable directly nor via routehints.
We would happily spin on attempts that are doomed to fail because we
don't know the entrypoint. Next up: remove routehints whose
entrypoints are known but unreachable.
We get a label clash: easy, just re-serve:
```
2021-02-18T04:29:37.474Z **BROKEN** plugin-offers: Failed invoice_request lnr1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcyyqwtp0rmsgquvuacqcl5cdfzwzmu3v8tqgvpqs8e80dlmxm7ey4xwrqdsqqqqqqqqqqqqqqqq2pqqfqpqynzqzx9rylzy40ernj4jzc3p2dwy3n8x6lqeaywwk725ghx4kx63pcfxgg2z3nsn80jzge06nt3ks8pr6rvnujq48376lpmrr3cq04nurpy783eyr0awh5773lrlmjek07rjf0nx4g9235ulkcs7jp2h5gumjyquhadh846da3jptxm9g0qz5lne4hjhag for offer 1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67: Got JSON error: {\"code\":900,\"message\":\"Duplicate label\",\"data\":{\"label\":\"1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67-08c5193e2255f91ce5590b110a9ae2466736be0cf48e75bcaa22e6ad8da88709-1\",\"bolt12\":\"lni1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcyyqwtp0rmsgquvuacqcl5cdfzwzmu3v8tqgvpqs8e80dlmxm7ey4xwzqrw4lauzsc2ajk26mv0ysxxmmxvejk2grxdaezqun4wd68jggvpkqqqqqqqqqqqqqqqqpgyqq7ypymf9efe2jj5r2mzunlqz67d75ht3ukxk0x9ftkcuknrgepsgupwfqpqynzqzx9rylzy40ernj4jzc3p2dwy3n8x6lqeaywwk725ghx4kx63pcf9qzxqt0dxq4zqwtz2qu44gzx7nzczc494cce2tgph5xgu5sn7vh8frky9z5n08xj9sp3yaxe9cqs5vss59r8pxwlyy3jl4xhrdqwz85xe9qqgcpda590qs9khxdx5qpetlx0j6ap0wsxagssmy2qjvhjp2kc3na54pht3pp76c405upne360lh8rzye32xxq6l0phpkk9pu9lwxnqkxuwt2nqqr9u\",\"payment_hash\":\"396250395aa046f4c58162a5ae31952d01bd0c8e5213f32e748ec428a9379cd2\",\"msatoshi\":7700446,\"amount_msat\":\"7700446msat\",\"status\":\"unpaid\",\"description\":\"Weekly coffee for rusty!\",\"expires_at\":1614832137,\"local_offer_id\":\"1cb0bc7b8201c673b8063f4c352270b7c8b0eb02181040f93bdbfd9b7ec92a67\"}}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to only set it for single-use offers (where it's required),
but it's still interesting for multi-use offers, so let's keep it
there.
We also put this field in the documentation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We weren't waiting for the channel to get to normal state:
E pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt20m1pszfpezpp549um3vuyt52rgea32g7u55a5fv29yv7t94cmr2f0vjgcc33m3dvqdqzdgxqyjw5qcqp9sp59vu4tat2n53ylzrgxa95s5wu2s885a9llud64c0f6gjqts7h6tks9qy9qsq7u0j3w2h3sxxp9axpjxkz525znjn0t92gnrgk7y6plyq39zw9994g88xxjx52egk4965dp5qt2w08hk009eq9hm8nykwmxe7r705k8gpkqu9mw'}, error: {'code': 210, 'message': 'Ran out of routes to try after 1 attempt: see `paystatus`', 'attempts': [{'status': 'failed', 'failreason': 'Cannot attempt payment, we have no channel to which we can add an HTLC', 'partid': 1, 'amount': '2000000000msat'}]}
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we turned many errors into warnings, we want our tests to fail
when they happen unexpectedly. We make WARNING clear in the strings
we print, too, to help out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And make all the callers choose which one. In general, I prefer warn,
which lets them reconnect and try again, however some places are either
stated that they must be errors in the spec itself, or in openingd
where we abandon the channel when we close the connection anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: we now send warning messages and close the connection, except on unrecoverable errors.
We construct the route manually so we may not have the
channel_announcement yet. But we can get an update from the error
packet, which can lead to:
```
2021-01-29T01:38:23.4767334Z ValueError:
2021-01-29T01:38:23.4767987Z Node errors:
2021-01-29T01:38:23.4768767Z - lightningd-1: had bad gossip messages
2021-01-29T01:38:23.4769512Z Global errors:
2021-01-29T01:38:23.4770300Z
2021-01-29T01:38:23.4771109Z contrib/pyln-testing/pyln/testing/fixtures.py:197: ValueError
...
2021-01-29T01:38:23.7820197Z lightningd-1: 2021-01-29T01:26:57.460Z DEBUG gossipd: Extracted channel_update 01027217b3086ad9f3dee1fa55b94c5fd2a4b0637bec70ba727ba4151a8de5173ddc749db3502d41ab0ae164addc8fd013d2088b6a12a2f478ae0affa94d76d8845c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000067000001000160136459010000060000000000000000000000010000000a000000003b023380 from onionreply 100d0000007500887217b3086ad9f3dee1fa55b94c5fd2a4b0637bec70ba727ba4151a8de5173ddc749db3502d41ab0ae164addc8fd013d2088b6a12a2f478ae0affa94d76d8845c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000067000001000160136459010000060000000000000000000000010000000a000000003b023380
2021-01-29T01:38:23.7837450Z lightningd-1: 2021-01-29T01:26:57.461Z DEBUG gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 103x1x1/0
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We fix up the test by using pay, instead of sendpay (and making pay log
the expected message).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: sendpay no longer extracts updates from errors, the caller should do it from the `raw_message`.
The fetchinvoice and offers plugins disable themselves if the option
isn't enabled (it's enabled by default on EXPERIMENTAL_FEATURES).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `experimental-offers` enables fetch, payment and creation of (early draft) offers.
As per lastest revision of the spec, we can specify amounts in invoice
requests even if the offer already specifies it, as long as we exceed
the amount given. This allows for tipping, and amount obfuscation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Means a reshuffle of our logic: we want to multiply by quantity before
conversion for maximum accuracy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids a footgun where they create an offer then we can't create
the invoice because they don't have a converter plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We split `send_invoice` offers inoo offerout (for want of a better name).
This simplifies the API.
Also took the opportunity to move the `vendor` tag to immediately
follow `description` (our tests use arguments by keywords, so no
change there).
Suggested-by: shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
refund offers are implied send_invoice offers. And send_invoice offers
are implied single-use offers, though it can also make sense to have
a non-send_invoice offer be single-use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Hoist 7200 constant into the bolt12 heade2.
2. Make preimage the last createinvoice arg, so we could make it optional.
3. Check the validity of the preimage in createinvoice.
4. Always output used flag in listoffers.
5. Rename wallet offer iterators to offer_id iterators.
6. Fix paramter typos.
7. Rename `local_offer_id` parameter to `localofferid`.
8. Add reference constraints on local_offer_id db fields.
9. Remove cut/paste comment.
10. Clarify source of fatal() messages in wallet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were blindly initiating the keysend payment, which could lead to
confusing outcomes. This adds a very specific error message to the
error returned.
Changelog-Fixed: keysend: Keysend now checks whether the destination supports keysend before attempting a payment. If not a more informative error is returned.
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets. We need to know this so we know when
we can make them compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In both cases the flakyness arises from the destination not knowing about the
modified fees of the forwarding node, thus including the outdated details in
the routehint, and the sender being unlucky and always trying with the
routehint anyway.
The long-term solutions to this is going to be #4111, this commit just reduces
the flakyness to get back to business.
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Fixes: #3926
(probably)
Changelog-Fixed: pay: Also limit the number of splits if the payee seems to have a low number of channels that can enter it, given the max-concurrent-htlcs limit.
I screwed up the rotation logic in an earlier varient of this PR, and
it lead me to discover why test_mpp_interference_2 was flaky.
Really, we should keep a fuzzy estimator of how much payment is
outstanding, but in practice rotation is probably good enough.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With a feerate of 7500perkw and subtracting 660 sats for anchors, a
20,000 sat channel has capacity about 9800 sat, below our default:
You gave bad parameters: channel capacity with funding 20000sat, reserves 546sat/546sat, max_htlc_value_in_flight_msat is 18446744073709551615msat, channel capacity is 9818sat, which is below 10000000msat
So bump channel amounts.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're failing this too often: we'd fail it more but it's disabled
with VALGRIND (it shouldn't be: @slow_test removes VALGRIND if SLOW_MACHINE
is set).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the simplest possible fix: increase the target amount until we get
the desired number of parts, while still bucketizing payments together that
are in approximately the same size.
The current logic puts all payments that are in the range x < amount <= 16*x
in the same bucket, making them harder to distinguish.
Changelog-Fixed: pay: The `presplit` modifier now supports large payments without exhausting the available HTLCs.
Anchor outputs break many assumptions in our tests:
1. Remove some hardcoded numbers in favor of a fee calc, so we only have to
change in one place.
FIXME: This should also be done for elements!
2. Do binary search to get feerate for a given closing fee.
3. Don't assume output #0: anchor outputs perturb them.
4. Don't assume we can make 1ksat channels (anchors cost 660 sats!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: ZmnSCPxj
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: pay: Correct a case where we put the sub-payment value instead of the *total* value in the `total_msat` field of a multi-part payment.
The worst effect is that unpublished nodes are harder to pay, but
even published ones make us do unnecessary work, since we are
losing routehints from the published ones that could help us
actually route better to them.
listpays: make doc-all missed
Changelog-Added: JSON-RPC: `listpays` can be used to query payments using the `payment_hash`
Changelog-Added: JSON-RPC: `listpays` now includes the `payment_hash`
And when it's set, and we're SLOW_MACHINE, simply disable valgrind.
Since Travis (SLOW_MACHINE=1) only does VALGRIND=1 DEVELOPER=1 tests,
and VALGRIND=0 DEVELOPER=0 tests, it was missing tests which needed
DEVELOPER and !VALGRIND.
Instead, this demotes them to non-valgrind tests for SLOW_MACHINEs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I started replacing all get_node() calls, but got bored, so then just did the
tests which call get_node() 3 times or more.
Ends up not making a measurable speed difference, but it does make some
things neater and more standard.
Times with SLOW_MACHINE=1 (given that's how Travis tests):
Time before (non-valgrind):
393 sec (had 3 failures?)
Time after (non-valgrind):
410 sec
Time before (valgrind):
890 seconds (had 2 failures)
Time after (valgrind):
892 sec
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: pay: Fixed a bug where routehints would be ignored if the payment exceeded 10,000 satoshi. This is particularly bad if the payee is only reachable via routehints in an invoice.
```
# Excludes channel, then ignores routehint which includes that, then
# it excludes other channel.
> assert len(status) == 2
E assert 1 == 2
E -1
E +2
```
The invoice we use at the end has a routehint: 50% of the time it's
to l2 (which fails), 50% to l5 (which succeeds).
Change it to create invoice before channel with l5 so it does the
retry like we expect here.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We sum up the amounts just like we do with `amount_sent`, however we may have
parts that don't have that annotation, so make it optional.
Suggested-by: Rusty Russell <@rustyrussell>
Since we started using `sendonion` in the `pay` plugin we no longer
automatically have the `amount` annotation on (partial) payments. This
replicates the issue so we can fix it.
Reported-by: Rusty Russell <@rustyrussell>
The adaptive MPP test was showing an issue with always using a routehint, even
when it wasn't necessary: we would insist on routhing to the entrypoint of the
routehint, even through the actual destination. If a channel on that loop
would result being over capacity we'd slam below 0, and then increase again by
unapplying the route. The solution really is not to insist on routing through
a routehint, so we implement random skipping of routehints, and we rotate them
if we have multiples.
This PR includes the fix discussed on PR #3855. This fix was tested with the use case described inside the issue and worked.
Fixes: #3855
Changelog-None
The test had part 1 and 2 backward, but still worked. When I copied that to
*after* the test had succeeded, it complained. It should always complain,
to catch bugs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This wasn't important before, but now we have MPP it's good to enforce.
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This exercises something that is simply not possible without MPP, i.e., the
bundling of multiple paths to get sufficient capacity to perform the payment.
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.
Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.
Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
This makes use of the payment modifier structure to just add the preimage to
the TLV payload for the last hop.
Changelog-Added: JSON-RPC: The `keysend` command allows sending to a node without requiring an invoice first.
This commit collects the changes required to the tests caused by the changes
to the `pay` and `paystatus` commands. They are also rather good hints as to
what these changes entail.
Make sure we've actually confirmed the HTLC; if it's not confirmed yet
then we won't fast-fail it, and we'll timeout instead:
```
> l1.rpc.waitsendpay(payment_hash=inv['payment_hash'], timeout=TIMEOUT, partid=1)
E AssertionError: Pattern 'WIRE_PERMANENT_CHANNEL_FAILURE \\(reply from remote\\)' not found in "RPC call failed: method: waitsendpay, payload: {'payment_hash': 'c186643391469aa8190415496c85b1eb789cb2b756a76d4c9ce21dd34c698d92', 'timeout': 30, 'partid': 1}, error: {'code': 200, 'message': 'Timed out while waiting'}"
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The status of what started as a simple JSON-RPC call is now spread across an
entire tree of partial payments and payment attempts. So we collect the status
in a single struct in order to report back success of failure.
This commit can be reverted/skipped once we have implemented all the logic and
have feature parity with the normal `pay`. It's main purpose is to expose the
unfinished functionality to test it, without completely breaking the existing
`pay` command.
We've been seeing some Travis timeouts under VALGRIND, with the
10 second timeout here: use TIMEOUT as per standard.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And the percentage of the initial amount, not the constently increasing
one !
Changelog-Fixed: pay: we now respect maxfeepercent, even for tiny amounts.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Previously we've used the term 'funder' to refer to the peer
paying the fees for a transaction; v2 of openchannel will make
this no longer true. Instead we rename this to 'opener', or the
peer sending the 'open_channel' message, since this will be universally
true in a dual-funding world.
The documentation was wrong, and I copied my mistake to `libplugin` where it
was then ignored instead of ORed into the node's featurebits. This fixes both.
We weren't actually waiting until l3 got the channel_update from l2,
so it might not be able to create the routehint.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When route returns a result which is too expensive, we try to figure out which
hop is most expensive to exclude it for next time.
If it's a single-hop route, we don't count it, since the first hop is free.
That's not usually a problem, since single-hop routes can't exceed our limits
(they're always "free"!).
But if we are using a routehint, the total cost could exceed our limits,
even if the start of the routehint is a single hop away.
This reproduces that test case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is what actually lets us pay blinded invoices.
Unfortunately, our internal logic assumes every hop in a path has a
next `short_channel_id`, so we have to use a dummy. This is
sufficient for testing, however.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sending update_fee immediately after channel establishment seems to
upset LND, so work around it by deferring it. The reason we increase
the fee after establishment is because now we might need to close the
channel in a hurry due to htlcs, but until there are htlcs that's
unnecessary.
Fixes: #3596
Changelog-Changed: Added workaround for lnd rejecting our commitment_signed when we send an update_fee after channel confirmed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The spec states that invoices with an amount, but lacking a multiplier, should
be interpreted as integer Bitcoin amounts:
`amount`: optional number in that currency, followed by an optional
`multiplier` letter. The unit encoded here is the 'social' convention of a
payment unit -- in the case of Bitcoin the unit is 'bitcoin' NOT satoshis.
Suggested-by: Stefano Pellegrini <@St333p>
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: invoice: The invoice parser assumed that an amount without a multiplier was denominated in msatoshi instead of bitcoins.
A CONSERVATIVE/3 target for them.
Some noisy changes to the tests as we had to update the estimatesmartfee
mock.
Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
We are returning a `BADONION` error despite the cause being an invalid onion
payload containing an unknown even TLV type. It really should return
`INVALID_ONION_PAYLOAD` errors instead.
Add new check if we're funder trying to add HTLC, keeping us
with enough extra funds to pay for another HTLC the peer might add.
We also need to adjust the spendable_msat calculation, and update
various tests which try to unbalance channels. We eliminate
the now-redundant test_channel_drainage entirely.
Changelog-Fixed: Corner case where channel could become unusable (https://github.com/lightningnetwork/lightning-rfc/issues/728)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using it with a different value to the amount sent causes a crash in 0.8.0,
which is effectively deprecating it, so let's disallow it now.
Changelog-Changed: If the optional `msatoshi` param to sendpay for non-MPP is set, it must be the exact amount sent to the final recipient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Detect a previously non-permanent error (`final_cltv_too_soon`) that has been merged into a permanent error (`incorrect_or_unknown_payment_details`), and retry that failure case in `pay`.
We still close the channel if we *send* an error, but we seem to have hit
another case where LND sends an error which seems transient, so this will
make a best-effort attempt to preserve our channel in that case.
Some test have to be modified, since they don't terminate as they did
previously :(
Changelog-Changed: quirks: We'll now reconnect and retry if we get an error on an established channel. This works around lnd sending error messages that may be non-fatal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Thanks to @t-bast, who made this possible by interop testing with Eclair!
Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
