Commit Graph

1515 Commits

Author SHA1 Message Date
Christian Decker
b813974e13 mpp: Add the presplit MPP modifier
Changelog-Added: The MPP presplit modifier splits large payments into 10k satoshi parts to maximize chances of performing the payment and to obfuscate the overall amount being sent.
2020-07-15 11:32:58 +02:00
Rusty Russell
371cabf976 txprepare: revert 1fb9a078b6 (psbt field)
We're actually going to deprecate this, so don't add new features!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: ***REMOVE*** JSON-API: `txprepare` returns a psbt version of the created transaction
2020-07-15 18:49:02 +09:30
Rusty Russell
e85c895c5b pytest: restore test_sign_and_send_psbt.
It uses reservations heavily, and assumed we generated change, etc.
It's now a simpler test, in many ways.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell
31d7e013bd pytest: test fundpsbt.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell
4ee527a59c pytest: test reserve and unreserve.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell
56ea215ba0 wallet: new JSON commands reserveinputs and unreserveinputs.
reserveinputs marks UTXOs reserved for 12 hours, so we won't select them
for spending: unreserveinputs marks them available again.

Exposes param_psbt() for wider use.

Disabled the test_sign_and_send_psbt since we're altering the API;
the final patch restores it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-15 18:49:02 +09:30
Rusty Russell
899ec2b3d4 JSON API: fix up two existing warnings to be conformant.
Technically an API break, but nobody relies on these I hope!

Note that the feerates warning was buried inside the style object:
it should be top-level.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-14 21:28:26 +02:00
Christian Decker
02e44e9903 paymod: Set the STOPPED_RETRYING status code if we stopped retrying 2020-07-14 06:25:29 +09:30
Christian Decker
5776a33116 paymod: Activate paymod and move legacy pay to legacypay command
As suggested during the paymod-03 review it is better to activate the new code
right away, and give users an escape hatch to use the legacy code instead. The
way I implemented it allows using either `legacypay` or `pay` and then set
`legacy` to switch to the other implementation.

Changelog-Added: JSON-RPC: The `pay` command now uses the new payment flow, the new `legacypay` command can be used to issue payment with the legacy code if required.

Suggested-by: Rusty Russell <@rustyrussell>
Suggested-by: ZmnSCPxj <@zmnscpxj>
2020-07-14 06:25:29 +09:30
Christian Decker
01a475161d paymod: Implement keysend sending support as a native RPC command
This makes use of the payment modifier structure to just add the preimage to
the TLV payload for the last hop.

Changelog-Added: JSON-RPC: The `keysend` command allows sending to a node without requiring an invoice first.
2020-07-13 14:19:16 +02:00
niftynei
ccb78712cf elements,tests: fixup output amounts for elements
no idea why the fees are different here now post pset, but lo and behold
they are...
2020-07-13 11:37:24 +09:30
niftynei
bea05b1250 elements,tests: skip over fee type outputs
They don't contain addresses
2020-07-13 11:37:24 +09:30
niftynei
eec5dfe6aa elements: update output amounts for elements test
The amount we expect back, post change, is different for an elements tx
2020-07-13 11:37:24 +09:30
Christian Decker
0aa5c197ec pytest: Fix tests broken by the pay and paystatus changes
This commit collects the changes required to the tests caused by the changes
to the `pay` and `paystatus` commands. They are also rather good hints as to
what these changes entail.
2020-07-07 23:25:45 +02:00
Christian Decker
2ac87c1490 paymod: Reorder paymod arguments to match pay
We've been adding modifiers and arguments out of order, and we need the
arguments order to match up if we want `paymod` to be a drop-in replacement.
2020-07-07 23:25:45 +02:00
Christian Decker
da8eb6fb4c paymod: Expose riskfactor and wire through to getroute 2020-07-07 23:25:45 +02:00
Christian Decker
5601a1ab21 paymod: Add a deadline to the pay command and retry modifier 2020-07-07 23:25:45 +02:00
Christian Decker
f557955515 paymod: Consolidate parsing RPC results in libplugin
We handle these in a number of different ways, and regularly get the parsing
and logic for optional fields wrong, so let's consolidate them here.
2020-07-07 23:25:45 +02:00
Christian Decker
6a75497eb2 pytest: Add a compat() fixture allowing checks for specific flags
Since we change the interface for `pay` and `paystatus` we need a way to
determine which one to expect. This just parses the COMPAT_Vxyz flags and
allows us to check if, based on the configuration, we are in compat mode for
that set of changes or not.
2020-07-07 23:25:45 +02:00
Christian Decker
cb00cbac7c paymod: Add support for the msatoshi override argument 2020-07-07 23:25:45 +02:00
Christian Decker
2331cd62e1 paymod: Add user-provided label back into the paystatus result 2020-07-07 23:25:45 +02:00
Christian Decker
935578e567 paymod: Exclude most expensive/slowest chan if limits are exceeded 2020-07-07 23:25:45 +02:00
Christian Decker
4648d2867f paymod: Compute fee and CLTV delta limits for a payment
So far we got away with not caring about these but since we're implementing
modifiers that impact these limits, we better keep track of them.
2020-07-07 23:25:45 +02:00
Christian Decker
2f0e535b81 paymod: Add reason why a payment was attempted
This is a slight change in interface in that the string no longer spells out
that a specific node was excluded.
2020-07-07 23:25:45 +02:00
Rusty Russell
3fec96a7b1 tests: fix flake in test_partial_payment_htlc_loss
Make sure we've actually confirmed the HTLC; if it's not confirmed yet
then we won't fast-fail it, and we'll timeout instead:

```
>           l1.rpc.waitsendpay(payment_hash=inv['payment_hash'], timeout=TIMEOUT, partid=1)
E           AssertionError: Pattern 'WIRE_PERMANENT_CHANNEL_FAILURE \\(reply from remote\\)' not found in "RPC call failed: method: waitsendpay, payload: {'payment_hash': 'c186643391469aa8190415496c85b1eb789cb2b756a76d4c9ce21dd34c698d92', 'timeout': 30, 'partid': 1}, error: {'code': 200, 'message': 'Timed out while waiting'}"
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-07 17:02:21 +09:30
Rusty Russell
1708fb91e5 wallet: clean up json output creation, part 2.
It looked like we weren't printing the address on closing outputs.

But we are, because the 'scriptPubkey' field is in the 'outputs' db
table since 0.7.3 (66a47d2761).

So make the logic clearer, and remove a completely bogus comment (UTXOs
with closing_info are definitely spendable!).

We export the json_add_utxos() for future use, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-06 19:25:05 +02:00
Rusty Russell
7aa8ffa2a0 bitcoin: add weight calculation helpers.
These are pulled from wallet/wallet.c, with the fix now that we grind sigs.

This reduces the fees we pay slightly, as you can see in the coinmoves changes.

I now print out all the coin moves in suitable format before we match:
you only see this if the test fails, but it's really helpful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-06 19:25:05 +02:00
Rusty Russell
fc2561fd9e pytest: fix test breakage added by d0c85033d2
By setting nLocktime to the current block, the reorg test
"test_funding_reorg_remote_lags" actually drops the funding transaction
entirely when a reorg happens.

Except the 1 in 10 cases where nLocktime is randomly set to 1-10
blocks earlier.

This implies, strongly, that we hit "restart" too often on Travis.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-07-06 14:55:14 +09:30
Christian Decker
ddb0424a4d paymod: Remove the dummy payment modifier
It was there only for demonstration purposes, and is no longer useful.
2020-07-02 11:54:12 +02:00
Christian Decker
81dce4096e pytest: Fix up test_pay_no_secret to wait for node_announcements
This was racy since we didn't know whether the peer supports TLV payloads yet
so we defaulted to legacy, which doesn't support secrets.
2020-07-02 11:54:12 +02:00
Christian Decker
b5c9dcab5a pytest: Fix up the payload replacement test
There is a race between `getroute` learning that our peer accepts TLVs and us
initiating the payment. Waiting for announcements ensures we always use TLVs,
matching our expectation in the test / plugin.
2020-07-02 11:54:12 +02:00
ZmnSCPxj jxPCSnmZ
deabab8934 lightningd/opening_control.c: fundchannel_cancel no longer requires a channel_id argument.
Fixes: #3785

Changelog-Changed: `fundchannel_cancel` no longer requires its undocumented `channel_id` argument after `fundchannel_complete`.
2020-07-02 01:10:59 +00:00
niftynei
8185474bfb liquid-compat: update coinmove check to be liquid compatible
🎉 :waterfall:
2020-07-01 19:50:02 -05:00
niftynei
ba6e4b6ef2 tests: mark as regtest only 2020-07-01 19:50:02 -05:00
niftynei
20193496bb tests: use chain's directory for finding sqlite3 file (works for liquid-regtests)
Move hardcoded 'regtest' to chainparams one
2020-07-01 19:50:02 -05:00
ZmnSCPxj jxPCSnmZ
d0c85033d2 wallet/walletrpc.c: txprepared transactions now use current tip blockheight by default.
Changelog-Changed: `txprepare` now prepares transactions whose `nLockTime` is set to the tip blockheight, instead of using 0. `fundchannel` will use `nLockTime` set to the tip blockheight as well.
2020-07-01 15:17:33 +00:00
Christian Decker
04c9c5e0d1 paymod: Collect and return results of a tree of partial payments
The status of what started as a simple JSON-RPC call is now spread across an
entire tree of partial payments and payment attempts. So we collect the status
in a single struct in order to report back success of failure.
2020-07-01 12:19:02 +02:00
Christian Decker
23b4dca3c7 paymod: Add a simple test-command to test the paymod state-machine
This commit can be reverted/skipped once we have implemented all the logic and
have feature parity with the normal `pay`. It's main purpose is to expose the
unfinished functionality to test it, without completely breaking the existing
`pay` command.
2020-07-01 12:19:02 +02:00
Rusty Russell
6b3bcd49d2 tests: fix flake in test_penalty_htlc_tx_fulfill
If the daemon already knows about the channel before it was stopped,
it won't get this message from gossipd.  That's OK, since we explicitly
test for the channel being active two lines down.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-30 13:50:30 +00:00
Rusty Russell
dd4a1a3510 tests: fix flake in test_txprepare_restart
Detection of reserved outputs is async, going via bitcoind.  Wait for them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-30 13:50:30 +00:00
niftynei
2e9c387f45 coin_moves: update withdrawal logic to account for 'variable owner' txs
Our existing coin_moves tracking logic assumed that any tx we had an
input in belonged to *all* of our wallet (not a bad assumption as long
as there was no way to update a tx that spends our wallets)

Now that we've got `signpsbt` implemented, however, we need to be
careful about how we account for withdrawals. For now we do a best guess
at what the feerate is, and lump all of our spent outputs as a
'withdrawal' when it's impossible to disambiguate
2020-06-29 16:10:05 +02:00
niftynei
9830c94778 rpc: new signpsbt + sendpsbt rpcs
Changelog-Added: JSON-RPC: new call `signpsbt` which will add the wallet's signatures to a provided psbt
Changelog-Added: JSON-RPC: new call `sendpsbt` which will finalize and send a signed PSBT
2020-06-29 16:10:05 +02:00
niftynei
103dce63ef reserve/unreserve input: new RPC commands for reserving inputs/outputs
Reserve and unreserve wallet UTXOs using a PSBT which includes those
inputs.

Note that currently we unreserve inputs everytime the node restarts.
This will be addressed in a future commit.

Changelog-Added: JSON-RPC: Adds two new rpc methods, `reserveinputs` and `unreserveinputs`, which allow for reserving or unreserving wallet UTXOs
2020-06-29 16:10:05 +02:00
niftynei
a66415a43e plugins-test: use pyln.client, not lightning to import Plugin 2020-06-23 14:49:32 +02:00
ZmnSCPxj jxPCSnmZ
c100de6d93 common/utils.c: Correctly handle NULL taken pointer in tal_dup_talarr.
Fixes: #3757

Reported-by: @sumBTC

Changelog-None
2020-06-23 12:48:20 +09:30
ZmnSCPxj jxPCSnmZ
a25af262e9 tests/test_pay.py: Add test to replicate #3757. 2020-06-23 12:48:20 +09:30
ZmnSCPxj jxPCSnmZ
5db69f1b41 lightningd/opening_control.c: Remove 'Try fundchannel_cancel again' error.
Changelog-Changed: `fundchannel_cancel` will now succeed even when executed while a `fundchannel_complete` is ongoing; in that case, it will be considered as cancelling the funding *after* the `fundchannel_complete` succeeds.

Let me introduce the concept of "Sequential Consistency":
All operations on parallel processes form a single total order agreed upon by all processes.

So for example, suppose we have parallel invocations of `fundchannel_complete` and `fundchannel_cancel`:

                          +--[fundchannel_complete]-->
                          |
    --[fundchannel_start]-+
                          |
                          +--[fundchannel_cancel]---->

What "Sequential Consistency" means is that the above parallel operations can be serialized as a single total order as:

    --[fundchannel_start]--[fundchannel_complete]--[fundchannel_cancel]-->

Or:

    --[fundchannel_start]--[fundchannel_cancel]--[fundchannel_complete]-->

In the first case, `fundchannel_complete` succeeds, and the `fundchannel_cancel` invocation also succeeds, sending an `error` to the peer to make them forget the chanel.

In the second case, `fundchannel_cancel` succeeds, and the succeeding `fundchannel_complete` invocation fails, since the funding is already cancelled and there is nothing to complete.

Note that in both cases, `fundchannel_cancel` **always** succeeds.

Unfortunately, prior to this commit, `fundchannel_cancel` could fail with a `Try fundchannel_cancel again` error if the `fundchannel_complete` is ongoing when the `fundchannel_cancel` is initiated.
This violates Sequential Consistency, as there is no single total order that would have caused `fundchannel_cancel` to fail.

This commit is a minimal patch which just reschedules `fundchannel_cancel` to occur after any `fundchannel_complete` that is ongoing.
2020-06-22 03:15:55 +00:00
Antoine Poinsot
4302afd9a5 rpc: don't go below feerate_floor when converting vbytes
We passed below the floor when the user specified `1000perkb`.
Matt Whitlock says :

    I was withdrawing with feerate=1000perkb, which should be the minimum-allowed fee rate. Indeed, bitcoin-cli getmempoolinfo reports:

    {
      "loaded": true,
      "size": 15097,
      "bytes": 9207924,
      "usage": 32831760,
      "maxmempool": 64000000,
      "mempoolminfee": 0.00001000,
      "minrelaytxfee": 0.00001000
    }

Changelog-fixed: rpc: The `feerate` parameters now correctly handle the standardness minimum when passed as `perkb`.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Reported-by: Matt Whitlock
2020-06-18 14:07:00 +02:00
Antoine Poinsot
0b99e2cc42 pytest: reproduce issue #3591
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-06-18 14:07:00 +02:00
niftynei
2900da6112 migrations: ignore channels that don't have a peer_id
We erase peer data after the last channel close transaction for that
peer is 100 blocks deep. We were failing to finish the migration because
the peer_id lookup on these was failing.

Now we ignore any channel with a null peer_id.

Fixes #3768
2020-06-17 14:30:41 +02:00
Christian Decker
cbc959ef30 pytest: Mark test_funding_push as network-specific
It checks for both exact values, that may differ due to fees, as well as the
unit which varies from network to network.
2020-06-15 16:14:53 +02:00
niftynei
971f615695 psbt: affirm database upgrade works for last_tx -> psbt
We use a database snapshot with 3 channels -- two of which have HTLCs
dangling and one is an initial open channel tx in the 'old' tx hex
format in last_tx and confirm that they are successfully updated to PSBT
format on start.
2020-06-11 13:13:13 +02:00
Rusty Russell
ce9e559aed features: do not set option_support_large_channel in channel_announcement.
Spec is wrong (it says it should be compulsory), and Eclair doesn't set it
at all, leading to an error when they send their announcement_signatures.

Fixes: #3703
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: large-channels: negotiate successfully with Eclair nodes.
2020-05-26 19:57:29 -05:00
Rusty Russell
b3eef81f19 pytest: add test for channel_announcement feature bits.
And neaten current feature mangling code now we have a wumbo flag for
expected_peer_features().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-26 19:57:29 -05:00
Rusty Russell
f4f8a363dd pytest: fix feature mask for EXPERIMENTAL_FEATURES, add wumbo support.
There are various places where our tests failed with
--enable-expimental-features.  And our plugin test overlapped an
existing feature.

We make our expected_feature functions more generic, and use them
everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-26 19:57:29 -05:00
Antoine Poinsot
5dfd2436db wallet: show input txid in little endian for json_listtransactions
Changelog-Fixed: jsonrpc: `listtransactions` now displays all txids as little endian

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-22 15:11:51 +02:00
Antoine Poinsot
1061cd95c9 pytest: add a sanity check for the listtransactions command
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-22 15:11:51 +02:00
lisa neigut
1fb9a078b6 txprepare: return psbt serialized version of tx as well
Changelog-Added: JSON-API: `txprepare` returns a psbt version of the created transaction
2020-05-21 18:45:07 +09:30
Antoine Poinsot
0088147d9e pytest: add sanity check for --commit-fee
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-20 06:09:24 +09:30
Rusty Russell
cfb320c972 wire: move remaining bitcoin functions out to bitcoin/ files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-18 14:51:12 +02:00
Rusty Russell
abb16b4226 cli: implement new 'flattened JSON' mode.
Much nicer for grepping, since `{ "foo": { "bar": [7] } }` is turned into
`foo.bar[0]=7`.

Changelog-Added: cli: New `--flat` mode for easy grepping.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 15:25:44 +02:00
Rusty Russell
679d3494b4 cli: tweak -H output to remove format-hint fields.
-H removes the top-level if there's only one, and 'format-hint'
breaks this heuristic, so we end up with:

```
help=command=autocleaninvoice [cycle_seconds] [expired_by]
category=plugin
description=Set up autoclean of expired invoices.
verbose=Perform cleanup every {cycle_seconds} (default 3600), or disable autoclean if 0. Clean up expired invoices that have expired for {expired_by} seconds (default 86400).
command=check command_to_check
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-15 15:25:44 +02:00
Rusty Russell
1fb505d118 pytest: extend timeout for test_payment_duplicate_uncommitted
We've been seeing some Travis timeouts under VALGRIND, with the
10 second timeout here: use TIMEOUT as per standard.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-12 21:57:43 +02:00
lisa neigut
d35149e5ca try to fix travis test flakes 2020-05-12 15:46:58 +09:30
lisa neigut
487771aee5 tests: skip longer htlc tests on valgring+slow machines
These guys take a while to run, so let's just skip them on the
valgrind/slow-machine combos :/
2020-05-12 15:46:58 +09:30
lisa neigut
6e4b4c50f0 coin tests: try to make a bit more travis robust 2020-05-12 15:46:58 +09:30
lisa neigut
8acbbca05d coins: use the chain's BIP173 name instead of a 'unit of account'
Updates the unit of account to be the chain_id, which is the BIP173 name
of the chain that the coins moved on.

Suggested-By: @rustyrussell
2020-05-12 15:46:58 +09:30
lisa neigut
4bfbb58c56 coins: fix feerate for withdraw so test work? 2020-05-12 15:46:58 +09:30
lisa neigut
087ab166b3 coins, fix: don't crash if the to_us amount is greater than our_msat
It's possible for our peer to publish a commitment tx that has already
updated our balance for an htlc before we've completed removing it from
our commitment tx (aka before we've updated our balance). This used to
crash, now we just update our balance (and the channel balance logs!)
and keep going.

If they've removed anything from our balance, we'll end up counting it
as chain_fees below. Not ideal but fine... probably.
2020-05-12 15:46:58 +09:30
lisa neigut
ffd9467f14 coin moves: persist the coin movement index counter to disk
Should make it easier to track when coin moves in the plugin are
disjoint from what c-lightning says it's broadcast already.
2020-05-12 15:46:58 +09:30
lisa neigut
de86e29e16 coin moves: log all withdrawals when confirmed in a block
This moves the notification for our coin spends from when it's
successfully submited to the mempool to when they're confirmed in a
block.

We also add an 'informational' notice tagged as `spend_track` which
can be used to track which transaction a wallet output was spent in.
2020-05-12 15:46:58 +09:30
lisa neigut
e9d26a46e0 coin moves: actually record the blockheight for all chain moves
Previously we were annotating every movement with the blockheight of
lightningd at notification time. Which is lossy in terms of info, and
won't be helpful for reorg reconciliation. Here we switch over to
logging chain moves iff they've been confirmed.

Next PR will fix this up for withdrawals, which are currently tagged
with a blockheight of zero, since we log on successful send.
2020-05-12 15:46:58 +09:30
lisa neigut
9caf20f636 coin moves: don't log coin moves in onchaind if we're replaying
On node start we replay onchaind's transactions from the database/from
our loaded htlc table.  To keep things tidy, we shouldn't notify the
ledger about these, so we wrap pretty much everything in a flag that
tells us whether or not this is a replay.

There's a very small corner case where dust transactions will get missed
if the node crashes after the htlc has been added to the database but
before we've successfully notified onchaind about it.

Notably, most of the obtrusive updates to onchaind wrappings are due to
the fact that we record dust (ignored outputs) before we receive
confirmation of its confirmation.
2020-05-12 15:46:58 +09:30
lisa neigut
1b5221cbf5 coin moves tests: push_msat and the wallet withdrawal
Check that we account for push_msat and wallet withdrawal/deposits
correctly
2020-05-12 15:46:58 +09:30
lisa neigut
8c986d67db coin moves: test all of the onchaind variants
Mostly we update existing tests to account for channel balances. In a
few places, new tests were needed as there wasn't an existing pathway
that tested the chain-fees for a few penalty cases
2020-05-12 15:46:58 +09:30
lisa neigut
41d3471c7f coin moves tests: save updates to disk
If we don't save to disk, if the node restarts we'll lose them all
and the resulting balance check at the end will be incorrect.
2020-05-12 15:46:58 +09:30
lisa neigut
6ee6cdc280 coin moves: record their attempted cheat (and our handling thereof)
For cheats, we do a little bit of weird accounting. First we 'update'
our on-ledger balance to be the entirety of the channel's balance. Then,
as outputs get resolved, we record the fees and outputs as withdrawals
from this amount.

It's possible that they might successfully 'cheat', in which case we
record those as 'penalty' but debits (not credits).
2020-05-12 15:46:58 +09:30
lisa neigut
9c4c0f10fb coin_mvt: add integration tests for in-channel htlc movements 2020-05-12 15:46:58 +09:30
Christian Decker
8f2ce1e638 pytest: Add a test for the commitment_revocation hook 2020-05-07 15:05:39 +09:30
Rusty Russell
046b402c18 gossipd: return channel_announcement features for listchannels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON API: `listchannels` now shows channel `features`.
2020-05-07 08:44:58 +09:30
Rusty Russell
855debcfe1 gossipd: upgrade v7 gossip_store to v8.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Rusty Russell
46793bdaaf pytest: test gossip_store upgrade from version 7 to version 8.
The previous patch changed the gossip_store, but in a trivial way.
The next patch will implement upgrading, so this is the test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Rusty Russell
78ffea61e1 channeld: tell gossipd what the features are for our local channels.
This msg is stored in the gossip_store, so it means a version bump.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Antoine Poinsot
4bb7b4621f pay: respect maxfeepercent when choosing a shadow route
And the percentage of the initial amount, not the constently increasing
one !

Changelog-Fixed: pay: we now respect maxfeepercent, even for tiny amounts.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-05 14:10:42 +09:30
Antoine Poinsot
f83a1e0f2d Reproduce issue #3684
Fixed-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-05-05 14:10:42 +09:30
Rusty Russell
fe365f930f lightningd: list disabled plugins in listconfig.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-05 13:45:17 +09:30
Rusty Russell
24063ca972 lightningd: have plugin-disable be more persistent.
The previous implementation was a bit lazy: in particular, since we didn't
remember the disabled plugins, we would load them on rescan.

Changelog-Changed: config: the `plugin-disable` option works even if specified before the plugin is found.
2020-05-05 13:45:17 +09:30
Rusty Russell
8c59de5ee5 lightningd: make plugin_kill take a simple string.
That's more convenient for most callers, which don't need a fmt.

Fixed-by: Darosior <darosior@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-05 13:45:17 +09:30
Rusty Russell
051cbf7cc4 lightningd: make plugin_kill() free the plugin.
This is what I expected from plugin_kill, and now all the callers do the
equivalent anywat, it's easy.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-05 13:45:17 +09:30
Rusty Russell
4de11950ec lightningd: unify dynamic and static plugin initialization.
This means we now clean up options in startup plugins (that was only
done by dynamic code!), and now they both share the 60 second timeout
instead of 20 seconds for dynamic.

For the dynamic case though, it's 60 seconds to both complete
getmanifest and init, which seems fair.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-05 13:45:17 +09:30
Rusty Russell
006ab1e367 pytest: test loading all plugins at once, including failing ones.
We modify the slow_init() so it doesn't go too slowly for this test.

This demonstrates a crash, where we currently try to fail a command
multiple times.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-05 13:45:17 +09:30
Rusty Russell
6441233d2b pytest: add test for a plugin which falls over outside a command.
This actually passes fine, but it's an interesting case to test.

Fixed-by: Darosior <darosior@protonmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-05 13:45:17 +09:30
Christian Decker
7b752e00d5 pyln: Set the bitcoin datadir when running tests
Telling `lightningd` to pass a `-datadir` to `bitcoin-cli` so it doesn't go
snooping where it doesn't belong (i.e., the user's home directory and config).

Changelog-None
Suggested-by: Simon Vrouwe <@SimonVrouwe>
Signed-off-by: Christian Decker <@cdecker>
2020-05-05 13:28:54 +09:30
rbndg
241fa00e97 plugin:added invoice creation event
New invoice_creation event triggered when an new invoice is created

Changelog-Added: plugin: New invoice_creation plugin event
2020-05-04 19:18:05 +02:00
Rusty Russell
7abff309e5 pytest: Fix flakiness in test_feerate_stress.
We occasionally get a temporary_channel_failure too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-04 18:56:33 +09:30
Rusty Russell
4eb1233ccb lightningd: don't report spurious temporary_node_failure on local failures.
I noticed the following in logs for tests/test_connection.py::test_feerate_stress:

```
DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Failing HTLC 18446744073709551615 due to peer death
DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: local_routing_failure: 8194 (WIRE_TEMPORARY_NODE_FAILURE)
```

This is because it reports the (transient) node_failure error, because
our channel_failure message is incomplete.  Fix this wart up.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-04 18:56:33 +09:30
Rusty Russell
75496ad168 pytest: fix flaky test_peerinfo
If we don't wait for close tx to reach mempool, it might not get to
depth 100, and we don't get 'onchaind complete, forgetting peer'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-04 18:56:33 +09:30
Rusty Russell
4df9b2c5f2 pytest: fix flake in test_onchain_different_fees()
We didn't wait until l2 processed the final state of HTLC #2, so
it might not include it when it drops onchain, leading to us only
getting 3 (not 4) sendrawtx calls.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-04 18:56:33 +09:30
lisa neigut
0e20e3c5e7 df: rename 'funder' to 'opener'
Previously we've used the term 'funder' to refer to the peer
paying the fees for a transaction; v2 of openchannel will make
this no longer true. Instead we rename this to 'opener', or the
peer sending the 'open_channel' message, since this will be universally
true in a dual-funding world.
2020-05-04 10:22:26 +09:30
Christian Decker
e7b555dd90 pytest: Stabilize test_penalty_{in,out}htlc tests
They were looking for specific amounts which proved to be rather flaky. Now
they look for specific outputs being available in the `listfunds` result after
everything was settled.
2020-05-01 18:20:33 -05:00
lisa neigut
f2aa770a3e tests: since features are longer now, so bytes read is longer
Since the node announcements now include the 55th bit flag for keysends,
the total amount of bytes read from disk is now +8
2020-04-28 10:39:07 -05:00
lisa neigut
d6d26dd056 features: split expected feature bits into node/peer sets
The new `keysend` plugin modifies the node features that we send to
peers. This commit breaks out the 'expected_features' we use for tests
to encompass this differentiation.
2020-04-28 10:39:07 -05:00
Christian Decker
9f298330b2 libplugin: It's featurebits, not features
The documentation was wrong, and I copied my mistake to `libplugin` where it
was then ignored instead of ORed into the node's featurebits. This fixes both.
2020-04-28 10:39:07 -05:00
Christian Decker
8c23f22591 pytest: Reproduce issue #3666
It seems we aren't registering the featurebits correctly.
2020-04-28 10:39:07 -05:00
Rusty Russell
bf053dcf7c plugins: simplify htlc_accepted hook payload-setting API.
As discussed with Christian, prepending the length to the payload returned
is awkward, but it's the only way to set a legacy payload.  As this will
be soon deprecated, simplify the external API.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-16 18:03:35 +09:30
Rusty Russell
63441075b5 lightningd: allow htlc_accepted hook to replace onion payload.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-added: `htlc_accepted` hook can now offer a replacement onion `payload`.
2020-04-16 18:03:35 +09:30
Christian Decker
e03acd9663 libplugin: Add features to plugin_main and getmanifest 2020-04-16 18:03:35 +09:30
Christian Decker
568773daad pytest: Add a test for keysend
It currently uses a borrowed sending implementation from the noise plugin, but
we'll implement that functionality in the native keysend plugin next.
2020-04-16 18:03:35 +09:30
Rusty Russell
284bd7de0c pytest: fix flaky test_excluded_adjacent_routehint
We weren't actually waiting until l3 got the channel_update from l2,
so it might not be able to create the routehint.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-15 20:52:13 -05:00
Rusty Russell
1dc281c111 pytest: test for pay adjacent routehint crash.
When route returns a result which is too expensive, we try to figure out which
hop is most expensive to exclude it for next time.

If it's a single-hop route, we don't count it, since the first hop is free.
That's not usually a problem, since single-hop routes can't exceed our limits
(they're always "free"!).

But if we are using a routehint, the total cost could exceed our limits,
even if the start of the routehint is a single hop away.

This reproduces that test case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 14:13:26 -05:00
Rusty Russell
df31af5221 invoice: ignore dead-end heuristic on explicitly specified channels.
This makes testing easier, and makes sense: lightningd might not
*know* about other connected channels, depending on gossip, but if the
user specifies it we should obey it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON: `invoice` `exposeprivatechannels` now includes explicitly named channels even if they seem like dead-ends.
2020-04-14 14:13:26 -05:00
Rusty Russell
1d29228136 lightningd: make sendpay support blinding and enctlv fields (EXPERIMENTAL)
This is what actually lets us pay blinded invoices.

Unfortunately, our internal logic assumes every hop in a path has a
next `short_channel_id`, so we have to use a dummy.  This is
sufficient for testing, however.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 12:51:18 +09:30
Rusty Russell
bf2a42ada5 channeld: defer first update_fee until we have an HTLC to send.
Sending update_fee immediately after channel establishment seems to
upset LND, so work around it by deferring it.  The reason we increase
the fee after establishment is because now we might need to close the
channel in a hurry due to htlcs, but until there are htlcs that's
unnecessary.

Fixes: #3596
Changelog-Changed: Added workaround for lnd rejecting our commitment_signed when we send an update_fee after channel confirmed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-13 20:58:06 -05:00
Christian Decker
197a144505 plugin: Ensure RPC passthrough calls are terminated when plugin dies
We now track all pending RPC passthrough calls, and terminate them with an
error if the plugin dies.

Changelog-Fixed: JSON-RPC: Pending RPC method calls are now terminated if the handling plugin exits prematurely.
2020-04-14 11:07:55 +09:30
darosior
af76bce0fa pyln-client: send proper JSONRPC2 errors on failed call
Changelog-Added: pyln now sends proper error on bad calls to plugin methods

Co-Authored-By: Sergi Delgado Segura <sergi.delgado.s@gmail.com>
2020-04-11 12:17:51 +09:30
Christian Decker
8370a627bb invoice: Parse amounts without a multiplier as BTC not msats
The spec states that invoices with an amount, but lacking a multiplier, should
be interpreted as integer Bitcoin amounts:

   `amount`: optional number in that currency, followed by an optional
   `multiplier` letter. The unit encoded here is the 'social' convention of a
   payment unit -- in the case of Bitcoin the unit is 'bitcoin' NOT satoshis.

Suggested-by: Stefano Pellegrini <@St333p>
Signed-off-by: Christian Decker <@cdecker>
Changelog-Fixed: invoice: The invoice parser assumed that an amount without a multiplier was denominated in msatoshi instead of bitcoins.
2020-04-11 08:43:06 +09:30
Christian Decker
b68066e8e8 python: Consolidate requirements.txt files in a single place
We had them split according the separate use-cases:

 - testing
 - doc-gen
 - wire-gen

But that was causing new contributors to miss some dependencies when they
first got hacking. So this consolidates all of our own dependencies in a root
requirements.txt, with the notable exception of `pyln-client`, `pyln-testing`
and `pyln-proto` which are distributed as PyPI modules and therefore have
their own dependencies that need to be tracked in the module root.

Closes #3518
2020-04-09 15:14:06 +02:00
Vasil Dimov
158d2212c2 closingd: configurable closing fee negotiation step
When negotiating the transaction fee for closing a channel [1], we used
to always pick the middle of the range between our proposal and the
peer's proposal.

Introduce a new option `fee_negotiation_step` to the close command, so
the peer who initiates the close can choose his back off step.

Partially resolves https://github.com/ElementsProject/lightning/issues/3270

[1] https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#closing-negotiation-closing_signed

Changelog-Added: New optional parameter to the `close` command to control the closing transaction fee negotiation back off step
2020-04-07 13:52:48 +09:30
darosior
5aafef1484 pytest: skip some test when DEPRECATED_APIS is enabled. 2020-04-05 11:06:27 +09:30
Rusty Russell
28e3ffc66b plugins/fundchannel: make 'all' do the right thing for wumbo.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-03 13:13:21 +10:30
Rusty Russell
07a281faf8 lightningd: add large-channels / wumbo option.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-added: `large-channels` option to negotiate opening larger channels.
2020-04-03 13:13:21 +10:30
Rusty Russell
1d90f21833 lightningd / openingd: remove limits if we negotiate option_support_large_channel.
Note that now we check capacity once we've figured out which peer, which
broke a test (we returned "unknown peer" instead of "capacity exceeded"),
so we rework that too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-03 13:13:21 +10:30
Rusty Russell
7a95e90ee4 plugins: add feature_set to init object.
Shows what features we use in various contexts, including those added
by plugins in getmanifest.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugin: `feature_set` object added to `init`
2020-04-03 13:13:21 +10:30
darosior
7c0af81c21 bcli: use a more urgent feerate for HTLCs and penalty transactions
A CONSERVATIVE/3 target for them.

Some noisy changes to the tests as we had to update the estimatesmartfee
mock.

Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
2020-04-01 23:02:47 -05:00
Rusty Russell
77e7beeb06 pytest: test the reply functionality (via blinded path) using a plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-02 14:32:38 +10:30
Rusty Russell
8b8cbb9397 channeld: handle encblob and blinding in messages.
This is based on https://github.com/lightningnetwork/lightning-rfc/blob/route-blinding/proposals/route-blinding.md

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-02 14:32:38 +10:30
Rusty Russell
e3dfba8944 pytest: test sendonionmessage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-02 14:32:38 +10:30
darosior
eead65350f wallet: allow to withdraw with unconfirmed utxos
Changelog-Fixed: Passing 0 as minconf to withdraw allows you to use unconfirmed transaction outputs, even if explicitly passed as the `utxos` parameter
2020-04-01 22:58:18 -05:00
Rusty Russell
afb76392e4 common/features: use bitmaps internally, have explicit init function.
This is to prepare for dynamic features, including making plugins first
class citizens at setting them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-03-31 13:36:02 +02:00
Rusty Russell
a430abf899 connectd: permit multiple descriptors of the same type.
This restriction was removed from the spec as of
86c2ebcc5973a4133d3ce4d80ae1c203061a1646.

We also fix up some strange formatting in that part of the documentation.

Changelog-changed: We now announce multiple addresses of the same type, if given.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-03-31 13:36:02 +02:00
Rusty Russell
5d4620484c lightningd: remove json_close force option.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON: `close` `force` parameter removed (deprecated in 0.7.2.1)
2020-03-30 12:47:01 +02:00
Rusty Russell
75838341a7 fundchannel/fundchannel_start: remove deprecated satoshi parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON: `fundchannel` and `fundchannel_start` `satoshi` parameter removed (renamed to `amount` in 0.7.3).
2020-03-30 12:47:01 +02:00
darosior
8a3295b6a8 chaintopology: Add the cost of HTLCs transactions to json_feerates
Changelog-Changed: "htlc_timeout_satoshis" and "htlc_success_satoshis" fields have been added to the `feerates` command.
2020-03-30 20:17:18 +10:30
darosior
5e72b22e80 bcli: adapt interface to the new fees estimation interface
We keep the same behaviour as lightningd before.
2020-03-30 20:17:18 +10:30
darosior
d4fe4073a4 lightning/bitcoind: adapt and batch fees estimations
This adapts our fee estimations requests to the Bitcoin backend to the
new semantic, and batch the requests.

This makes our request for fees much simpler, and leaves some more
flexibility for a plugin to do something smart (it could still lie before
but now it's explicit, at least.) as we don't explicitly request
estimation for a specific mode and a target.

Changelog-Changed: We now batch the requests for fee estimation to our Bitcoin backend.
Changelog-Changed: We now get more fine-grained fee estimation from our Bitcoin backend.
2020-03-30 20:17:18 +10:30
darosior
dce2e87928 chaintopology: better feerate targets differentiation
We kept track of an URGENT, a NORMAL, and a SLOW feerate. They were used
for opening (NORMAL), mutual (NORMAL), UNILATERAL (URGENT) transactions
as well as minimum and maximum estimations, and onchain resolution.

We now keep track of more fine-grained feerates:
- `opening` used for funding and also misc transactions
- `mutual_close` used for the mutual close transaction
- `unilateral_close` used for unilateral close (commitment transactions)
- `delayed_to_us` used for resolving our output from our unilateral close
- `htlc_resolution` used for resolving onchain HTLCs
- `penalty` used for resolving revoked transactions

We don't modify our requests to our Bitcoin backend, as the next commit
will batch them !

Changelog-deprecated: The "urgent", "slow", and "normal" field of the `feerates` command are now deprecated.
Changelog-added: The fields "opening", "mutual_close", "unilateral_close", "delayed_to_us", "htlc_resolution" and "penalty" have been added to the `feerates` command.
2020-03-30 20:17:18 +10:30
Michael Schmoock
0998164214 test: receivable_msat similar to spendable_msat 2020-03-27 12:33:23 +01:00
Michael Schmoock
80ff9c5b63 test: refactor lockup_drain add helpers drain and force_feerates 2020-03-27 12:33:23 +01:00
Christian Decker
e99ab03f79 pytest: Fix test_closing_fee regression in elements
Constants once again.
2020-03-24 09:52:33 +10:30
Christian Decker
9736c7bfd0 pytest: Make test_pay_retry less flaky for liquid-regtest 2020-03-24 09:52:33 +10:30
Christian Decker
5532305d76 pytest: Unbreak the test_feerate_spam test for elementsd
Looking for specific feerates, but not adjusting the amounts involved doesn't
work.
2020-03-24 09:52:33 +10:30
Christian Decker
7201cb7315 pytest: Configure logging in a fixture to match stdout capturing
pytest captures the output by monkey patching out `sys.stdout`. This may
conflict with our use of `sys.stdout` when configuring logging, resulting in
the "Write to closed file" issue that is spamming the logs. By making the
logging configuration a fixture hopefully we always use the correct
stdout (after pytest has monkey-patched it).
2020-03-24 09:52:33 +10:30
Christian Decker
3e3b05e1b2 pyln: Migrate remaining uses of the deprecated pylightning module
`pylightning` is not much more than an alias for `pyln-client`, so this
removes the need to install that as well just to run the tests.
2020-03-24 09:52:33 +10:30
Christian Decker
24aaf73982 pytest: Actually make sure that the direcory exists
Some tests may not spawn a node at all, so make sure that our assumption that
the directory exists in the fixture cleanup is correct by creating the
directory.
2020-03-24 09:52:33 +10:30
lisa neigut
563b46814b plugins: remove 'blank' option parsing for bool
bools must be set!
2020-03-21 16:29:52 +10:30
lisa neigut
42cce55b45 plugins: add 'flag' type for plugin options
Updates the plugin docs to include more detailed info about how options
work.

Changelog-Added: Plugins: 'flag'-type option now available.
2020-03-21 16:29:52 +10:30
lisa neigut
a08905c344 plugin tests: actually check log result + move over to '' -> true
a few things. one is that `is_in_log` returns a result rather than
enforcing a condition. so these lines all need asserts

two is that with the 'allow_deprecated_apis' option on, the python json
parser overwrites the now typed input with the later-added string
version, so the only option value present in the option key-value set is
the last, string one. the check for this has been updated to only verify
that the string version is included (i manually verified that both are
printed to the JSON message)
2020-03-21 16:29:52 +10:30
Michael Schmoock
af7e879308 fix: rfc #740 requires 100% feespike margin
Changelog-Fixed: Use lightning-rfc #740 feespike margin factor of 2
2020-03-19 18:56:12 +01:00
Rusty Russell
b162a0e5e2 tests: note the private keys of our test nodes.
I needed them to debug the onion messages API, so might as well record them
somewhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-03-17 18:47:52 +01:00
Christian Decker
4724d55e80 pytest: Add test for compressed onion
This one generates a compressed onion, decompresses it, and then proceeds with
normal processing.
2020-03-12 10:25:01 +10:30
Christian Decker
bc74e49534 pytest: Add tests for the sphinx onion generation and processing
These just run the test vectors and add a test for the devtools/onion tool so
we don't accidentally break them.
2020-03-12 10:25:01 +10:30
lisa neigut
41b18050a0 plugin: add in deprecated_api behavior and test
we also check that the node isn't running now, for extra pedancity
2020-03-10 13:25:36 +10:30
lisa neigut
b25a8ba29d plugins: test for option value checking and parsing 2020-03-10 13:25:36 +10:30
Christian Decker
d2688bbaf5 plugin: Unwrap the rpc_command payload
We were nesting like the following:

```json
{"params": {
  "rpc_command": {
    "rpc_command": {
    }
  }
}
```

This is really excessive, so we unwrap once, and now have the following:

```json
{"params": {
  "rpc_command": {
  }
}
```

Still more wrapping than necessary (the method is repeated in the `params`
object), but it's getting closer.

Changelog-Deprecated: JSON-RPC: Removed double wrapping of `rpc_command` payload in `rpc_command` JSON field.

Suggested-by: @fiatjaf
Signed-off-by: Christian Decker <@cdecker>
2020-03-10 11:42:58 +10:30
Vasil Dimov
66eca2ff3f lightningd: always broadcast the latest close tx
Before this patch we would only update `channel->last_tx` with the newly
proposed closure tx from the peer if the fee of the new one was lower.

In negotiations where we are at the higher end and the peer starts
lower, all peer's subsequent proposals will be higher than his initial
proposal and in this case we would never update `channel->last_tx`
and would wrongly broadcast his initial proposal at the end of the
negotiation.

Fixes https://github.com/ElementsProject/lightning/issues/3549

Changelog-Fixed: Always broadcast the latest close transaction at the end of the close fee negotiation, instead of sometimes broadcasting the peer's initial closing proposal.
2020-03-10 11:40:19 +10:30
Christian Decker
ac5cba2243 htlc: Return an invalid payload error if payload is invalid
We somehow were always returning `badonion` when really it is an invalid
payload inside the onion.
2020-03-04 22:57:24 +01:00
Christian Decker
9529529815 pytest: Repro wrong error being returned on invalid onion payloads
We are returning a `BADONION` error despite the cause being an invalid onion
payload containing an unknown even TLV type. It really should return
`INVALID_ONION_PAYLOAD` errors instead.
2020-03-04 22:57:24 +01:00
Rusty Russell
3572d598bf tests: fix flake in libplugin test.
My test machine is fast enough that we might not have seen the plugin
msg yet.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-03-04 16:42:34 +01:00
ZmnSCPxj jxPCSnmZ
d9b2482415 lightningd/hsm_control.c: Implement getsharedsecret.
ChangeLog-Added: New `getsharedsecret` command, which lets you compute a shared secret with this node knowing only a public point. This implements the BOLT standard of hashing the ECDH point, and is incompatible with ECIES.
2020-02-28 14:45:50 +10:30
Rusty Russell
40e3566e9a lightningd: use the async mechanism for channel_update access.
Instead of saving a stripped_update, we use the new
local_fail_in_htlc_needs_update.

One minor change: we return the more correct
towire_temporary_channel_failure when the node is still syncing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-28 09:44:47 +10:30
Christian Decker
23149c3daa plugin: Actually wait the 20 seconds promised in the docs
We promised we'd be waiting up to 20 seconds, but were only waiting for
10. Fix that by bumping to the documented 20.
2020-02-27 09:21:44 +10:30
Christian Decker
4a21883553 plugin: Fix hanging hook calls if the plugin dies
Changelog-Fixed: plugin: A crashing plugin will no longer cause a hook call to be delayed indefinitely
2020-02-27 09:21:44 +10:30
Christian Decker
72757933f0 pytest: Test a plugin crash while handling a hook call 2020-02-27 09:21:44 +10:30
Rusty Russell
faac4b28ad plugins: support failure_message in invoice and htlc_accepted hooks.
As promised in the Changelog when we converted from failcodes to messages
internally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-25 11:12:12 +10:30
darosior
cd15cec2f3 txprepare: don't crash if we are passed unconfirmed utxos
Changelog-added: `txprepare` doesn't crash lightningd anymore if you pass unconfirmed utxos
2020-02-21 17:52:02 +01:00
darosior
9e9e23c81a pytest: test txprepare with unconfirmed utxos 2020-02-21 17:52:02 +01:00
darosior
a47fd8cf3e pytest: test Bitcoin plugin registration and the bcli plugin 2020-02-12 11:45:07 +10:30
darosior
4c5862eab1 pytest: fixup flaky test_closing_specified_destination 2020-02-12 11:45:07 +10:30
darosior
bbc5c10919 pytest: adjust mocks and logs for bitcoind
For bitcoind_fail_first:
We only ever send `getblock` if we got a successful block hash from
`getblockhash`, and if we can't get the block in that case it means
our Bitcoin backend is faulty and we shouldnt continue.

So, mock `getblockhash` instead, which is authorized to spuriously fail.

For both bitcoind_fail_first and bitcoind_failure:
Adapt the logs.
2020-02-12 11:45:07 +10:30
darosior
e6a6ef23fd pytest: make the libplugin test non flaky
Rusty fixed the underlying bug in 8b8e4bb5fd
2020-02-12 09:30:02 +10:30
Rusty Russell
86c28b2272 channeld: channel drain mitigation.
Add new check if we're funder trying to add HTLC, keeping us
with enough extra funds to pay for another HTLC the peer might add.

We also need to adjust the spendable_msat calculation, and update
various tests which try to unbalance channels.  We eliminate
the now-redundant test_channel_drainage entirely.

Changelog-Fixed: Corner case where channel could become unusable (https://github.com/lightningnetwork/lightning-rfc/issues/728)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 15:34:05 +01:00
Rusty Russell
6efde31ade pytest: simplified drain test (xfail)
This is inspired by @m-schmook's https://github.com/ElementsProject/lightning/pull/3498
except this is simply a two-channel version which probes for the amount.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 15:34:05 +01:00
Christian Decker
490550d508 plugin: Allow multiple plugins to register the htlc_accepted hook
Make the `htlc_accepted` hook the first chained hook in our repertoire. The
plugins are called one after the other in order until we have no more plugins
or the HTLC was handled by one of the plugins. If no plugins handles the HTLC
we continue to handle it internally like always.

Handling in this case means the plugin returns either `{"result": "resolve",
...}` or `{"result": "fail", ...}`.

Changelog-Changed: plugin: Multiple plugins can now register for the htlc_accepted hook.
2020-02-11 15:57:22 +10:30
Rusty Russell
30580731a6 Minor fixups on PR #3477
Feedback from @niftynei and me; nothing major, but avoids
another round-trip.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-02-11 13:53:31 +10:30
Christian Decker
4737977128 plugin: Allow custom features only if the plugin is not dynamic
This is in order to avoid having to update featurebits as plugins get
activated and deactivated.
2020-02-11 13:53:31 +10:30
Christian Decker
ebd3a8d7f2 plugin: Add featurebits registered by plugins to node_announcements
This is the last venue we need to add custom featurebits to, so we also unmark
the test as xfail.

Changelog-Added: plugin: Plugins can now signal support for experimental protocol extensions by registering featurebits for `node_announcement`s, the connection handshake, and for invoices. For now this is limited to non-dynamic plugins only
2020-02-11 13:53:31 +10:30
Christian Decker
966ac95098 pytest: Add a test for plugin featurebits 2020-02-11 13:53:31 +10:30
lisa neigut
27c7707874 Add note about rationale for check 2020-02-10 15:59:26 +10:30
lisa neigut
ec1b86d9d5 channel: make error 'hard' if awaiting lockin
if the channel hasn't been locked in yet, allow for a 'hard' error
to kill the channel
2020-02-10 15:59:26 +10:30
lisa neigut
1e3a411b74 channel control: check that peer is connected before canceling
Use the new forget_channel method to cancel, which checks that
peer is still connected before attempting to send message.
2020-02-10 15:59:26 +10:30
lisa neigut
278b69dfbe channel control: don't allow peer to cancel channel
restrict fundchannel_cancel usage to only the opener side

Changelog-Changed: Only the opener of a fundchannel can cancel the channel open with fundchannel_cancel
2020-02-10 15:59:26 +10:30
Michael Schmoock
e92ab89087 test: uses wait_for_htlcs helper in test_pay 2020-02-10 12:14:19 +10:30
darosior
972b4def57 lightningd/plugin: unregister a plugin's options when stopping it
This also remove the now duplicate plugin_hook_unregister_all(), added
in the tal destructor of the struct plugin.
2020-02-10 09:49:15 +10:30
darosior
84e3fa39b3 pytest: test libplugin's send_outreq 2020-02-10 09:49:15 +10:30
darosior
75becf7227 pytest: test hooks and notifications with libplugin 2020-02-10 09:49:15 +10:30
darosior
ceeb5503cc libplugin: fix 'dynamic' field in getmanifest
As a separated commit because it was pre-existent (changelog + xfail test).

This also fix a logical problem in lightningd/plugin_control: we were
assuming a plugin started with 'plugin start' but which did not comport
a 'dynamic' entry in its manifest to be dynamic, though it should have
been treated as static.

Changelog-fixed: plugins: Dynamic C plugins can now be managed when lightningd is up
2020-02-10 09:49:15 +10:30
darosior
da476848d1 pytest: add a C testing plugin to test libplugin
We mark the test as xfail() as it exposes that libplugin's PLUGIN_RESTARTABLE
was not taken into account !
2020-02-10 09:49:15 +10:30
darosior
9030e5d779 pytest plugins: use the standard 'no operation' syntax for hooks 2020-02-04 01:07:59 +00:00
darosior
d2638c99af pytest: test withdrawal transactions' nLockTime 2020-02-03 00:45:27 +00:00
Rusty Russell
f3600d22a0 lightningd: disallow msatoshi arg to sendpay unless exact when non-MPP.
Using it with a different value to the amount sent causes a crash in 0.8.0,
which is effectively deprecating it, so let's disallow it now.

Changelog-Changed: If the optional `msatoshi` param to sendpay for non-MPP is set, it must be the exact amount sent to the final recipient.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-01-31 14:23:18 +01:00
Rusty Russell
295ca2a436 lightningd: remove things we deprecated 6 months ago.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Relative plugin paths are not relative to startup (deprecated v0.7.2.1)
Changelog-Removed: Dummy fields in listforwards (deprecated v0.7.2.1)
2020-01-31 03:48:01 +00:00
Rusty Russell
bfcef9002e lightningd: fix handling of WIRE_UPDATE_FAIL_MALFORMED_HTLC.
1. We asserted that there wouldn't be a raw failcode.
2. We didn't pass the failure information via JSON in this case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-01-29 21:15:25 +01:00
Rusty Russell
c11212bb52 pytest: test that we handle WIRE_UPDATE_FAIL_MALFORMED_HTLC correctly.
We could use sendonion to do this, but it actually takes a different path through
pay, and I wanted to test all of it, so I made a new dev flag.

We currently get upset with the response:

	lightningd/pay.c:556: payment_failed: Assertion `!hout->failcode' failed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-01-29 21:15:25 +01:00
Christian Decker
b0529843ac pytest: Add a plugin for custommsgs and check that they get the msgs
This completes the custommsg epic, finally we are back where we began all that
time ago (about 4 hours really...): in a plugin that implements some custom
logic.
2020-01-28 23:50:52 +01:00
Christian Decker
b18c1ea543 json-rpc: Restrict custommsgs to be odd-typed
This solves a couple of issues with the need to synchronously drop the
connection in case we were required to understand what the peer was talking
about while still allowing users to experiment, just not kill connections.
2020-01-28 23:50:52 +01:00
Christian Decker
e9fc9aef34 channeld: Send messages if instructed to by lightningd 2020-01-28 23:50:52 +01:00
Christian Decker
3c88d5c8c4 openingd: Implement sendcustommsg handling in openingd
Most of the work is done in `lightningd`, here we just need to queue the
message itself.
2020-01-28 23:50:52 +01:00
ZmnSCPxj jxPCSnmZ
ccb895083f tests/test_invoices.py: Add test for timeout param of waitanyinvoice. 2020-01-28 14:07:52 +01:00
lisa neigut
d36af2c340 txprepare: make output finding map for withdraw_tx variable len
The number of outputs got updated, but the map used to calculate the
change output's location did not (still assumes only one output). This
patch fixes this to make the output map a variable size.

Changelog-Fixed: JSON API: `txprepare` no longer crashes when more than two outputs are specified
2020-01-27 22:59:41 +01:00
lisa neigut
f15d34465d txprepare: add test for multiple withdraw-to outputs
adds (currently failing test) for a txprepare with more than two outputs
2020-01-27 22:59:41 +01:00