Commit Graph

9051 Commits

Author SHA1 Message Date
Rusty Russell
fa006fd2f7 pyln-testing: require bitcoin v0.20 for PSBT handling, and create wallet for v0.21
With older bitcoind, PSBTs fail:

```
    def test_utxopsbt(node_factory, bitcoind, chainparams):
...
>       psbt = bitcoind.rpc.decodepsbt(funding['psbt'])

tests/test_wallet.py:561:
...
self = <bitcoin.rpc.RawProxy object at 0x7f4ec602e100>, service_name = 'decodepsbt'
args = ('cHNidP8BADMCAAAAAaoMihSVXlpdBHGcJePiroqtwq/b1zu09j8IkTG4OKs7AQAAAAD9////AGYAAAAAAQDeAgAAAAABAefqB6BkZE1/AqXaf36T02a7.../7Stf971PEgvUXgvASECXPTIO6tIVxDih6tfKy6suj6WJhhjycwoaTeuso/AQ8llAAAAAQEfQEIPAAAAAAAWABQB+tkKvNZml+JZIWRyLeSpXr7hZQA=',)
postdata = '{"version": "1.1", "method": "decodepsbt", "params": ["cHNidP8BADMCAAAAAaoMihSVXlpdBHGcJePiroqtwq/b1zu09j8IkTG4OKs7AQ...gvUXgvASECXPTIO6tIVxDih6tfKy6suj6WJhhjycwoaTeuso/AQ8llAAAAAQEfQEIPAAAAAAAWABQB+tkKvNZml+JZIWRyLeSpXr7hZQA="], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -22, 'message': 'TX decode failed PSBT is not sane.: iostream error'}, 'id': 1, 'result': None}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: *** Requires bitcoind v0.20.1 or above ***
2020-11-09 20:05:30 -06:00
positiveblue
fa1483a00d hsm_secret generation from a seed-phrase
tools: Add `generatehsm` method to hsmtool to derivate BIP32 seeds from a
mnemonic using the BIP39 standard.

The new method uses libwally for the BIP39 to BIP32 derivation. It also
fails if an hsm_secret file already exists, so we do not overwrite
someone else's wallet without noticing.

It allows the use of passphrases, the ECHO mode in the terminal is
disable for higher security.

It currently supports "en", "es", "fr", "it", "jp", "zhs", "zht".

Changelog-Added: hsmtool: `hsm_secret` generation from a seed-phrase following BIP39.
2020-11-09 17:49:24 -06:00
Rusty Russell
9b0af9f046 gossipd: minor cleanups.
Thanks to m-schmook's feedback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 17:37:10 -06:00
Rusty Russell
9a575a98a0 gossipd: simplify large reply code.
We used to create the entire reply, the if it was too big, split in
half and retry.

Now that the main network is larger, this always happens with a full
request, which is inefficient.

Instead, produce a reply assuming no compression, then compress as a
bonus.  This is simpler and more efficient, at cost of sending more
packets.

I also renamed an internal dev var to make it clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 17:37:10 -06:00
Rusty Russell
8db5fb7345 gossipd: new struct to hold scids and timestamps together.
It's not (yet?) compulsory to have the timestamps, but handing them around
together makes sense (a missing timestamp has the same effect as a zero
timestamp).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 17:37:10 -06:00
Rusty Russell
59f23bf23c gossipd: use straight counter instead of bitmap for query_channel_range replies.
The spec (since d4bafcb67dcf1e4de4d16224ea4de6b543ae73bf in March
2020) requires that reply_channel_range be in order (and all
implementations did this anyway).

But when I tried this, I found that LND doesn't (always) obey this,
since don't divide on block boundaries.  So we have to loosen the
constraints here a little.

We got rid of the old LND compat handling though, since everyone should
now be upgraded (there are CVEs out for older LNDs).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Support for receiving full gossip from ancient LND nodes.
2020-11-09 17:37:10 -06:00
Rusty Russell
4745e7e3d0 pytest: make test_hook_dep tests more stable.
I think this is what Travis is having an issue with, but it work
fine locally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Michael Schmoock
f3804daabb pytest: simplify test_openchannel_hook_chaining now order is deterministic. 2020-11-09 15:22:33 -06:00
Rusty Russell
fb295ffb51 plugin: sort topological candidates by specified order.
We previously registered hooks up in who-replies-to-getmanifest-first
order, but then if any had dependencies it would scatter that order.

This allows users to manually set dependencies developers have
forgotten by specifying the plugins manually in their configuration or
cmdline.  This was an excellent consideration by @mschmook.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
d429e21db3 pytest: test that we maintain load order unless hook deps require a change.
Suggested-by: @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
852e14c947 plugins: check order once all plugins have returned from getmanifest.
This means we need to stop at this stage even in the runtime-loaded
case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
a4f79e8e7e pytest: test that we don't complain about missing dependencies at startup.
We fail this, because we check dependencies as they come in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
ac93b780d5 plugins: deprecate old form of hooks.
Now both python and c libraries are updated, we can officially
deprecate the old form.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: plugins: hooks should now be specified using objects, not raw names.
2020-11-09 15:22:33 -06:00
Rusty Russell
62c52fe868 libplugin: add support for before and after deps on hooks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
e16ed0e207 pyln: add support for dependent hooks.
And use that to add simple tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
6a55b4367e lightningd: actually order the hooks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
Rusty Russell
e2a31f42f2 plugins: allow 'before' and 'after' arrays for hooks.
The next patch will use these to order the hooks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: plugins: hooks can now specify that they must be called 'before' or 'after' other plugins.
2020-11-09 15:22:33 -06:00
Rusty Russell
7c3c0b1013 common: allow tal_arr_expand() to have an n argument.
It needs to use a non-clashing name for the internal var.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00
ZmnSCPxj jxPCSnmZ
2604a81aba lightningd/opening_control.c: Remove predeclaration. 2020-11-07 17:18:49 +00:00
ZmnSCPxj jxPCSnmZ
0fd87b85da openingd/: Fail fundchannel_start if we already are, or will become, the fundee.
Fixes: #4108

Changelog-Fixed: Network: Fixed a race condition when us and a peer attempt to make channels to each other at nearly the same time.
2020-11-07 17:18:49 +00:00
ZmnSCPxj jxPCSnmZ
712e0796fc tests/test_connection.py: Test for #4108. 2020-11-07 17:18:49 +00:00
Michael Schmoock
86137852d2 doc: update channel_state_changed notification 2020-11-06 14:47:04 -06:00
Michael Schmoock
4a8722bf1d pytest: state_change history 2020-11-06 14:47:04 -06:00
Michael Schmoock
6cc96c07dc feat: adds state_changes to listpeers output
Changelog-Added: RCP: Added 'state_changes' history to listpeers channels
2020-11-06 14:47:04 -06:00
Michael Schmoock
88c1dc56e8 db: adds a state_change history to database 2020-11-06 14:47:04 -06:00
Michael Schmoock
68ce25c92d feat: adds timestamp to state_change notification 2020-11-06 14:47:04 -06:00
Michael Schmoock
4d765003dd rpc: adds json_add_timeiso helper 2020-11-06 14:47:04 -06:00
Michael Schmoock
67f4970eb1 pytest: test opener and closer of listpeers 2020-11-06 14:47:04 -06:00
Michael Schmoock
b7c18517df rpc: adds opener and closer to listpeers channels
Changelog-Added: RPC: Added 'opener' and 'closer' to listpeers channels
2020-11-06 14:47:04 -06:00
Michael Schmoock
083a856c31 db: persist channel closer and state change cause 2020-11-06 14:47:04 -06:00
Michael Schmoock
c52efe0f37 pytest: channel state change cause and message 2020-11-06 14:47:04 -06:00
Michael Schmoock
8a8dabaa58 feat: adds state change cause and message
This adds a `state_change` 'cause' to a channel.
A 'cause' is some initial 'reason' a channel was created or closed by:

  /* Anything other than the reasons below. Should not happen. */
  REASON_UNKNOWN,
  /* Unconscious internal reasons, e.g. dev fail of a channel. */
  REASON_LOCAL,
  /* The operator or a plugin opened or closed a channel by intention. */
  REASON_USER,
  /* The remote closed or funded a channel with us by intention. */
  REASON_REMOTE,
  /* E.g. We need to close a channel because of bad signatures and such. */
  REASON_PROTOCOL,
  /* A channel was closed onchain, while we were offline. */
  /* Note: This is very likely a conscious remote decision. */
  REASON_ONCHAIN

If a 'cause' is known and a subsequent state change is made with
`REASON_UNKNOWN` the preceding cause will be used as reason, since a lot
(all `REASON_UNKNOWN`) state changes are a subsequent consequences of a prior
cause: local, user, remote, protocol or onchain.

Changelog-Added: Plugins: Channel closure resaon/cause to channel_state_changed notification
2020-11-06 14:47:04 -06:00
Rusty Russell
d5d9858b7b lightningd: fix similar race in stop.
Tested by putting a sleep in the rpc_command hook.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-06 14:05:44 -06:00
Rusty Russell
5e6b0f9445 lightningd: fix crash if we abort after enabling notifications.
The rpc_command hook means that we have a delay between receiving
a JSON command and actually calling the handler.  In this case, the
caller can go away:

```
==1348== Invalid write of size 1
==1348==    at 0x130EA6: json_notifications (jsonrpc.c:1350)
==1348==    by 0x12EE9E: command_exec (jsonrpc.c:636)
==1348==    by 0x12F3C6: rpc_command_hook_callback (jsonrpc.c:752)
==1348==    by 0x15AA08: plugin_hook_callback (plugin_hook.c:210)
==1348==    by 0x155C9D: plugin_response_handle (plugin.c:398)
==1348==    by 0x155E84: plugin_read_json_one (plugin.c:504)
==1348==    by 0x15603D: plugin_read_json (plugin.c:548)
==1348==    by 0x1D4AB3: next_plan (io.c:59)
==1348==    by 0x1D5630: do_plan (io.c:407)
==1348==    by 0x1D566E: io_ready (io.c:417)
==1348==    by 0x1D7834: io_loop (poll.c:445)
==1348==    by 0x12CFAC: io_loop_with_timers (io_loop_with_timers.c:24)
==1348==  Address 0x58 is not stack'd, malloc'd or (recently) free'd
==1348==
lightningd: FATAL SIGNAL 11 (version v0.9.1-266-ga4df315)
0x180f7e send_backtrace
	common/daemon.c:38
0x181024 crashdump
	common/daemon.c:51
0x5bd7fcf ???
	???:0
0x130ea6 json_notifications
	lightningd/jsonrpc.c:1350
0x12ee9e command_exec
	lightningd/jsonrpc.c:636
0x12f3c6 rpc_command_hook_callback
	lightningd/jsonrpc.c:752
0x15aa08 plugin_hook_callback
	lightningd/plugin_hook.c:210
0x155c9d plugin_response_handle
	lightningd/plugin.c:398
0x155e84 plugin_read_json_one
	lightningd/plugin.c:504
0x15603d plugin_read_json
	lightningd/plugin.c:548
0x1d4ab3 next_plan
	ccan/ccan/io/io.c:59
0x1d5630 do_plan
	ccan/ccan/io/io.c:407
0x1d566e io_ready
	ccan/ccan/io/io.c:417
0x1d7834 io_loop
	ccan/ccan/io/poll.c:445
0x12cfac io_loop_with_timers
	lightningd/io_loop_with_timers.c:24
0x132825 main
	lightningd/lightningd.c:1016
0x5bbab96 ???
	???:0
0x1159e9 ???
	???:0
0xffffffffffffffff ???
	???:0
Log dumped in crash.log.20201106001723
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-06 14:05:44 -06:00
fiatjaf
15b2546d45 doc: move "next_onion" and "shared_secret" to inside "onion". 2020-11-05 14:15:18 -06:00
Antoine Poinsot
8846c35ac6 dual_open: correct silent enum conversions
We were silently converting a side enum (3 variants) to a tx_role enum
(2 variants).

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-04 14:29:22 -06:00
niftynei
e99335e5b3 df: move tx_role functions (used in next patch) 2020-11-04 14:29:22 -06:00
Rusty Russell
cce3e717d9 bitcoin/signature: wrap libwally call.
If a tx is larger than 2k, libwally will do an alloc:

```
lightning_hsmd: common/setup.c:11: wally_tal: Assertion `wally_tal_ctx' failed.
0x11c283 wally_tal
	common/setup.c:11
0x15ebd1 wally_malloc
	../../../libwally-core/src/internal.c:233
0x171e9e tx_to_bip143_bytes
	../../../libwally-core/src/transaction.c:1918
0x172cda tx_to_bytes
	../../../libwally-core/src/transaction.c:2086
0x1759df tx_get_signature_hash
	../../../libwally-core/src/transaction.c:2776
0x175afd wally_tx_get_signature_hash
	../../../libwally-core/src/transaction.c:2800
0x175b62 wally_tx_get_btc_signature_hash
	../../../libwally-core/src/transaction.c:2810
0x1297d9 bitcoin_tx_hash_for_sig
	bitcoin/signature.c:139
0x1298ca sign_tx_input
	bitcoin/signature.c:161
0x10e701 handle_sign_remote_commitment_tx
	hsmd/hsmd.c:1011
0x110f7f handle_client
	hsmd/hsmd.c:1968
0x147a71 next_plan
	ccan/ccan/io/io.c:59
0x1485ee do_plan
	ccan/ccan/io/io.c:407
0x14862c io_ready
	ccan/ccan/io/io.c:417
0x14a7f2 io_loop
	ccan/ccan/io/poll.c:445
0x111125 main
	hsmd/hsmd.c:2040
```

I reduced that constant in libwally to 200, and ran the entire
test suite, and found no other places.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-04 14:28:06 -06:00
Michael Schmoock
bde2806290 chore: fixes a cppcheck false positive
Just applied the same suppression as rusty in:
6635fe12e4 (Rusty Russell    2020-05-15 15:57:29 +0930 146)
/* cppcheck-suppress uninitvar - false positive on f1->bits */

My cppcheck was complaining about the same issue in the following functions.
I wonder why travis does not care though.

Changelog-None
2020-11-03 10:18:08 -06:00
YOSHIDA Masanori
9f893e3f6d wire: remove declaration of towire_tlvs since its implementation is already removed
Signed-off-by: YOSHIDA Masanori <masanori.yoshida@gmail.com>
Changelog-None
2020-11-02 10:59:05 -06:00
Christian Decker
d151d554b3 travis: Add a pytest run with EXPERIMENTAL_FEATURES=1
Signed-off-by: Lisa <@niftynei>
2020-10-29 13:42:32 -05:00
Christian Decker
62f7e646ec pyln: Bump cryptography dependency to 3.2 due to upstream bug
https://cryptography.io/en/latest/changelog.html

Changelog-None
2020-10-29 11:11:48 +10:30
Christian Decker
d487ec5c9b wallet: Remove orphan statements.po file 2020-10-29 11:11:48 +10:30
Christian Decker
9a0327cd25 pyln-proto: Add compactsize alias for varint_{encode,decode}
We were mistakenly calling it varint, while Bitcoin refers to it as
CompactSize.
2020-10-29 11:11:48 +10:30
Jan Sarenik
895722fd8a nit: lightningd.c: fix "Richard Stevens' advice"
See https://github.com/rustyrussell/ccan/pull/88

Changelog-None
2020-10-28 14:32:35 -05:00
Jan Sarenik
90ad541cd2 doc: STYLE.md fix "parameters get"
Changelog-None
2020-10-28 14:32:20 -05:00
Rusty Russell
ae4139ba10 lightningd: note whether payment secret is used or not.
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets.  We need to know this so we know when
we can make them compulsory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-28 12:03:14 +01:00
Rusty Russell
cd5a93d0bd gossmap: fix reutrn of gossmap_xxx_has_feature, rename.
1. One place returned false instead of -1.
2. The names implied it returned a bool, and it doesn't.

Fix both, and curse C's loose typing a little.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-28 15:32:12 +10:30
niftynei
1e173463c9 df: rename 'feerate_per_kw' to 'feerate_per_kw_commitment'
Differentiate btw feerate for commitment txs and feerate for funding
transaction.
2020-10-27 19:52:05 -05:00
niftynei
4ea9d9e928 df: Pass new feerate options through to plugin, set reasonable bounds
We let the plugin decide what feerate to accept/whether or not to add
funds to the open. To aid this decision, we also send the plugin what we
(c-lightning) currently have as our max and min acceptable feerates.

We also now use these as our default for max/min acceptable feerate
range when sending an openchannel offer to a peer.

In the future, it might be a good idea to make these more easily
changeable, either via a config setting (?) or a command param.
2020-10-27 19:52:05 -05:00