Commit Graph

5993 Commits

Author SHA1 Message Date
Rusty Russell
fb6870c139 param: implement helpers for multiplex commands.
Our previous param support was a bit limited in this case.

We create a dev- command multiplexer, so we can exercise it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-16 21:39:14 +00:00
lisa neigut
eaa19f23fa bolt: update to latest RFC lightning-rfc:ffeece
Update to latest rfc checkpoint; no material source changes
2019-07-16 06:10:58 +00:00
lisa neigut
81c40da3b5 bolt: update to lightning-rfc:636b9f, change opcode reference format
Updates BOLT reference to match spec; replaces OP_CSV ->
OP_CHECKSEQUENCEVERIFY
2019-07-16 06:10:58 +00:00
lisa neigut
85e325cb1f bolt: update to lightning-rfc:6f6ea63233c new fundamental types
RFC tweaks to the types used to specify message fields
2019-07-16 06:10:58 +00:00
lisa neigut
5c07afac7d bolt: update to BOLT spec changes (extract format + type specifications)
updates the bolt version to 6639cef095a2ecc7b8f0c48c6e7f2f906fbfbc58.

this requires us to use the new bolt parser at generate-bolt.py
and updates to all of the type specifications (ie. from u8 -> byte)
2019-07-16 06:10:58 +00:00
lisa neigut
6c240ab589 bolts: new parsing script and templates for new bolt format
the RFC's extract-format.py is switching to a new format.
this script can correctly parse them.

mostly moves logic over from generate-wire.py, uses a
Python formatting libarary called mako, which needs to be
installed prior to running this script.

you can add it to your system with

    sudo apt-get install python3-mako
2019-07-16 06:10:58 +00:00
lisa neigut
01e292a3ff Travis: use inline mako install
Needed for next patch.
2019-07-16 06:10:58 +00:00
lisa neigut
118caae45b tools: simplify check
in this case, we always need a pointer, so remove the check
and just add it to the template
2019-07-16 06:10:58 +00:00
lisa neigut
254ee8a5d3 bolt-spec: bring up to date at 309e86d471faf90c1f6c531701c16887684badb9 2019-07-16 06:10:58 +00:00
Juraj Bednar
ab52638f93 Fix typos and options
--proxy= instead of --proxy:
--always-use-proxy needs value (true)
2019-07-15 01:00:57 +00:00
Rusty Russell
068b593d98 plugins/pay: fix crash when we reach retry timeout.
pay: ccan/ccan/json_out/json_out.c:144: check_fieldname: Assertion `!fieldname' failed.
pay: FATAL SIGNAL 6 (version v0.7.1rc4-1-gf1bea55)
0x11d914 send_backtrace
        common/daemon.c:40
0x11d9ba crashdump
        common/daemon.c:53
0x5430f1f ???
        ???:0
0x5430e97 ???
        ???:0
0x5432800 ???
        ???:0
0x5422399 ???
        ???:0
0x5422411 ???
        ???:0
0x1325c3 check_fieldname
        ccan/ccan/json_out/json_out.c:144
0x132627 json_out_member_direct
        ccan/ccan/json_out/json_out.c:162
0x10c4a4 json_out_add_raw_len
        plugins/pay.c:122
0x10c4f9 json_out_add_raw
        plugins/pay.c:130
0x10c9ae waitsendpay_expired
        plugins/pay.c:236
0x10ce39 waitsendpay_error
        plugins/pay.c:322
0x111aa7 handle_rpc_reply
        plugins/libplugin.c:431

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-14 16:16:30 +02:00
Christian Decker
53488e5739 pytest: Strengthen the htlc_accepted tests
We were having a few issues with malformed data in the past, so this time we
really check that stuff.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-07-13 14:44:12 +02:00
lisa neigut
a6516f477b bash-completion: include underscores
`fundchannel_start` etc are not included in the bash-completion as
they include underscores in their names. this fixes this by including
underscores.
2019-07-09 13:23:14 +02:00
lisa neigut
4638995037 tests: log funder_faileds as unusual not broken
test_funding_cancel_race explicitly attempts to trigger this via a race
condition; this conflicts with our post-test checks that no broken
logs were logged. as a middle ground, we log it as unusual, not broken,
as it's possible for it to attempt to fail if it was begun at the same
time as the complete is.
2019-07-09 04:18:48 +00:00
Rusty Russell
789d14299b pytest: mix up order of cancel vs complete in test_funding_cancel_race
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-08 15:14:31 +00:00
Rusty Russell
ee8edfe6a6 pytest: severely reduce scope of test_funding_cancel_race under valgrind.
Otherwise we get timeouts across the board.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-08 15:14:31 +00:00
Rusty Russell
9fdcb1a122 openingd: remove unnecessary is_err flag.
It probably doesn't matter to "fundchannel_cancel" exactly why the
fundchannel didn't work (though it can read the error msg), and we
should always fail any pending fundchannel_complete command.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-08 15:14:31 +00:00
Rusty Russell
280bd60988 lightningd: allow multiple cancels on a single fundchannel command.
Instead of taking over the ->cmd pointer, append ourselves to a list
of cancels.  This fixes the test_funding_cancel_race.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-08 15:14:31 +00:00
Rusty Russell
4b1a300ee3 lightningd: fix fundchannel_complete race.
If we already have a fundchannel_complete, don't attach another one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-08 15:14:31 +00:00
Rusty Russell
13dbe6e74b pytest: stress test funding_cancel vs funding_complete
This fails, because they fight over the fc->cmd pointer, leaving
others hanging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-08 15:14:31 +00:00
Rusty Russell
2945b25b57 onchaind: don't free proposal, correctly check it's not resolved in billboard.
Instead of freeing proposals, which we did in *some* places, we just
set ->resolved and check that in billboard_update which didn't get it right.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-07 00:12:34 +00:00
Rusty Russell
e746e9c202 onchaind: fix resolution detection of complex outputs.
We shouldn't set out->resolved until we've actually resolved it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-07 00:12:34 +00:00
trueptolemy
6deed77d88 bitcoind: confirm chain with bitcoin-cli at the beginning
At the beginning of the lightningd, we use "echo" command to check if bitcoin-cli is running.
Now we raplace "echo" with "getblockchaininfo" for this check, and also check whether the "chain" field in response is same as the blockchain that  lightningd is on.
"getblockchaininfo" is also valid for litecoin-cli.
2019-07-04 16:13:09 +02:00
trueptolemy
285da33e9e Chainparam: Add 'bip70_name' field for blockchain
'bip70_name' is corresponding to the 'chain' field of
the API 'getblockchaininfo'.
At the beginning of lightningd, we use the 'chain' field of 'getblockchaininfo' to check if we are on right blockchain.
2019-07-04 16:13:09 +02:00
trueptolemy
a594196c7c bitcoin-cli: rename bcli_args() to bcli_args_direct(), and also warp it in bcli_args()
1. bcli_args_direct() will be used in wait_for_bitcoind;
At the beginning, we check if bitcoin-cli is running by "echo" command
whitout any bitcoin_cli struction. If this first command fails, we need
present the agrs gathered, like "-rpcuser", like "-rpcpassword".
Related changes include:
  i) rename bcli_args() to bcli_args_direct(), and use 'const char **'
      as the paramater for bcli_args_direct();
  ii) add a new function bcli_args() warpped on bcli_args_direct(), this
      warpping can reduce the large number of changes later in the file;
2. bcli_args() warpping on bcli_args_direct() is used like original.
2019-07-04 16:13:09 +02:00
Rusty Russell
c95b4eedf4 gossipd: fail clearly if we can't open/create gossip_store.
Otherwise we fail at the write, and then it's not clear
*why* we couldn't open file:

lightning_gossipd: Writing version to store: Bad file descriptor (version v0.7.1-16-g7ea5c5c)
0x560dcf1a3779 send_backtrace common/daemon.c:40
0x560dcf1a634d status_failed common/status.c:192
0x560dcf19726a gossip_store_new gossipd/gossip_store.c:195
0x560dcf199fd0 new_routing_state gossipd/routing.c:177
0x560dcf1a098b gossip_init gossipd/gossipd.c:2113
0x560dcf1a197a recv_req gossipd/gossipd.c:2946
0x560dcf1a38cd handle_read common/daemon_conn.c:31
0x560dcf1bae2c next_plan ccan/ccan/io/io.c:59
0x560dcf1bb314 do_plan ccan/ccan/io/io.c:407
0x560dcf1bb341 io_ready ccan/ccan/io/io.c:417
0x560dcf1bcb13 io_loop ccan/ccan/io/poll.c:445
0x560dcf1a1ba0 main gossipd/gossipd.c:3073

Reported-by: @JavierRSobrino
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-04 16:10:20 +02:00
darosior
7ea5c5c00f Pylightning: allow to set description and long description manually
This just adds the possibility to set a command's descriptionS by hand, so that the method docstring can be used for something else
2019-07-02 19:30:50 +02:00
Rusty Russell
54ce4ed1cf pytest: fail tests if we get any LOG_BROKEN level messages, unless flagged.
And clean up some dev ones which actually happen (mainly by calling
channel_fail_permanent which logs UNUSUAL, rather than
channel_internal_error which logs BROKEN).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-02 03:26:10 +00:00
Rusty Russell
5f02294b5b lightningd/log: prefix log messages with level.
In particular, this lets us spot UNUSUAL and BROKEN messages easily.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-07-02 03:26:10 +00:00
Nicola Squartini
cbf240e110 makefiles: fix quotation of BINTOPKGLIBEXECDIR macro 2019-07-01 00:27:11 +00:00
ZmnSCPxj
011a4bcd83 CHANGELOG.md: Reset for next release.
[ Added [Unreleased] URL -- RR ]
2019-07-01 00:21:18 +00:00
Rusty Russell
d659dc4080 peer_control: fix autodata compile issue. (#2783)
Caused by merge, and two files with autodata on same line.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 17:59:45 +08:00
Christian Decker
915c932e2d pytest: Make sure the channels entries don't get deleted anymore
Signed-off-by: Christian Decker <@cdecker>
2019-06-30 16:41:30 +09:30
Christian Decker
86d4362b65 wallet: Don't delete channels from DB, mark them closed.
Since we now have a couple of long-lived dependents it is time we stop
removing channels from the table once they are fully closed, and instead just
mark them as closed. This allows us to keep forwards and transactions foreign
keys intact, and it may help us debug things after the fact.

Fixes #2028

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-06-30 16:41:30 +09:30
Christian Decker
1dbdc74bc3 wallet: Add a final CLOSED state to channels
Instead of deleting the channels we will simple mark them as `CLOSED` from now
on. This is needed for some of the other tables not to end up with dangling
references that would otherwise survive the channel lifetime, e.g., forwards
and transactions.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-06-30 16:41:30 +09:30
lisa neigut
7046d0220c makefiles: move all unit tests under make check-units
Isolate unit tests under their own make directive.
2019-06-30 16:41:30 +09:30
Rusty Russell
267d627521 lightningd: free htlc maps on exit.
Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff02889063e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x555ce2ad8d2e in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x555ce2ad9698 in double_table ccan/ccan/htable/htable.c:226
    #3 0x555ce2ad9b62 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x555ce2a638e4 in htlc_in_map_add lightningd/htlc_end.h:113
    #5 0x555ce2a63beb in connect_htlc_in lightningd/htlc_end.c:39
    #6 0x555ce2a85cbc in channel_added_their_htlc lightningd/peer_htlcs.c:1382
    #7 0x555ce2a860e1 in peer_got_commitsig lightningd/peer_htlcs.c:1466
    #8 0x555ce2a5db04 in channel_msg lightningd/channel_control.c:228
    #9 0x555ce2a8d393 in sd_msg_read lightningd/subd.c:474
    #10 0x555ce2ada157 in next_plan ccan/ccan/io/io.c:59
    #11 0x555ce2adacd4 in do_plan ccan/ccan/io/io.c:407
    #12 0x555ce2adad12 in io_ready ccan/ccan/io/io.c:417
    #13 0x555ce2adcd67 in io_loop ccan/ccan/io/poll.c:445
    #14 0x555ce2a67c66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #15 0x555ce2a6e56b in main lightningd/lightningd.c:822
    #16 0x7ff028242b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 16:41:30 +09:30
Rusty Russell
aba7c7cc75 wallet/txfilter: free outpoint filter and scriptpubkeyset on exit.
Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f7678ee863e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55f8c7b0fce5 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55f8c7b1064f in double_table ccan/ccan/htable/htable.c:226
    #3 0x55f8c7b10b19 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55f8c7afac63 in scriptpubkeyset_add wallet/txfilter.c:30
    #5 0x55f8c7afafce in txfilter_add_scriptpubkey wallet/txfilter.c:77
    #6 0x55f8c7afb05f in txfilter_add_derkey wallet/txfilter.c:91
    #7 0x55f8c7aa4d67 in init_txfilter lightningd/lightningd.c:482
    #8 0x55f8c7aa52d8 in main lightningd/lightningd.c:721
    #9 0x7f767889ab6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f05f389563e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x55cac1e6bc99 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x55cac1e6c603 in double_table ccan/ccan/htable/htable.c:226
    #3 0x55cac1e6cacd in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x55cac1e56e48 in outpointset_add wallet/txfilter.c:61
    #5 0x55cac1e57162 in outpointfilter_add wallet/txfilter.c:116
    #6 0x55cac1e5ea3a in wallet_utxoset_add wallet/wallet.c:2365
    #7 0x55cac1deddc2 in topo_add_utxos lightningd/chaintopology.c:603
    #8 0x55cac1dedeac in add_tip lightningd/chaintopology.c:620
    #9 0x55cac1dee2de in have_new_block lightningd/chaintopology.c:694
    #10 0x55cac1deaab0 in process_rawblock lightningd/bitcoind.c:466
    #11 0x55cac1de9fb4 in bcli_finished lightningd/bitcoind.c:214
    #12 0x55cac1e6f5be in destroy_conn ccan/ccan/io/poll.c:244
    #13 0x55cac1e6f5de in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #14 0x55cac1e7baf5 in notify ccan/ccan/tal/tal.c:235
    #15 0x55cac1e7bfe4 in del_tree ccan/ccan/tal/tal.c:397
    #16 0x55cac1e7c370 in tal_free ccan/ccan/tal/tal.c:481
    #17 0x55cac1e6dddd in io_close ccan/ccan/io/io.c:450
    #18 0x55cac1e6fcf9 in io_loop ccan/ccan/io/poll.c:449
    #19 0x55cac1dfac66 in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #20 0x55cac1e0156b in main lightningd/lightningd.c:822
    #21 0x7f05f3247b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 16:41:30 +09:30
Rusty Russell
7c760dafe2 lightningd/chaintopology: free block map and watches on exit.
Direct leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f4dc279163e in calloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c63e)
    #1 0x564ee8a24bb1 in htable_default_alloc ccan/ccan/htable/htable.c:19
    #2 0x564ee8a2551b in double_table ccan/ccan/htable/htable.c:226
    #3 0x564ee8a259e5 in htable_add_ ccan/ccan/htable/htable.c:331
    #4 0x564ee89a5300 in block_map_add lightningd/chaintopology.h:83
    #5 0x564ee89a6ece in add_tip lightningd/chaintopology.c:626
    #6 0x564ee89a72c3 in have_new_block lightningd/chaintopology.c:694
    #7 0x564ee89a3ab0 in process_rawblock lightningd/bitcoind.c:466
    #8 0x564ee89a2fb4 in bcli_finished lightningd/bitcoind.c:214
    #9 0x564ee8a284d6 in destroy_conn ccan/ccan/io/poll.c:244
    #10 0x564ee8a284f6 in destroy_conn_close_fd ccan/ccan/io/poll.c:250
    #11 0x564ee8a34a0d in notify ccan/ccan/tal/tal.c:235
    #12 0x564ee8a34efc in del_tree ccan/ccan/tal/tal.c:397
    #13 0x564ee8a35288 in tal_free ccan/ccan/tal/tal.c:481
    #14 0x564ee8a26cf5 in io_close ccan/ccan/io/io.c:450
    #15 0x564ee8a28c11 in io_loop ccan/ccan/io/poll.c:449
    #16 0x564ee89b3c3b in io_loop_with_timers lightningd/io_loop_with_timers.c:24
    #17 0x564ee89ba540 in main lightningd/lightningd.c:822
    #18 0x7f4dc2143b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 16:41:30 +09:30
Rusty Russell
98de92c50b lightningd/plugin: use ld's timer loop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 16:41:30 +09:30
Rusty Russell
ef541afd8a lightningd: free strmap of commands on shutdown.
Indirect leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x55d11b77d270 in strmap_add_ ccan/ccan/strmap/strmap.c:90
    #2 0x55d11b704603 in command_set_usage lightningd/jsonrpc.c:891
    #3 0x55d11b733cb5 in param common/param.c:295
    #4 0x55d11b6f7b37 in json_connect lightningd/connect_control.c:96
    #5 0x55d11b7042ef in setup_command_usage lightningd/jsonrpc.c:841
    #6 0x55d11b70443b in jsonrpc_command_add_perm lightningd/jsonrpc.c:863
    #7 0x55d11b704533 in jsonrpc_setup lightningd/jsonrpc.c:876
    #8 0x55d11b705695 in new_lightningd lightningd/lightningd.c:210
    #9 0x55d11b706062 in main lightningd/lightningd.c:644
    #10 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 16:41:30 +09:30
Rusty Russell
d5bd1682f5 lightningd: free timers on shutdown.
Direct leak of 1024 byte(s) in 2 object(s) allocated from:
    #0 0x7f4c84ce4448 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10c448)
    #1 0x55d11b782c96 in timer_default_alloc ccan/ccan/timer/timer.c:16
    #2 0x55d11b7832b7 in add_level ccan/ccan/timer/timer.c:166
    #3 0x55d11b783864 in timer_fast_forward ccan/ccan/timer/timer.c:334
    #4 0x55d11b78396a in timers_expire ccan/ccan/timer/timer.c:359
    #5 0x55d11b774993 in io_loop ccan/ccan/io/poll.c:395
    #6 0x55d11b72322f in plugins_init lightningd/plugin.c:1013
    #7 0x55d11b7060ea in main lightningd/lightningd.c:664
    #8 0x7f4c84696b6a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26b6a)

To fix this, we actually make 'ld->timers' a pointer, so we can clean
it up last of all.  We can't free it before ld, because that causes
timers to be destroyed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-30 16:41:30 +09:30
Rusty Russell
43d5492619 v0.7.1
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-29 11:02:57 +09:30
Rusty Russell
fc6de9443f v0.7.1rc5.
This has definitely reinforced that we need shorter release cycles!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-28 05:29:48 +00:00
Rusty Russell
f1bea55395 lightningd: fix occasional missing txid detection.
I was working on rewriting our (somewhat chaotic) tx watching code
for 0.7.2, when I found this bug: we don't always notice the funding
tx in corner cases where more than one block is detected at
once.

This is just the one commit needed to fix the problem: it has some
unnecessary changes, but I'd prefer not to diverge too far from my
cleanup-txwatch branch.

Fixes: #2352
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-28 03:31:43 +00:00
Rusty Russell
707ce4cf66 0.7.1rc4
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-27 09:41:53 +00:00
Rusty Russell
a450962b49 channeld/lightningd/hsmd: strengthen our checks against 0-output txs.
If we ever do this, we'd end up with an unspendable commitment tx anyway.
It might be able to happen if we have htlcs added from the non-fee-paying
party while the fees are increased, though.  But better to close the
channel and get a report about it if that happens.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-27 05:55:26 +00:00
Rusty Russell
960bfb89f5 lightningd: don't crash on old zero-output commitment_txs.
We used to produce these, but they're invalid.  When we switched to
libwally it (correctly) refuses to get a txid for them.

Fixes: #2772
Fixes: #2759
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-27 05:55:26 +00:00
Rusty Russell
c1328b3fb7 CHANGELOG.md: v0.7.1rc3
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-06-23 03:26:58 +00:00
Christian Decker
ee587af99e json-rpc: Disable listtransactions for the 0.7.1 release
Due to API instability we are disabling the RPC method for this release, but
will re-enable it after the release again.

Signed-off-by: Christian Decker <@cdecker>
2019-06-23 00:41:19 +00:00