Commit Graph

2202 Commits

Author SHA1 Message Date
practicalswift
61c47c09d0 Fix typos 2017-12-08 13:07:20 +01:00
Rusty Russell
5444084625 onchaind: fix reversed CSV delays.
Our config -> their CSV delay, their config -> our CSV delay.

Fixes: #395
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-04 13:00:57 +01:00
Rusty Russell
af5dbbc9f8 json_connect: separate port arg so we can parse IPv6 addresses.
Fixes: #391
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-04 11:15:59 +01:00
Rusty Russell
70cebb8653 test_addfunds_from_block: fix test flakiness.
If you run locally, it fails occasionally; presumably because it
sees previous funds.  Use a random HSM key for that teste.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-12-02 12:52:18 +01:00
Christian Decker
12944ffc9b pytest: Switch to 'Hello world' as startup marker
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-29 14:39:12 +01:00
Christian Decker
88ecae301c pytest: Add a test for funds coming from blocks
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-29 14:39:12 +01:00
Rusty Russell
cfde208e82 tests: prepare for bitcoind 0.16.
I noticed some breakage with git master:
1. getinfo no longer supported (for us, use getblockchaininfo)
2. generate no longer supported (use generatetoaddress)

Both these options are supported at least in 0.15, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-28 12:15:22 +01:00
Christian Decker
547d3f0a0b pytest: Attempting to fix the flaky gossip test
This seems to happen when we manage to check between the
channel_announcement and the channel_update being processed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-23 23:45:28 +00:00
Christian Decker
25725c0aa4 pytest: Add benchmarks for direct payments
Add two simple tests: one for a single direct payment and one with
hundreds of parallel payments, reusing the same route.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-23 23:45:28 +00:00
Rusty Russell
6c79550c22 test_lightningd.py: test updating feerates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-23 12:40:32 +01:00
Rusty Russell
88af0f5bf8 tests: change --override-fee-rate to --override-fee-rates and use in tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-23 12:40:32 +01:00
Rusty Russell
35a6ab8151 invoice: don't allow payments in expired invoices.
Fixes: #363
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-23 00:11:04 +01:00
Christian Decker
4d8ad4e17d pytest: Test that upon reconnect the channel gets re-enabled
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-20 06:13:45 +00:00
Christian Decker
f524df721d pytest: Add test for waitanyinvoice
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-15 00:07:44 +00:00
Christian Decker
07e5a9ef9f htlc: Allow for exactly min_final_cltv_expiry cltv delta
We are announcing that we are willing to accept incoming payments with
current_height + min_final_cltv_expiry + slack, assuming that the
sender adds some slack. In particular we'd reject the payment if
slack=0 which is allowed by the spec.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-11 01:28:03 +00:00
Christian Decker
d6af14a869 pytest: Valgrind errors trump exit value errors
Raising the exception about non-zero exit values into the
teardown. This was previously masking the valgrind errors. Now
valgrind errors > crash errors > non-zero return value.

Still hoping to catch that elusive [7, 0] return value on travis.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-11-10 02:20:00 +00:00
Rusty Russell
8cef36cbd7 lightningd: fail htlcs we fulfill if peer unresponsive after deadline.
Closes: #241
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-03 05:18:49 +00:00
Rusty Russell
1142c44c29 lightningd: fail htlcs we offer if peer unresponsive after deadline.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-03 05:18:49 +00:00
Rusty Russell
c14b159166 lightningd: remove --deadline-blocks option.
We will derive it from other factors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-03 05:18:49 +00:00
Rusty Russell
e1dc75e954 pay: new all-in-one RPC command.
Closes: #240
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-01 01:09:23 +00:00
Rusty Russell
d577769350 invoice: provide bolt11 invoice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-01 01:09:23 +00:00
Rusty Russell
8d09734536 bolt11: add decodepay RPC helper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-01 01:09:23 +00:00
Rusty Russell
7ed81cdf7b pay: factor out actual payment sending from json part of sendpay.
We're going to reuse this for the new 'pay' all-in-one command.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-11-01 01:09:23 +00:00
Rusty Russell
0000dc5908 lightningd: Allow in excess of ctlv_expiry.
We add a test which both overpays and over-sets the delta on the
intermediate and final nodes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 14:36:50 +01:00
Rusty Russell
71e794a046 lightningd: split ctlv_expiry and final_cltv.
These need to be different for testing the example in BOLT 11.

We also use the cltv_final instead of deadline_blocks in the final hop:
various tests assumed 5 was OK, so we tweak utils.py.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 14:36:50 +01:00
Rusty Russell
b257b8960b test_lightning.py: more debugging for the tx decoding fail under travis.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 04:14:33 +00:00
Rusty Russell
390bf6359e test_lightning.py: don't get confused by valgrind core files.
I run with ulimit -c unlimited, and valgrind leaves core files like
valgrind-errors.22114.core.22114 which test_lightning.py tries to
parse as log files.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 04:14:33 +00:00
Rusty Russell
bc9918ad46 dev: option not to do backtracing.
It crashes under valgrind, causing a valgrind error: valgrind gives us a
backtrace anyway, so we don't need it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-31 04:14:33 +00:00
Rusty Russell
2aadd351f8 test_lightningd.py: more information when we fail to find payment.
Travis has been hitting this intermittantly, can't reproduce here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-28 13:33:00 +02:00
Rusty Russell
0c7ca9ab7c gossipd: call to return all connected peers.
And we report these through the getpeers JSON RPC again (carefully: in
our reconnect tests we can get duplicates which this patch now filters
out).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-26 21:01:09 +00:00
Rusty Russell
3c6eec87e3 Add DEVELOPER flag, set by default.
This is a bit messier than I'd like, but we want to clearly remove all
dev code (not just have it uncalled), so we remove fields and functions
altogether rather than stub them out.  This means we put #ifdefs in callers
in some places, but at least it's explicit.

We still run tests, but only a subset, and we run with NO_VALGRIND under
Travis to avoid increasing test times too much.

See-also: #176
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-26 12:53:09 +02:00
Rusty Russell
b6a2b8c58b Add --rgb and --alias options.
And derive random ones from nodeid if they don't choose.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-25 09:16:14 +00:00
Rusty Russell
66a0c55322 test_lightning.py: fix float insanity with values.
When is 0.01 != 0.01?  When there are floats involved!  Jenkins hit an
error once, I have no idea why.

This works around the following intermittant error:

ERROR: test_closing_negotiation_reconnect (__main__.LightningDTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/test_lightningd.py", line 1601, in test_closing_negotiation_reconnect
self.fund_channel(l1, l2, 10**6)
File "tests/test_lightningd.py", line 241, in fund_channel
raise ValueError("Can't find {} payment in {}".format(amount, tx))
ValueError: Can't find 1000000 payment in 02000000000101b0b27be92916faee59f197c263e1ae7b44c0e59acdf2c385c55b5b04670ac026010000001716001401fad90abcd66697e2592164722de4a95ebee165ffffffff02651d0f0000000000160014c2ccab171c2a5be9dab52ec41b825863024c546640420f00000000002200205b8cd3b914cf67cdd8fa6273c930353dd36476734fbd962102c2df53b90880cd02473044022017bd19a0ee85532f67a280c71ed02d1321f08975334bd281527478022265225702202ec448bf9c0890a31a26f0ef4f03d298c8ec16b277faff09a70ddd335df44b6e012103d745445c9362665f22e0d96e9e766f273f3260dea39c8a76bfa05dd2684ddccf00000000

For testing, that tx decodes to:
{
  "txid": "0165e92be762b352b665b76b9872d5189e1b2a8faf4918ab3cca7cd5d4b6a5fa",
  "hash": "8af9a36c79ee5243468c5cbed1c80f10238fba405f0ad957a0c2cfc46fb632f5",
  "version": 2,
  "size": 257,
  "vsize": 176,
  "locktime": 0,
  "vin": [
    {
      "txid": "26c00a67045b5bc585c3f2cd9ae5c0447baee163c297f159eefa1629e97bb2b0",
      "vout": 1,
      "scriptSig": {
        "asm": "001401fad90abcd66697e2592164722de4a95ebee165",
        "hex": "16001401fad90abcd66697e2592164722de4a95ebee165"
      },
      "txinwitness": [
        "3044022017bd19a0ee85532f67a280c71ed02d1321f08975334bd281527478022265225702202ec448bf9c0890a31a26f0ef4f03d298c8ec16b277faff09a70ddd335df44b6e01",
        "03d745445c9362665f22e0d96e9e766f273f3260dea39c8a76bfa05dd2684ddccf"
      ],
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00990565,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 c2ccab171c2a5be9dab52ec41b825863024c5466",
        "hex": "0014c2ccab171c2a5be9dab52ec41b825863024c5466",
        "type": "witness_v0_keyhash"
      }
    },
    {
      "value": 0.01000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 5b8cd3b914cf67cdd8fa6273c930353dd36476734fbd962102c2df53b90880cd",
        "hex": "00205b8cd3b914cf67cdd8fa6273c930353dd36476734fbd962102c2df53b90880cd",
        "type": "witness_v0_scripthash"
      }
    }
  ]
}

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-24 16:12:22 +02:00
Rusty Russell
c2dd0cb295 test_lightningd.py: return short channel id from fund_channel helper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-24 16:12:22 +02:00
Rusty Russell
c3bed51b2d test_lightningd.py: make HSM seeds constant for tests.
Makes it easier to compare before/after failures.  Ideally, we should
run under Travis both with this option and with the seed based on the
entire tmp path (which is still reproducible with determination, but
not fixed every run like this is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-24 16:12:22 +02:00
Rusty Russell
5a256c724a subd: simplify and cleanup lifetime handling.
There are now only two kinds of subdaemons: global ones (hsmd, gossipd) and
per-peer ones.  We can handle many callbacks internally now.

We can have a handler to set a new peer owner, and automatically do
the cleanup of the old one if necessary, since we now know which ones
are per-peer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-20 18:31:32 +02:00
Rusty Russell
474887512d gossipd: rewrite to do the handshake internally.
Now the flow is much simpler from a lightningd POV:

1. If we want to connect to a peer, just send gossipd `gossipctl_reach_peer`.
2. Every new peer, gossipd hands up to lightningd, with global/local features
   and the peer fd and a gossip fd using `gossip_peer_connected`
3. If lightningd doesn't want it, it just hands the peerfd and global/local
   features back to gossipd using `gossipctl_handle_peer`
4. If a peer sends a non-gossip msg (eg `open_channel`) the gossipd sends
   it up using `gossip_peer_nongossip`.
5. If lightningd wants to fund a channel, it simply calls `release_channel`.

Notes:
* There's no more "unique_id": we use the peer id.
* For the moment, we don't ask gossipd when we're told to list peers, so
  connected peers without a channel don't appear in the JSON getpeers API.
* We add a `gossipctl_peer_addrhint` for the moment, so you can connect to
  a specific ip/port, but using other sources is a TODO.
* We now (correctly) only give up on reaching a peer after we exchange init
  messages, which changes the test_disconnect case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-20 18:31:32 +02:00
Rusty Russell
2273ce783e dev_disconnect: support multiple disconnects in the same daemon.
We currently assume the daemon gives up; gossipd won't, and we want to
test it there too.

This reveals a bug (returning io_close() is bad if the call is to
duplex()), and breaks a test which now continues after dropping a
packet..

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-20 18:31:32 +02:00
Rusty Russell
6ceec17943 dev_disconnect: make commit suppression a "-nocommit" modifier.
Useful if we want to drop & suppress, for example.  We change '=' to mean
do nothing to the packet.

We use this to clean up the test_reconnect_sender_add test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-20 18:31:32 +02:00
Rusty Russell
98ad6b9231 lightningd: change connect RPC args.
We're going to make the ip/port optional, so they should go at the end.
In addition, using ip:port is nicer, for gethostbyaddr().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-20 18:31:32 +02:00
Rusty Russell
871d0b1d74 lightningd: simplify peer destruction.
We have to do a dance when we get a reconnect in openingd, because we
don't normally expect to free both owner and peer.  It's a layering
violation: freeing a peer should clean up the owner's pointer to it,
to avoid a double free, and we can eliminate this dance.

The free order is now different, and the test_reconnect_openingd was
overprecise.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-20 18:31:32 +02:00
Rusty Russell
8f057f7fc7 Revert "gossip: send the *other* node's cltv_expiry_delta in channel_announce."
This reverts commit 297e278132.
2017-10-11 11:54:50 +02:00
Rusty Russell
297e278132 gossip: send the *other* node's cltv_expiry_delta in channel_announce.
Include tests from example doc.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-10 20:17:37 +02:00
Rusty Russell
cd8edd1716 test_lightningd: allow multiple extra options to lightningd
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-10 20:17:37 +02:00
Christian Decker
f5a412d90d pytest: Change the channel persistence test to add inflight HTLCs
Now that we have HTLC persistence we'd also like to test it. This
kills the second node in the middle of an HTLC, it'll recover and
finish the flow.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-10-09 11:51:13 +10:30
Christian Decker
a4967d74f1 pytest: Added a simple breakpoint shorthand
I was typing this over and over again, adding this shorthand mainly to
remember it :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-10-09 11:51:13 +10:30
Christian Decker
cf15670c3f pytest: Disabling test_penalty_outhtlc since it is too flaky
The test fails very often and causes the CI build to die. Needs to be
fixed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-10-09 11:51:13 +10:30
Rusty Russell
243fc2c05a utils: save log file on failure.
This broke somewhere in the recent changes, because we override
TailalbleProc stop().  Break out log extractor.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-10-03 12:56:08 +02:00
Christian Decker
5029f95066 pytest: Use futures to wait for failing sendpays
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-29 10:06:56 +09:30
Christian Decker
0b350d2f5f pytest: Integrate with known/allowed failures
Moved the flagging for allowed failures into the factory getter, and
renamed into `may_fail`. Also stopped the teardown of a node from
throwing an exception if we are allowed to exit non-cleanly.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-29 10:06:56 +09:30
Christian Decker
c1f4c86589 pytest: Make sure to clean up all lightningds after failures
A failed returncode check could result in the cleanup for other
lightningds to be skipped. Now make sure to cleanup all and then
rethrow an exception that contains all returncodes.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-29 10:06:56 +09:30
Christian Decker
11eaabdbe6 pytest: Stopping daemon cleanly
We used to simply kill the daemon, which in some cases could result in
half-written crashlogs and similar artifacts such as half-completed
RPC calls. Now we ask lightningd to stop nicely, give it some time and
only then kill it. We also return the returncode of the daemon.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-29 10:06:56 +09:30
Rusty Russell
2fe2a0bcf9 peer_control: don't double-free on permanent fail of non-persistent peer.
peer_fail_permanent() frees peer->owner, but for bad_peer() we're
being called by the sd->badpeercb(), which then goes on to
io_close(conn) which is a child of sd.

We need to detach the two for this case, so neither tries to free the
other.

This leads to a corner case when the subd exits after the peer is gone:
subd->peer is NULL, so we have to handle that too.

Fixes: #282
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 15:37:43 +02:00
Rusty Russell
8741043a0e test_lightningd: rename has_failed to allow_failure.
Note that it should really be a flag to daemon on construction, too,
but that may interfere with another concurrent branch so I've deferred.

Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
96a85a786c opening: save the correct commitment tx when we are the fundee.
We were saving *our* commitment tx, not theirs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
aea77653d3 onchaind: update bolt #5, and implement failure of timed-out onchain HTLCs.
We re-use the value for reasonable_depth given by the master, and we
tell it when our timeout transactions reach that depth.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
5666ba551f onchaind: fail htlcs which are not included in commitment tx.
As per update 149cf020d6

Fixes: #249
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
1ac9e0b532 onchain: implement penalty transaction.
Fixes: #242
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
d640dc3a6f test_lightningd.py: fix wait_for_logs with duplicate entries.
In the next test, we wait for multiple 'sendrawtx exit 0' which
doesn't work because we use a set not a list, and the current code
would match multiple against the same thing.  The result was we didn't
wait for the final sendrawtransaction, and occasionally had test
failures as a result.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
cbe72b658e onchaind: extract payment_preimage from onchain HTLC redemption.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
f96a04e2a6 onchaind: resolve HTLC transactions.
When we sent out an HTLC-Timeout or HTLC-Success tx, we need to spend
it after the timeout so it's safely in our wallet.

We generalize the tx_type OUR_UNILATERAL_TO_US_RETURN_TO_WALLET to
OUR_DELAYED_RETURN_TO_WALLET, since we use it for HTLC transactions too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Rusty Russell
a76a53a933 onchaind: fulfill HTLCs onchain.
When we see an offered HTLC onchain, we need to use the preimage if we
know it.  So we dump all the known HTLC preimages at startup, and send
new ones as we discover them.

This doesn't cover preimages we know because we're the final
recipient; that can happen if an HTLC hasn't been irrevocably
committed yet.  We'll do that in a followup patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-28 13:07:05 +09:30
Christian Decker
081413e247 pytest: Added test to check node_announcement signatures
Reported-by: Rusty Russell
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-16 09:02:06 +09:30
Rusty Russell
584b160cdc lightningd: avoid errors on shutdown.
lightningd can crash on shutdown if it's in the middle of getchaintips;
we free the conn, the finished callback is called (process_chaintips),
and it reports that it received an empty result.

The simplest fix is to set a flag in the struct bitcoind destructor,
and avoid the callback.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-12 23:00:53 +02:00
Rusty Russell
ec63c0d10b lightningd: give option to crash if a subdaemon fails.
Either when it exits with a signal, or sends an error status message.
Then we make test_lightningd.py use it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-12 23:00:53 +02:00
Rusty Russell
64a26b06e7 tests/test_lightningd.py: fail on crash.log.
We simply kill lightningd; we should stop it properly and have a timeout
to kill it if that fails.  However, that's beyond my python skills :(

So we just look for crash.log.  Unfortunately, we usually kill
lightningd before it's finished writing it.  So we look for it and
don't kill lightningd, just wait in this case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-12 23:00:53 +02:00
Rusty Russell
f219955dc6 wallet: fix spending of change.
We recorded our own change as P2SH; it's not.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-12 22:59:24 +02:00
Christian Decker
fa13190595 wallet: Hook into the hsm_funding_sig to extract change outputs
This is the step where we broadcast the transaction to the network and
a nice place to extract the change from the transaction.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-10 10:48:53 +09:30
Christian Decker
f2d67f88c0 pytest: Run sendpay in a future so it doesn't complain about failing
For the permfail tests the sendpay call is supposed to fail, so this
was printing stacktraces upon success. Running in futures captures any
thrown exceptions and rethrows them when calling `result()`, in our
case we just ignore them.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-08 17:02:04 +09:30
Christian Decker
641a3b64cf channel: Use min_cltv_expiry in the channel announcements
So far we were always using the deadline in the announcements, that's
obviously not good, so this introduces the parameter as per spec.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-09-08 17:02:04 +09:30
Rusty Russell
b19a4516d4 lightningd: fix crash when old openingd still around.
We weren't killing it.  Eventually it would die, and peer_owner_finished()
would access subd->peer->owner, but that peer was freed already.

Closes: #261
Reported-by: Christian Decker <decker.christian@gmail.com>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-06 19:11:21 +02:00
Rusty Russell
5889ad5fc4 dev-disconnect: add blackhole option.
To reproduce the next bug, I had to ensure that one node keeps thinking it's
disconnected, then the other node reconnects, then the first node realizes
it's disconnected.

This code does that, adding a '0' dev-disconnect modifier.  That means
we fork off a process which (due to pipebuf) will accept a little
data, but when the dev_disconnect file is truncated (a hacky, but
effective, signalling mechanism) will exit, as if the socket finally
realized it's not connected any more.

The python tests hang waiting for the daemon to terminate if you leave
the blackhole around; to give a clue as to what's happening in this
case I moved the log dump to before killing the daemon.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-06 19:11:21 +02:00
Rusty Russell
40fc95921d Cleanup: remove unused IRC flag.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-04 08:47:13 +02:00
Rusty Russell
5a5e23c011 test_lightning.py: use NO_VALGRIND instead of NOVALGRIND.
This variable also turns of valgrind on the unit tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-09-03 02:01:54 +02:00
Rusty Russell
1cf33eefe2 lightningd: handle case where channeld fails locally-generated HTLC.
jl777 reported a crash when we try to pay past reserve.  Fix that (and
a whole class of related bugs) and add tests.

In test_lightning.py I had to make non-async path for sendpay() non-threaded
to get the exception passed through for testing.

Closes: #236
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-30 11:36:37 +02:00
Rusty Russell
bbed5e3411 Rename subdaemons, move them into top level.
We leave the *build* results in lightningd/ for ease of in-place testing though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-29 17:54:14 +02:00
Rusty Russell
88bb38f63b daemon/lightningd: remove building, and main files
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-29 17:54:14 +02:00
Christian Decker
4b64b7f2aa lightningd: Fix channel-persistence for channels with commits
I was hoping to defer HTLC updates until we actually store HTLCs, but
we need to flush to DB whenever balances update as well.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-08-26 16:37:53 +09:30
Rusty Russell
0a88948849 test_lightningd: explicitly test the new commit point case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-23 17:38:22 +02:00
Rusty Russell
e7633613a9 test_lightningd: onchain tests.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-23 17:38:22 +02:00
Christian Decker
e51d261f51 lightningd: Load persisted channels on startup
This is the big one, and it's completely anticlimactic: it loads all
channels that have reached opening and are not marked as
closingd_complete into memory, that's it.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2017-08-23 10:23:54 +09:30
Christian Decker
29785d4990 pytest: Added a simple channel-persistence test
This test opens a channel, stops the nodes and the restarts them to
see if we can successfully reload the channel state from the database.
2017-08-23 10:23:54 +09:30
Christian Decker
c0aefad8e3 pytest: Add a small helper to run queries against the node's DB
This should simplify checking that some actions have been persisted to
DB.
2017-08-23 10:23:54 +09:30
Christian Decker
121595935a pytest: Reduce noise from bitcoind
I was hoping to trigger on more things from the bitcoind process, but
stuff like mempool is hard to trigger on. Reducing to info so we can
work a bit easier with pdb and the log becomes less noisy.
2017-08-23 10:23:54 +09:30
Rusty Russell
fc24c71196 tests/test_lightningd.py: add async option for pay command.
We'll need this for testing nodes going down during payment.

However, there's no good way to silence the threads that I can tell,
so we get a nasty backtrace from it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-20 13:06:41 +09:30
Rusty Russell
8e0c19c76a lightningd: add dev-fail command to inject permenant failure.
A couple of double-free bugs founnd doing this, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-20 13:06:41 +09:30
Rusty Russell
1ad960413d test_lightningd: fix valgrind detection on children.
I tracked down a bug, and couldn't figure out why valgrind wasn't
finding it.

From man valgrind:

       --log-file=<filename>
           Specifies that Valgrind should send all of its messages to the
           specified file. If the file name is empty, it causes an abort.
           There are three special format specifiers that can be used in the
           file name.

           %p is replaced with the current process ID. This is very useful for
           program that invoke multiple processes. WARNING: If you use
           --trace-children=yes and your program invokes multiple processes OR
           your program forks without calling exec afterwards, and you don't
           use this specifier (or the %q specifier below), the Valgrind output
           from all those processes will go into one file, possibly jumbled
           up, and possibly incomplete.

"possibly incomplete" indeed!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-08-20 13:06:41 +09:30
Christian Decker
9ad8365589 pytest: Split test_disconnect
These were in part reusing some of the nodes, and creating new
instances for other, so I split them to avoid having cross pollution
between the tests.
2017-08-17 10:31:55 +09:30
Christian Decker
943b6526c7 pytest: Configure non-legacy tests correctly and filter legacy 2017-07-12 11:30:23 +09:30
Christian Decker
843e21826a opts: Remove --regtest flag and pass chainparams to bitcoind 2017-07-12 11:30:23 +09:30
Christian Decker
8e043fc448 pytest: Consolidating sync_blockheight and fixing flaky test 2017-07-12 11:00:26 +09:30
Christian Decker
1a1e29a4bc gossip: Re-initiate the broadcast timer upon reconnect
We weren't registering reconnecting peers for broadcasts. Just
starting a timer is enough. Also added an integration test to check
that the gossip sync is being resumed.
2017-07-12 11:00:26 +09:30
Rusty Russell
f04222cb12 test_lightningd.py: make pay helper use random label if we don't care.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell
87232ec25c lightningd: new peer state CLOSINGD_COMPLETE.
This is a transitional state, while we're waiting to see the
closing tx onchain (which is To Be Implemented).

The simplest way to do re-transmission is to re-use closingd, and just
disallow any updates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell
87f6a57d75 tests/test_lightningd.py: test closing and shutdown reconnection.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell
a827d2b2bb lightningd: add in support for closingd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell
d40334ce18 tests/test_lightningd.py: add pay helper for making payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell
6fa90c926a channeld: exit after shutdown when no more HTLCs.
Ready for the introduction of closingd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-12 10:21:16 +09:30
Rusty Russell
4a3c309e84 tests/test_lightningd.py: fix assumption about gossip order
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-03 19:43:34 +09:30
Rusty Russell
9fae82c175 tests: generalize wait_for_log into wait_for_logs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-03 19:43:34 +09:30
Rusty Russell
c05f63a4db peer_control: fix double-state transition on reconnect.
We actually don't need to transition if we're reconnecting, and logic
to go to CHANNELD_NORMAL was wrong: we checked that we'd seen funding tx
locked, but not that we'd received a msg from the remote peer.

We need to fix the tests now we no longer double-transition, too.

Fixes: #188
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-07-03 19:43:34 +09:30
Rusty Russell
5edc14ee1a tests/test_lightningd.py: make test less flaky.
When we drop an HTLC_ADD packet, sometimes the commit timer fires
before we try to read from the fd.  In this case, the payment is
considered committed and we don't fail.

We need manual commit to work around this, and also we'd need to do
the pay command asynchronously from python because it will block.
That's a bit out of scope for now, so just handle either way.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-27 10:25:53 +09:30
Rusty Russell
866ac79850 channeld: shutdown support.
We keep the scriptpubkey to send until after a commitment_signed (or,
in the corner case, if there's no pending commitment).  When we
receive a shutdown from the peer, we pass it up to the master.

It's up to the master not to add any more HTLCs, which works because
we move from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-27 10:25:53 +09:30
Christian Decker
462d936f25 wallet: Add unit test for withdrawal 2017-06-23 16:02:20 +09:30
Rusty Russell
e80ef7dc19 test_lightningd.py: test reconnections on receiver, and htlc_fulfill.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
6dbaa3ace6 test_lightningd.py: test reconnection on htlc_add.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
6b61e9ab0a channeld: exchange channel_reestablish.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
a55b58d0d5 lightningd: track balance in way which matches channeld.
Currently it's fairly ad-hoc, but we need to tell it to channeld when
it restarts, so we define it as the non-HTLC balance.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
ae62509b58 lightningd: remove dev-newhtlc command.
It added another path to the local-htlc handling, and we have full
invoice support now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
4a161b710d channeld: pass htlcs to master daemon in batches.
When adding their HTLCs, it needs all the information.  When failing,
it needs the id as key and the failure reason.  When fulfilling, it
needs the id and payment preimage.

It also needs to know when we have received an revoke_and_ack or a
commitment_signed, to place in the database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
733e0aeccf tests/utils.py: wait_for_log automatic offset.
I have a test which waits for multiple occurrences of the same string,
but doesn't want them to overlap.  Make wait_for_log() do the right thing,
so that it only looks for log entries since the last wait_for_log.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
e161c11e40 tests/utils.py: poll mempool instead of sleeping.
On my laptop under load, 5 seconds was no longer enough for legacy.
But this breaks async (they all see mempool increase, and fire
prematurely), so stop doing that.

I can't get this test to work at all, in fact, without this patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-23 09:29:42 +09:30
Rusty Russell
87fc3439c4 test_lightningd.py: fix obscure corner case in gossip.
I actually hit this very hard to reproduce race: if we haven't process
the channeld message when block #6 comes in, we won't send the gossip
message.  We wait for logs, but don't generate new blocks, and timeout
on l1.daemon.wait_for_log('peer_out WIRE_ANNOUNCEMENT_SIGNATURES').

The solution, which also tests that we don't send announcement signatures
immediately, is to generate a single block, wait for CHANNELD_NORMAL,
then (in gossip tests), generate 5 more.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-06-07 09:19:04 +09:30
Christian Decker
b1d5283ac7 pytest: Testing addfunds json-rpc call 2017-06-06 09:16:10 +09:30
Rusty Russell
06bc035f59 Minor fixes: feedback from Christian
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Rusty Russell
7d5b923719 test_lightningd.py: test reconnect after funding_signed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Rusty Russell
1df40156f6 test_lightningd.py: disconnection tests.
In particular, we test the cases where it will simply forget the old peer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Rusty Russell
780b3870ad lightningd: peer state cleanup.
1. We explicitly assert what state we're coming from, to make transitions
   clearer.
2. Every transition has a state, even between owners while waiting for HSM.
3. Explictly step though getting the HSM signature on the funding tx
   before starting channeld, rather than doing it in parallel: makes
   states clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Rusty Russell
662dfef436 lightningd/gossip: Move INIT message handling to handshake daemon.
We need to do this on every connection, whether reconnecting or not,
so it makes sense for the handshake daemon to handle it and return
the feature fields.

Longer term I'm considering having the handshake daemon handle the
listening and connecting, and simply hand the fds back once the peers
are ready.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Rusty Russell
f6495d3310 lightningd/peer_control: don't create peer struct until we've connected.
We currently create a peer struct, then complete handshake to find out
who it is.  This means we have a half-formed peer, and worse: if it's
a reconnect we get two peers the same.

Add an explicit 'struct connection' for the handshake phase, and
construct a 'struct peer' once that's done.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Rusty Russell
34e6e56471 lightningd: introduce peer_state enum.
The actual state names are place holders for now, really.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-25 14:24:47 +09:30
Christian Decker
80486a6669 pytest: Do not check for valgrind errors if disabled
Checking for valgrind errors if we disabled valgrind will kill the
teardown, so don't!
2017-05-20 19:59:16 +09:30
Rusty Russell
3d2f166364 test_lightningd.py: don't fail if valgrind turned off.
eg:
test_routing_gossip (__main__.LightningDTests) ... ERROR

======================================================================
ERROR: test_routing_gossip (__main__.LightningDTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/test_lightningd.py", line 150, in tearDown
    err_count += self.printValgrindErrors(node)
  File "tests/test_lightningd.py", line 137, in printValgrindErrors
    errors, fname = self.getValgrindErrors(node)
  File "tests/test_lightningd.py", line 132, in getValgrindErrors
    with open(error_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/lightning-l106st0a/test_routing_gossip/lightning-1/valgrind-errors'

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-19 13:30:32 +02:00
Christian Decker
f6d60b9076 pytest: Fail tests that produce valgrind errors
Check and print all valgrind errors and fail tests that produce
errors. We abort using an exception since `fail()` is not allowed in
the teardown.
2017-05-10 12:38:12 +09:30
Rusty Russell
42601c29d7 test_lightning.py: run valgrind on child daemons too
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-09 11:43:20 +09:30
Christian Decker
6d06303789 pytest: Remove obsolete test and fix two flaky tests 2017-05-07 12:06:56 +09:30
Christian Decker
23da30a2a4 pytest: Better dealing with env variables to configure tests 2017-05-07 12:06:56 +09:30
Rusty Russell
162dac1271 peer_control: don't complete fundchannel command until broadcast.
Under stress, the tests can mine blocks too soon, and the funding never
locks.  This gives more of a chance, at least.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-05 16:11:46 +09:30
Rusty Russell
6dcd7f9d6d tests: dump more information when we fail to find something in logs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-05 16:11:46 +09:30
Rusty Russell
2641ecaafa test_lightningd.py: dump logs on error.
I couldn't actually figure out how to just dump them on error, so I
dump all the time.  When running 3 lightningd + bitcoind, this separates
the logs nicely.

TODO: We should delete the directories on success!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-03 11:47:10 +09:30
Rusty Russell
39fb7d82fb test_lightningd.py: re-enable and fix forwarding test.
But it breaks:

	test_forward (__main__.LightningDTests) ... lightningd_channel: Computed MAC does not match expected MAC, the message was modified.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-03 11:47:10 +09:30
Christian Decker
f700662a56 routing: Reading the channel_id from routes passed in from JSONRPC 2017-05-02 11:47:52 +02:00
Christian Decker
f24aab1916 sphinx: Updating daemon to new sphinx implementation
Mainly switching from the old include to the new include and adjusting
the actual size of the onion packet. It also moves `channel.c` to use
`struct hop_data`.

It introduces a dummy next hop in `channel.c` that will be replaced in
the next commit.
2017-05-02 11:47:52 +02:00
Rusty Russell
778b756369 channel: support HTLC forwarding.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-02 11:47:19 +02:00
Rusty Russell
aa01b7d120 test_lightningd.py: helper to sync blockheight.
We need this to avoid spurious errors in the next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-05-02 11:46:46 +02:00
Christian Decker
dc7b832e83 pytest: Added gossip test for new daemon 2017-05-02 11:59:24 +09:30
Rusty Russell
f61da7eb64 tests/test_lightningd.py: incorporate everything from old test-basic shell test.
This moves all the non-legacy blackbox testing into python.

Before:
	real	10m18.385s

After:
	real	9m54.877s

Note that this doesn't valgrind the subdaemons: that patch seems to cause
some issues in the python framework which I am still chasing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-29 10:30:10 +02:00
Rusty Russell
366d67ae6d tests/test_lightningd.py: extract connect and fund_channel helpers.
Common pattern when setting up tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-29 10:30:10 +02:00
Rusty Russell
056f93e2d2 ping: move test to python.
Faster and neater.

Before:
	real	0m11.200s

After:
	real	0m9.101s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-04-29 10:30:10 +02:00
Christian Decker
810e12381a pytest: Simple wrapper around BitcoinProxy to avoid timeouts
Running long integration tests could result in `bitcoind` dropping the
connection inbetween calls, and since python-bitcoinlib does not
reconnect and/or retry, all subsequent tests would fail as well. This
patch switches to throwaway connections, each serving just one
request. It's easier than to reach into the bitcoinlib to reconnect
and reauth, but comes with some overhead, but I think it's acceptable
for the few bitcoin calls we actually perform.
2017-04-28 13:39:45 +09:30
Christian Decker
4e0be7a48f pytest: Added dynamic RPC call dispatch
Instead of having to define every single method we can now just call
any method, it'll fail upstream if it does not exist.
2017-04-24 05:18:11 +00:00
Christian Decker
b6fea5b429 pytest: Check that channels are announced and activated correctly 2017-04-11 16:56:27 -07:00
Christian Decker
5028f7b9d9 fixup! pylightning: RPC returns a failure if the connection is lost 2017-03-20 11:18:00 +10:30
Christian Decker
9c7c2a3f2f pytest: Added simple pytest for the new JSON-RPC 2017-03-20 11:18:00 +10:30
Christian Decker
b5c2d7d47d pytest: Gossip subdaemon had a flaky test
Whenever we were checking for the status before the daemon was
initialized the test would fail. Now just waiting for the state
transition.
2017-02-08 16:59:57 +01:00
Christian Decker
c7e6f197b0 pytest: Chaning startup sentinel for bitcoind.
By looking for 'Done loading' in the log output we should actually be
called after `SetRPCWarmupFinished` in bitcoind. Only then is it safe
to make RPC calls. This resulted in the test suite being a bit flaky.
2017-02-08 16:59:56 +01:00
Christian Decker
66175c7361 Merge pull request #115 from ElementsProject/pytest-pathfix
pytest: don't assume paths of valgrind and bitcoind.
2017-01-24 14:22:23 +01:00
Rusty Russell
1575181ef5 pytest: don't assume paths of valgrind and bitcoind.
In particular, my bitcoind is in /usr/local/bin

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2017-01-24 14:37:56 +10:30
Christian Decker
1948062250 pytest: Using python-bitcoinlib instead of python-bitcoinrpc
The latter is unmaintained and has a number of problems.
2017-01-23 10:37:34 +01:00
Christian Decker
180c96776b pytest: Split the python integration tests into new and legacy 2017-01-23 10:37:34 +01:00
Christian Decker
cf1c03f4d9 pytest: Implemented initial python integration tests. 2017-01-23 10:45:36 +10:30
Christian Decker
41a5e74db9 pytest: Implement utilities for python unittests 2017-01-23 10:45:36 +10:30