Commit Graph

1008 Commits

Author SHA1 Message Date
Rusty Russell
d24c850899 gossipd: restore a flag for fast pruning
I was seeing some accidental pruning under load / Travis, and in
particular we stopped accepting channel_updates because they were 103
seconds old.  But making it too long makes the prune test untenable,
so restore a separate flag that this test can use.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-27 00:01:34 +00:00
Rusty Russell
4e8141aa36 features: remove INITIAL_ROUTING_SYNC.
It only had an effect if the peer didn't support option_gossip_queries, but
still, we don't want a gossip blast any more.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-25 04:01:56 +00:00
Christian Decker
9915386c4a pytest: Stabilize test_no_fee_estimate against UTXO selection issues
The test was implicitly relying on us selecting the larger output and then not
touching the smaller, leaving it there for the final `withdraw` to claim. This
ordering of UTXOs is not guaranteed, and in particular can fail when switching
DB backends. To stabilize we just need to make sure to select the change
output as well.
2019-09-22 02:03:43 +00:00
Christian Decker
063b319da1 pytest: Have the DB provider search for the postgres binary
This replaces the hard-coded path to the `postgres` and `initdb` binaries with
a slightly more flexible search. It'll pick the newest version installed.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-22 02:03:43 +00:00
Christian Decker
136d4b2568 pytest: Consolidate fee-fetching in test_setchannelfee_usage
The short_channel_id is already in text format, no need to hexlify it :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-22 02:03:43 +00:00
Christian Decker
efc4aa94a3 pytest: Skip some tests that assume we have a sqlite3 db on postgres
These will not work since they touch the DB file itself.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-22 02:03:43 +00:00
Christian Decker
6b26ce5764 db: Split the vars table to have type-specific columns
This was weird right from the start, so we just split the table into integers
and blobs, so each column has a well-defined format. It is also required for
postgres not to cry about explicit casts in the `paramTypes` array.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-22 02:03:43 +00:00
Christian Decker
96a22b4003 pytest: Add db_provider and db instances for configurable backends
We will soon have a postgres backend as well, so we need a way to control the
postgres process and to provision DBs to the nodes. The two interfaces are the
dsn that we pass to the node, and the python query interface needed to query
from tests.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-22 02:03:43 +00:00
Rusty Russell
27790832a5 gossipd: gossip_queries_ex is not longer experimental.
The master spec has some typos which make it not parse, so I created
a PR and generated the CSV from that:

https://github.com/lightningnetwork/lightning-rfc/pull/673

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-22 01:17:11 +00:00
Rusty Russell
39c9dcbafc ratelimit: adjust based on --dev-fast-gossip, test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell
147eaced2e developer: consolidiate gossip timing options into one --dev-fast-gossip.
It's generally clearer to have simple hardcoded numbers with an
#if DEVELOPER around it, than apparent variables which aren't, really.

Interestingly, our pruning test was always kinda broken: we have to pass
two cycles, since l2 will refresh the channel once to avoid pruning.

Do the more obvious thing, and cut the network in half and check that
l1 and l3 time out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell
a92ead48bf gossipd: ignore redundant channel_update and node_announcement.
If you send a message which simply changes timestamp and signature, we
drop it.  You shouldn't be doing that, and the door to ignoring them
was opened by by option_gossip_query_ex, which would allow clients to
ignore updates with the same checksum.

This is more aggressive at reducing spam messages, but we allow refreshes
(to be conservative, we allow them even when 1/2 of the way through the
refresh period).

I dropped the now-unnecessary sleep from test_gossip_pruning, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell
0bab2580fc gossipd: clean up local channel updates.
Make update_local_channel use a timer if it's too soon to make another
update.

1. Implement cupdate_different() which compares two updates.
2. make update_local_channel() take a single arg for timer usage.
3. Set timestamp of non-disable update back 5 minutes, so we can
   always generate a disable update if we need to.
4. Make update_local_channel() itself do the "unchanged update" suppression.
   gossipd: clean up local channel updates.
5. Keep pointer to the current timer so we override any old updates with
   a new one, to avoid a race.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
Rusty Russell
70c4ac6d74 gossipd: suppress our own too-close node_announcement messages.
Never make them less than gossip_min_interval apart.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-20 06:55:00 +00:00
trueptolemy
c737fa6b91 pay: Fix logic of the intereface find_worst_channel 2019-09-17 21:06:12 +02:00
trueptolemy
d4104c15aa pytest: Test pay can exclude error node 2019-09-16 12:22:06 +08:00
trueptolemy
a3a0f651e3 pytest: Let fail_htlcs.py plugin generates TEMPORARY_NODE_FAILURE error 2019-09-16 12:22:06 +08:00
trueptolemy
059a6e0e0d pytest: Test excluding nodes in getroute 2019-09-16 12:22:06 +08:00
Rusty Russell
9aca38b279 pytest: increase non-developer timeout.
We overrode this with travis, but it's not a travis issue.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-12 05:11:56 +00:00
lisa neigut
904a1384ef tests: de-flake lightningd_still_loading 2019-09-11 23:56:27 +00:00
lisa neigut
a35677fa0f fundchannel: use plugin; delete interior impl
Switch over to using the fundchannel plugin.
2019-09-11 23:56:27 +00:00
lisa neigut
c261309f18 test: fund wallet before testing fundchannel
After switching to a plugin, we verify that we can fund a channel
before we check to contact a peer. We'll need to have a funded wallet
to pass the check in this test that verifies that 'fundchannel' cannot
be called for a peer after fundchannel_start is.
2019-09-11 23:56:27 +00:00
lisa neigut
5c70890efa rpc: add 'utxos' parameter to txprepare + withdraw
Allow a user to select the utxo set that will be added to a
transaction, via the `utxos` parameter. Optional.

Format for utxos should be of the form ["txid:vout","..."]
2019-09-11 23:56:27 +00:00
trueptolemy
8d28e52515 pytest: Test fundchannel_cancel can catch tx braodcast by bitcoind
For now, we can't fully ensure that the broadcast was catched from a third pary. Only when the transaction (broadcast by a third pary) is onchain, we can catch it.
2019-09-11 17:04:14 -05:00
trueptolemy
f1822e3650 pytest: Test fundchannel_cancel can work after fundchannel_complete 2019-09-11 17:04:14 -05:00
trueptolemy
619362ae19 pytest: Add a test for 'sendpay_success' and 'sendpay_failure' 2019-09-11 00:57:39 +00:00
trueptolemy
5fa3a50896 pytest: Add a simple plugin to test 'sendpay_success' and 'sendpay_failure' 2019-09-11 00:57:39 +00:00
Rusty Russell
226e2aee48 option_static_remotekey: update to latest draft.
531c8d7d9b

In this one, we always send my_current_per_commitment_point, though it's
ignored.  And we have our official feature numbers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-10 16:18:25 -05:00
Rusty Russell
0861279b65 channeld: don't exchange my_current_per_commitment_point if option_static_remotekey
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-10 16:18:25 -05:00
Rusty Russell
5203847025 common/features: if EXPERIMENTAL_FEATURES, advertise option_static_remotekey
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-10 16:18:25 -05:00
darosior
a7cbe93fb8 closingd: retransmit 'funding_locked' if we reconnect without any update
As per BOLT02 #message-retransmission :
if `next_commitment_number` is 1 in both the `channel_reestablish` it sent and received:
    - MUST retransmit `funding_locked`
2019-09-10 02:02:51 +00:00
Rusty Russell
ddab2cb287 pytest: remove flake8 warnings for tests/fixtures.py
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-07 09:51:03 +00:00
Christian Decker
cb96be9a6e pytest: Allow running tests in a shared memory directory
It seems we spend a lot of time waiting for `bitcoind` and `lightningd` to
talk to disks. This adds the `TEST_DIR` environment variable, allowing for
example to use `/dev/shm`, or a faster disk than the disk `/tmp` is on, as the
root directory for all test-related files.

Testing this on one of our builder machines cut the time to run the entire
suite under valgrind roughly in half (180-200 seconds vs 440-490 seconds).
2019-09-07 09:51:03 +00:00
Christian Decker
b90b4b4bb1 pytest: Consolidate node teardown checks a bit
These are following the same pattern over and over again, so I just added a
tiny wrapper so we reduce the amount of clutter.
2019-09-07 09:51:03 +00:00
Christian Decker
ce2bdeec70 pytest: Add a class that collects teardown checks
My machine would accumulate a number of zombie lightningd and bitcoind
processes over time while testing. Investigating this showed that if a fixture
raised an exception during fixture teardown then other fixtures that have not
been torn down would linger around. The issue is that pytest treats exceptions
in fixtures as non-recoverable and therefore will not catch them and call the
remaining ones.

This commit adds a new fixture, that is there just to collect eventual errors
from other fixtures and ensure that anything that needs to clean up something,
e.g., processes started by the fixture, are cleaned up before we raise an
eventual exception. This is achieved by making any fixture that needs cleaning
up dependent on the teardown_checks fixture, which also serves as central
point to collect errors and printer of eventual errors.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-07 09:51:03 +00:00
Rusty Russell
1c0d435f5e pytest: remove flaky part of test_gossip's test_gossip_no_empty_announcements
This "wait_for" failed on Travis, but it's unnecessary anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:35:01 +02:00
Rusty Russell
c99906a9a9 per-peer-daemons: tie in gossip filter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:35:01 +02:00
Rusty Russell
5292f11818 pytest: test (fail) that we don't repeat gossip back to the node we got it from
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:35:01 +02:00
Rusty Russell
714e7fb670 pytest: test db upgrade.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-06 14:19:14 +02:00
Rusty Russell
a134062f98 bolt11: handle 9 fields for new features.
This implements https://github.com/lightningnetwork/lightning-rfc/pull/656

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-05 23:39:05 -05:00
Christian Decker
716a3b11a5 sqlite3: Move begin transaction and commit into the driver
This has a slight side-effect of removing the actual begin and commit
statements from the `db_write` hooks, but they are mostly redundant anyway (no
harm in grouping pre-init statements into one transaction, and we know that
each post-init call is supposed to be wrapped anyway).

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
492d77f213 db: Add setup and teardown function to DB
These are used to do one-time initializations and wait for pending statements
before closing.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
trueptolemy
74055eada5 pytest: Test txprepare with multiple outputs 2019-09-05 16:05:36 -05:00
trueptolemy
cdcafdaf74 API: txprepare now support mutiple outputs 2019-09-05 16:05:36 -05:00
lisa neigut
10ed2ebdb4 wallet: track our change outputs
Add change scripts to our txfilter so that we mark them as confirmed
(and can subsequently spend them)
2019-09-05 15:48:49 +02:00
lisa neigut
01e0ba8232 test: add check that withdraw's change is confirmed
We're not tracking change outputs for withdraws, so they're not
confirming. This test exposes the problem.
2019-09-05 15:48:49 +02:00
lisa neigut
4d2f91f7ae test-fixtures: use helper for checking errors
log files were being deleted on memleak errors, since
we weren't marking the node has having an error.

this helper function is designed to exactly handle this, so
we use the helper function and modify it to print any additional
error messages that are handed back from killall.
2019-09-05 01:32:48 +00:00
lisa neigut
df1d92a7a2 test-fixtures: return error set, don't throw exception
Throwing an exception while killing all nodes meant that
we aren't cleaning up all the nodes properly. Instead,
collect the errors, and return them back to the upper level,
where we report them and terminate as expected.
2019-09-05 01:32:48 +00:00
lisa neigut
63c80e7aa9 test-fixtures: move 'broken' check to after memleak
Memleaks appear in the logs as 'broken', so the broken log
check captures them as well. This moves broken to after memleak
so we get more informative error messages.
2019-09-05 01:32:48 +00:00
Christian Decker
fd1e8a5999 pytest: Fix directory deletion on passed tests
We were checking the test request against the searched for string. This fixes
it by actually looking at the outcome instead and should clean up correctly
if tests do not fail.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-02 16:31:36 +02:00