Commit Graph

8403 Commits

Author SHA1 Message Date
Rusty Russell
6966cf99e1 bitcoin: add wally_tx_output helper to create standalone output.
In preparation for when we don't have a tx.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell
600d0a4a1d psbt: make psbt_from_b64 more conventional.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Rusty Russell
c1df8d586d utxo: remove unused scriptSig field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-11 08:39:24 +09:30
Christian Decker
03b4662c24 travis: Use a stripped bitcoind bundle
I went and stripped all the parts from the bitcoind tarball and uploaded it to
a GCS bucket. Let's see how much we save.
2020-08-11 08:38:49 +09:30
Christian Decker
33fae1f383 travis: Reorganize tests a bit to move non-integration tests out
We were running the integration tests and the install tests with every
test-group, which doesn't increase test coverage at all. So now we bundle them
with the source check which are short and remove them from the overloaded
configurations.
2020-08-11 08:38:49 +09:30
Vincenzo Palazzo
1521c29fcf listpays mod 1: add destination inside the response when bolt11 is null
Changelog-Added: JSON-RPC: `listpays` now lists the `destination` if it was provided (e.g., via the `pay` plugin or `keysend` plugin)
2020-08-09 16:03:03 +02:00
Vincent
700897f06e listpays: fixed bolt11 null with keysend and update doc command
listpays: make doc-all missed
Changelog-Added: JSON-RPC: `listpays` can be used to query payments using the `payment_hash`
Changelog-Added: JSON-RPC: `listpays` now includes the `payment_hash`
2020-08-09 16:03:03 +02:00
Christian Decker
723b7223b7 pay: Add timestamp of first part to listpays
Changelog-Added: JSON-RPC: The result returned by `listpays` now includes the timestamp of the first part of the payment
2020-08-08 12:53:12 +02:00
Rusty Russell
01a82d38f7 pytest: add slow_test marker.
And when it's set, and we're SLOW_MACHINE, simply disable valgrind.

Since Travis (SLOW_MACHINE=1) only does VALGRIND=1 DEVELOPER=1 tests,
and VALGRIND=0 DEVELOPER=0 tests, it was missing tests which needed
DEVELOPER and !VALGRIND.

Instead, this demotes them to non-valgrind tests for SLOW_MACHINEs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
51aae9cce7 pytest: make valgrind a per-node option.
Next patch will turn it off for slow-marked tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
79278b880f pytest: optimize join_nodes a little.
We can query all the txids at once, rather than one at a time.

Doesn't make any measurable difference to full runtime testing here
though.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
fde353ab00 pytest: use get_nodes more widely.
I started replacing all get_node() calls, but got bored, so then just did the
tests which call get_node() 3 times or more.

Ends up not making a measurable speed difference, but it does make some
things neater and more standard.

Times with SLOW_MACHINE=1 (given that's how Travis tests):

Time before (non-valgrind):
	393 sec (had 3 failures?)
Time after (non-valgrind):
	410 sec

Time before (valgrind):
	890 seconds (had 2 failures)
Time after (valgrind):
	892 sec

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
929fd3e2f6 pytest: make sure all nodes see funds using sync_blockheight
We might have funds prior to calling join_nodes(), so testing that
we've all seen the block is better.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
7bb461acd8 pytest: fix flake in test_channel_opened_notification
We can have the message before the node ready message which line_graph
waits for.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
7aa40b6bae pytest: fix assumption in test_onchain_their_unilateral_out
In fact, the 5 blocks generate above were not seen by nodes until
later, making me very confused.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
02b413a4dc pytest: make join_nodes / line_graph wait for updates in both dirs.
This is what fund_channel() does, which is more thorough than what
we were doing.  But since the order of the logs is undefined, we need
to be a little careful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
8f455c8b98 pytest: new join_nodes to allow you to get all the nodes then join some of them.
This lets you get_nodes() and join some later.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
Rusty Russell
64d40414b2 pytest: speed up test_restart_many_payments.
I thought this was timing out because I made it slow with the
change to txprepare as a plugin.  In fact, it was timing out
because sometimes gossip comes so fast it gets suppressed
and we never get the log messags.

Still, before this it took 98 seconds under valgrind and
24 under non-valgrind, so it's an improvement to time as
well as robustness.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-07 13:57:31 +02:00
ZmnSCPxj jxPCSnmZ
6fadd5aea2 Makefile: Remove gen_version.h from ALL_GEN_HEADERS.
This prevents recompiling everything when you are changing just a doc, or
touching only one file among hundreds of sources, just because the
`gen_version.h` is changed, especially since only one source actually
depends on that header.
2020-08-07 10:41:37 +09:30
ZmnSCPxj jxPCSnmZ
a5fc66c382 plugins/Makefile: Only link in libplugin-pay for plugins that need it.
Changelog-None: cleanup only.

Before this change:

```
$ ls -l plugins/bcli
-rwxrwxr-x 1 zmnscpxj zmnscpxj 1914976 Aug  5 21:54 plugins/bcli
```

After this change:

```
$ ls -l plugins/bcli
-rwxrwxr-x 1 zmnscpxj zmnscpxj 1830552 Aug  5 22:00 plugins/bcli
```

We already duplicate a lot of code between `lightningd` and eeach
builtin plugin because we do not use .so for `common/`, but including
an object file with code that is not referenced in the executable is
fairly low-hanging size optimization fruit.
2020-08-07 10:40:10 +09:30
Jon Griffiths
908a8399e2 psbt: Allocate correctly sized buffer for psbt_to_bytes
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
fef155a9e2 psbt: Use wallys function to check PST finalization status
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
1fe53880a7 psbt: use new wally functions to add PSBT inputs/outputs
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
68ffecba63 psbt: Use the newly exposed wally function to clone PSBTs
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
a51c6550ec psbt: avoid allocations when adding psbt outputs
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
499395636c psbt: remove now-unneeded const casts
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
21ee2c3a9c psbt: Remove workaround for now-fixed wally tx flag behaviour
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Jon Griffiths
95d3d65c62 wally: update to the latest wally version
Includes:
psbt: Use renamed functions for new wally version
psbt: Set the transaction directly to avoid script workarounds
psbt: Use low-S grinding when computing signatures
tx: Use wally_tx_clone from libwally now that its exported

Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-08-07 10:28:01 +09:30
Rusty Russell
06372e13d8 pyln.proto.message: don't let Message() init set implicit lengths.
We'll override them from field length anyway!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-06 17:53:25 +02:00
ZmnSCPxj jxPCSnmZ
4fde45669c plugins/pay.c: Fix the routehints/presplit conflict.
Changelog-Fixed: pay: Fixed a bug where routehints would be ignored if the payment exceeded 10,000 satoshi. This is particularly bad if the payee is only reachable via routehints in an invoice.
2020-08-06 17:36:58 +02:00
ZmnSCPxj jxPCSnmZ
a9b992a94a tests/test_pay.py: Add test for conflict between presplit and routehints paymods. 2020-08-06 17:36:58 +02:00
Sjors Provoost
2e10729690 Add missing test dependencies to requirements.txt
Supplements the consolidation in b68066e
2020-08-06 09:46:04 +09:30
Rusty Russell
fa829f23db amount: add amount_msat_scale, amount_msat_ratio, amount_{msat,sat}_div
It's not all that rare to do these operations, and requiring annotations
for it is a little painful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-06 09:36:47 +09:30
Rusty Russell
ffbb409b47 amount: use initializers everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-06 09:36:47 +09:30
Rusty Russell
36d43b871f amount: add amount_msat and amount_sat initializers.
Generally, importing amounts needn't be checked, and it cuts down on
the warnings we get.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-06 09:36:47 +09:30
niftynei
28b839f306 amount: add helper to convert u64 sats to amount_sat type 2020-08-06 09:36:47 +09:30
ZmnSCPxj jxPCSnmZ
1ded3fc52f lightningd/plugin.c: Add a --dev-builtin-plugins-unimportant for developers who want to mess around with the builtin plugins. 2020-08-04 13:27:51 -05:00
ZmnSCPxj jxPCSnmZ
1aa076845a tests/test_plugin.py: Test builtin plugins are important. 2020-08-04 13:27:51 -05:00
ZmnSCPxj jxPCSnmZ
4ca2e49812 lightningd/plugin.c: Make builtin plugins important.
Changelog-Changed: plugin: Builtin plugins are now marked as important, and if they crash, will cause C-lightning to stop as well.
2020-08-04 13:27:51 -05:00
ZmnSCPxj jxPCSnmZ
48f36904c8 tests/test_plugin.py: Add test for --important-plugin. 2020-08-04 13:27:51 -05:00
ZmnSCPxj jxPCSnmZ
a847487bbe lightningd/plugin.c: Add important plugins, which if they terminate, lightningd also terminates.
Changelog-Added: New option `--important-plugin` loads a plugin is so important that if it dies, `lightningd` will exit rather than continue.  You can still `--disable-plugin` it, however, which trumps `--important-plugin` and it will not be started at all.
2020-08-04 13:27:51 -05:00
ZmnSCPxj jxPCSnmZ
50600dce95 lightningd/lightningd.c: Create API to exit lightningd with an exit code. 2020-08-04 13:27:51 -05:00
ZmnSCPxj jxPCSnmZ
3df2333d5d lightningd/plugin.c: Add specific function to give the directory for built-in plugins. 2020-08-04 13:27:51 -05:00
Christian Decker
3dafddd717 channel: Base the channel forget timeout on the headercount
We were basing the 2016 block timeout on the blockchain height that we had
processed at the time completed the funding, which could lag considerably
behind the network-wide blockheight. For example this could happen if we
started with `--rescan` from a low height, taking some time to catch up.

This means that we could end up forgetting channels even before reaching the
network blockheight. This patch instead uses the headercount reported by the
backend plugin if we aren't caught up yet. While the chances of this happening
are still there, the window it might happen are much reduced, since headers
can be synced in a couple of minutes.

Reported-by: Riccardo Masutti
Changelog-Changed: Funding timeout is now based on the header count reported by the bitcoin backend instead of our current blockheight which might be lower.
2020-08-04 09:14:17 +09:30
ZmnSCPxj jxPCSnmZ
094eac4e95 plugins/libplugin-pay.c: Properly handle exclusions for routehints with two hops or more.
Arguably a low-priority bug since no current node ever generates routehints longer
than one hop.

However, it is possible as an edge case, if the destination is directly accessible
*and* supports multiple channels, that we route through the destination, one of the
*other* channels it has not in the routehint, to the entry point, and then through
the routehint.

This change removes the risk of the above edge case.

Changelog-None: arguably a low-priority bug.
2020-08-03 15:20:29 +09:30
Antoine Poinsot
3aad86ff9d pytest: test the blocksonly sanity checkin bcli
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-08-03 12:46:37 +09:30
Antoine Poinsot
5b45334d93 pyln: allow to signal may_fail in get_node()
This or `expect_fail`, as it implies that, well, the node may fail.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-08-03 12:46:37 +09:30
Antoine Poinsot
905730341a bcli: explicitly check at startup that bitcoind does relay transactions
We've had problems with blocksonly in the past and bitcoind still allows
to use outdated (thus potentially dangerous estimates) while running
bitcoin with -blocksonly.

ZmnSCPxj mentionned that we still don't document this, but i figured
that in this specific case an explicit check and error seems preferable.

Changelog-Added: We now explicitly check at startup that our default Bitcoin backend (bitcoind) does relay transactions.

Proposed-by: ZmnSCPxj <zmnscpxj@protonmail.com>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-08-03 12:46:37 +09:30
Antoine Poinsot
5be07c5fe3 bcli: poll getnetworkinfo at startup
This allows us to kill two birds with one stone: once connected, we use
the output of the successful call to do some sanity checks (only
checking the version for now, but more are yet to come!).

Changelog-Added: We now explicitly check at startup the version of our default Bitcoin backend (bitcoind).
Co-Authored-by: ZmnSCPxj <zmnscpxj@protonmail.com>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-08-03 12:46:37 +09:30
Christian Decker
a456d08ad0 pay: Be less aggressive when estimating channel capacity
We'd previously take the failed attempt and estimate the failing channel's
capacity at 3/4 of the attempted amount, which is rather aggressive. This
reduces this aggressiveness to use the exact amount tried, but excluding on
equality. This still skips attempting the same route with the same amount, but
also permits attempts that are in the range [3/4, 1] of the failed attempt
amount to still be attempted.
2020-08-03 12:15:08 +09:30