Commit Graph

8739 Commits

Author SHA1 Message Date
Christian Decker
81569de709 pytest: Stabilize test_setchannelfee_{restart,zero}
In both cases the flakyness arises from the destination not knowing about the
modified fees of the forwarding node, thus including the outdated details in
the routehint, and the sender being unlucky and always trying with the
routehint anyway.

The long-term solutions to this is going to be #4111, this commit just reduces
the flakyness to get back to business.
2020-10-07 09:36:30 +10:30
Christian Decker
0de7a289cc travis: Add blinker dependency
This is a new, not explicitly mentioned, dependency for pytest-sentry
2020-10-07 09:36:30 +10:30
Christian Decker
b6053d7cd6 Fix pytest major version
They released version 6, which is causing some trouble apparently. So fix it
to v5.
2020-10-07 09:36:30 +10:30
Christian Decker
66dc3ed665 pyln: Add pytest to type ignores
Reported-by: Rusty Russell <@rustyrussell>
2020-10-07 09:36:30 +10:30
Christian Decker
8d8b807793 pytest: Make LightningNode.fund_channel more resilient
It was really flaky, especially under `test_mpp_interference_2`, most likely
due to multiple calls to `fund_channel`. This commit looks for the specific
txids in the `listfunds` output and the `getrawmempool` output, avoiding
strange artifacts from multiple calls.
2020-10-07 09:36:30 +10:30
Christian Decker
1563bbc2fa pytest: Cleanup test_gossip_ratelimit
Drive-by code cleanups I stumbled over while investigating the issue fixed in
the previous commit.
2020-10-07 09:36:30 +10:30
Christian Decker
05934724dd pytest: Don't start 3 nodes in test_gossip_ratelimit
We really are just interested in their on-chain footprint, so actually
starting the nodes is pointless overhead, and caused a lot of flakyness due to
the output ordering sometimes not matching up. We now just use the `bitcoind`
API to fund, sign and send a raw transaction that matches the stashed gossip
messages.
2020-10-07 09:36:30 +10:30
Christian Decker
cafaad741b pyln: Add type-annotations to pyln-proto 2020-09-28 09:19:46 +09:30
Christian Decker
2a176d2ef7 make: Enable mypy checks on pyln-testing 2020-09-28 09:19:46 +09:30
Christian Decker
c498f949cc pyln: Add mypy typing exceptions for external dependencies
We don't control them, and starting to write type stubs for them is a
different can of worms.
2020-09-28 09:19:46 +09:30
Christian Decker
e13c435aca make: Set $PYTHONPATH once
This is used a number of times all over the place, and there is no advantage
in setting it over and over again.
2020-09-28 09:19:46 +09:30
Christian Decker
2740f2eccb pyln: Use pyln-client Makefile for checks
Kinda uses an antipattern of `cd`ing into the directory and calling `make`
there, but this keeps the contrib dirs self-contained so we can split them out
eventually.
2020-09-28 09:19:46 +09:30
Christian Decker
1d2c2b6aaf pyln: Parametrize and unify Makefiles for pyln package 2020-09-28 09:19:46 +09:30
Michael Schmoock
71381eb5be chore: reduce log buffer from 100MB to 10MB
The 100MB log buffer has been the biggest memory footprint for the daemon.
Keeping 10MB for emergency log dumps seems sufficient.
This has been mentioned in the last developer meeting.

Changelog-Changed: In-memory log buffer reduced from 100MB to 10MB
2020-09-26 10:38:44 +09:30
Christian Decker
1afa8b88c9 travis: Re-enable the postgres configuration again 2020-09-26 09:04:35 +09:30
Christian Decker
5809cc41f4 pytest: Reduce the db_provider fixture scope to function
For performance reasons we were starting one for each session, which caused
the same postgres DB to be re-used for multiple tests (all test run in the
same worker process), but this could lead to interactions if there is a
timeout or a test happens to touch the `db_provider`. It turns out that we
were only saving about 15 seconds on a 1250 second run anyway, which is a
small cost for increased test isolation.
2020-09-26 09:04:35 +09:30
Christian Decker
7e99a3b5fd pyln: Allow the base test directory to contain some files
We were not removing the base test directory if we had other files in there,
which was the case for postgres runs. This now explicitly check for `test_*`
directories which are an indicator of a failed test.
2020-09-26 09:04:35 +09:30
Christian Decker
aeb5f969e2 pyln: Start each postgres DB in its own sub-directory
We had a couple of issues with workers dying and attempting to re-initialize
the database while it was already initialized. This will look for a free
directory and just start the DB in there, allowing workers to be better
isolated.
2020-09-26 09:04:35 +09:30
Christian Decker
85dd19c9ac travis: Reduce reruns to 2 and early failure on test failure
While continuing to test all tests is good, it also means we potentially wait
much longer to get the final verdict, and it can truncate logs if there are
too many failures.
2020-09-26 09:04:35 +09:30
Jan Sarenik
a777d21fb7 pytest: Rename all fund_channel to fundchannel 2020-09-25 21:04:56 +02:00
Rusty Russell
7260d9ea3d plugins: generate list of plugins more atomically, respect V=1 and --quiet.
I got a corrupt file, which looked like multiple concurrent attempts
to build it.  So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).

Also move into plugins/ where it logically belongs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 13:19:59 +02:00
Rusty Russell
488b32b003 build: run update-mocks.
Some declarations are redundant now.  Removing them does nothing, but
it makes other PRs cleaner.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 13:19:59 +02:00
Antoine Poinsot
3382dafaee configure: allow to run clang with ASAN
Prefer adding LSAN_OPTIONS="suppressions=$PWD/tests/sanitizer_suppressions/lsan" when CC=clang instead.

Changelog-Added: build: clang build now supports --enable-address-sanitizer .
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-24 13:57:43 +09:30
Christian Decker
5a87e88ee5 pytest: Add test for failcode conversion
We added a conversion of failcodes that do not have sufficient information in
faac4b28ad. That means that a failcode that'd require additional information
in order to be a correct error to return in an onion is mapped to a generic
one since we can't backfill the information.

This tests that the mapping is performed correctly and replicates the
situation in #4070
2020-09-24 11:23:33 +09:30
Michael Schmoock
980a9517c6 fix: openchannel_hook log BROKEN on dup close_to
Changelog-None
2020-09-24 11:22:47 +09:30
Christian Decker
a351b9bf2f pyln: Migrate from binascii.hexlify to bytes.hex
Suggested-by: Lisa <@niftynei>
2020-09-24 11:14:22 +09:30
Christian Decker
04462f6a64 pyln: Add code to unwrap an encrypted onion at the intended node
Changelog-Added: pyln-proto: Added pure python implementation of the sphinx onion creation and processing functionality.
2020-09-24 11:14:22 +09:30
Christian Decker
e8dcd59b24 pyln: Add a warning that pyln-proto is not safe for production use 2020-09-24 11:14:22 +09:30
Christian Decker
96b182a084 pyln: Implement sphinx onion packet generation in python
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2020-09-24 11:14:22 +09:30
Jan Sarenik
0b5e6c5be1 nit: doc/lightningd-config.5: Remove redundancy 2020-09-24 11:06:32 +09:30
Jan Sarenik
b1355bf61e Update comment regarding signet using tb now 2020-09-24 09:24:14 +09:30
Dr. Maxim Orlovsky
0b01a5d7c4 Updating signet chainparams for the latest PoW fix
Changelog-Fixed: Protocol: `signet` is now compatible with the final bitcoin-core version
2020-09-24 09:24:14 +09:30
Rusty Russell
924cc04bd2 bolt11: have caller supply preferred chain.
This lets us distinguish testnet from signet invoices, since they
have the same prefix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 09:24:14 +09:30
Rusty Russell
1356700e2b bitcoin/chainparams: signet uses the same P2PKH and P2SH version bytes as testnet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 09:24:14 +09:30
Rusty Russell
d5cb0d85b5 utils: use a cleaner pattern to capture wally allocations.
We force use of tal_wally_start/tal_wally_end around every wally
allocation, and with "end" make the caller choose where to reparent
everything.

This is particularly powerful where we allocate a tx or a psbt: we
want that tx or psbt to be the parent of the other allocations, so
this way we can reparent the tx or psbt, then reparent everything
else onto it.

Implementing psbt_finalize (which uses a behavior flag antipattern)
was tricky, so I ended up splitting that into 'psbt_finalize' and
'psbt_final_tx', which I think also makes the callers clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:45:53 +02:00
Rusty Russell
66ffd7a73f bitcoin/: neaten use of libwally base58 and bas64 routines.
Now libwally uses tal, we don't need to make copies.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:45:53 +02:00
Rusty Russell
4e67b4fd03 Makefile: work with shasum -a as well as sha256sum.
At least on my Ubuntu box, they're compatible. If they're not, we need
to disable regeneration altogether.

Fixes: #4075
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:34:42 +02:00
Christian Decker
d9ba39d2cb pytest: Add test for key-value style postgresql DSNs 2020-09-23 20:20:40 +09:30
Christian Decker
1d9e7cf079 db: Add support for key-value pair DSNs in postgresql
These are simple space-separated key-value pair sets of options instead of the
URI style DSNs, but they are also much more flexible allowing the user to
specify client SSL certificates, server certificates, compression and
encryption levels, and much more (see [1] for more information)

[1]: https://www.postgresql.org/docs/9.1/libpq-connect.html

Changelog-Added: db: Added support for key-value DSNs for postgresql, allowing for a wider variety of configurations and environments.
2020-09-23 20:20:40 +09:30
Antoine Poinsot
15adcc915f Remove varint typedef for bigsize
It's not part of the spec anymore

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-23 16:30:36 +09:30
Rusty Russell
7db8680530 lightningd: don't hand around partially-assembled channel_info.
wallet_commit_channel would fill in the old_remote_per_commit and
fee_states, which is weird since the caller doesn't care.

Make the caller set all the channel_info fields, so wallet_commit_channel
is a simple consumer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 15:51:13 +09:30
Rusty Russell
58f6e316cc lightningd: separate fee_states out of struct channel_info.
It was always kind of weird in there anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 15:51:13 +09:30
niftynei
cf2a807d3c dual_open: don't release uc here, it gets freed elsewhere 2020-09-23 15:51:13 +09:30
niftynei
db2b013990 dual-fund: call daemon_shutdown only
daemon_shutdown calls common_shutdown internally; calling both frees the
same context twice
2020-09-23 15:51:13 +09:30
niftynei
56b3711bbc dual_open: alignment nit 2020-09-23 15:51:13 +09:30
niftynei
ed9f661e07 dual_open: memleak problem with fee_states 2020-09-23 15:51:13 +09:30
niftynei
7d836a5945 dual_open_control: nit, reword comment 2020-09-23 15:51:13 +09:30
niftynei
aa6420cb58 dual_open: split logic for deserializing a changed vs signed PSBT
We need to run different checks on these, so split them out for finer
grained control
2020-09-23 15:51:13 +09:30
niftynei
1f165c00ae psbt_txid: it's possible a psbt may already have the finalized scriptsig
If we've already got a scriptSig field filled out for a PSBT input, we
use that instead of 'deriving' the scriptSig from the redeemscript
(finalizing a PSBT removes the redeemscript field)
2020-09-23 15:51:13 +09:30
niftynei
c5f748e716 dualopend: tiny billboard capitalization change 2020-09-23 15:51:13 +09:30