Commit Graph

8831 Commits

Author SHA1 Message Date
Christian Decker
2a176d2ef7 make: Enable mypy checks on pyln-testing 2020-09-28 09:19:46 +09:30
Christian Decker
c498f949cc pyln: Add mypy typing exceptions for external dependencies
We don't control them, and starting to write type stubs for them is a
different can of worms.
2020-09-28 09:19:46 +09:30
Christian Decker
e13c435aca make: Set $PYTHONPATH once
This is used a number of times all over the place, and there is no advantage
in setting it over and over again.
2020-09-28 09:19:46 +09:30
Christian Decker
2740f2eccb pyln: Use pyln-client Makefile for checks
Kinda uses an antipattern of `cd`ing into the directory and calling `make`
there, but this keeps the contrib dirs self-contained so we can split them out
eventually.
2020-09-28 09:19:46 +09:30
Christian Decker
1d2c2b6aaf pyln: Parametrize and unify Makefiles for pyln package 2020-09-28 09:19:46 +09:30
Michael Schmoock
71381eb5be chore: reduce log buffer from 100MB to 10MB
The 100MB log buffer has been the biggest memory footprint for the daemon.
Keeping 10MB for emergency log dumps seems sufficient.
This has been mentioned in the last developer meeting.

Changelog-Changed: In-memory log buffer reduced from 100MB to 10MB
2020-09-26 10:38:44 +09:30
Christian Decker
1afa8b88c9 travis: Re-enable the postgres configuration again 2020-09-26 09:04:35 +09:30
Christian Decker
5809cc41f4 pytest: Reduce the db_provider fixture scope to function
For performance reasons we were starting one for each session, which caused
the same postgres DB to be re-used for multiple tests (all test run in the
same worker process), but this could lead to interactions if there is a
timeout or a test happens to touch the `db_provider`. It turns out that we
were only saving about 15 seconds on a 1250 second run anyway, which is a
small cost for increased test isolation.
2020-09-26 09:04:35 +09:30
Christian Decker
7e99a3b5fd pyln: Allow the base test directory to contain some files
We were not removing the base test directory if we had other files in there,
which was the case for postgres runs. This now explicitly check for `test_*`
directories which are an indicator of a failed test.
2020-09-26 09:04:35 +09:30
Christian Decker
aeb5f969e2 pyln: Start each postgres DB in its own sub-directory
We had a couple of issues with workers dying and attempting to re-initialize
the database while it was already initialized. This will look for a free
directory and just start the DB in there, allowing workers to be better
isolated.
2020-09-26 09:04:35 +09:30
Christian Decker
85dd19c9ac travis: Reduce reruns to 2 and early failure on test failure
While continuing to test all tests is good, it also means we potentially wait
much longer to get the final verdict, and it can truncate logs if there are
too many failures.
2020-09-26 09:04:35 +09:30
Jan Sarenik
a777d21fb7 pytest: Rename all fund_channel to fundchannel 2020-09-25 21:04:56 +02:00
Rusty Russell
7260d9ea3d plugins: generate list of plugins more atomically, respect V=1 and --quiet.
I got a corrupt file, which looked like multiple concurrent attempts
to build it.  So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).

Also move into plugins/ where it logically belongs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 13:19:59 +02:00
Rusty Russell
488b32b003 build: run update-mocks.
Some declarations are redundant now.  Removing them does nothing, but
it makes other PRs cleaner.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 13:19:59 +02:00
Antoine Poinsot
3382dafaee configure: allow to run clang with ASAN
Prefer adding LSAN_OPTIONS="suppressions=$PWD/tests/sanitizer_suppressions/lsan" when CC=clang instead.

Changelog-Added: build: clang build now supports --enable-address-sanitizer .
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-24 13:57:43 +09:30
Christian Decker
5a87e88ee5 pytest: Add test for failcode conversion
We added a conversion of failcodes that do not have sufficient information in
faac4b28ad. That means that a failcode that'd require additional information
in order to be a correct error to return in an onion is mapped to a generic
one since we can't backfill the information.

This tests that the mapping is performed correctly and replicates the
situation in #4070
2020-09-24 11:23:33 +09:30
Michael Schmoock
980a9517c6 fix: openchannel_hook log BROKEN on dup close_to
Changelog-None
2020-09-24 11:22:47 +09:30
Christian Decker
a351b9bf2f pyln: Migrate from binascii.hexlify to bytes.hex
Suggested-by: Lisa <@niftynei>
2020-09-24 11:14:22 +09:30
Christian Decker
04462f6a64 pyln: Add code to unwrap an encrypted onion at the intended node
Changelog-Added: pyln-proto: Added pure python implementation of the sphinx onion creation and processing functionality.
2020-09-24 11:14:22 +09:30
Christian Decker
e8dcd59b24 pyln: Add a warning that pyln-proto is not safe for production use 2020-09-24 11:14:22 +09:30
Christian Decker
96b182a084 pyln: Implement sphinx onion packet generation in python
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
2020-09-24 11:14:22 +09:30
Jan Sarenik
0b5e6c5be1 nit: doc/lightningd-config.5: Remove redundancy 2020-09-24 11:06:32 +09:30
Jan Sarenik
b1355bf61e Update comment regarding signet using tb now 2020-09-24 09:24:14 +09:30
Dr. Maxim Orlovsky
0b01a5d7c4 Updating signet chainparams for the latest PoW fix
Changelog-Fixed: Protocol: `signet` is now compatible with the final bitcoin-core version
2020-09-24 09:24:14 +09:30
Rusty Russell
924cc04bd2 bolt11: have caller supply preferred chain.
This lets us distinguish testnet from signet invoices, since they
have the same prefix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 09:24:14 +09:30
Rusty Russell
1356700e2b bitcoin/chainparams: signet uses the same P2PKH and P2SH version bytes as testnet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-24 09:24:14 +09:30
Rusty Russell
d5cb0d85b5 utils: use a cleaner pattern to capture wally allocations.
We force use of tal_wally_start/tal_wally_end around every wally
allocation, and with "end" make the caller choose where to reparent
everything.

This is particularly powerful where we allocate a tx or a psbt: we
want that tx or psbt to be the parent of the other allocations, so
this way we can reparent the tx or psbt, then reparent everything
else onto it.

Implementing psbt_finalize (which uses a behavior flag antipattern)
was tricky, so I ended up splitting that into 'psbt_finalize' and
'psbt_final_tx', which I think also makes the callers clearer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:45:53 +02:00
Rusty Russell
66ffd7a73f bitcoin/: neaten use of libwally base58 and bas64 routines.
Now libwally uses tal, we don't need to make copies.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:45:53 +02:00
Rusty Russell
4e67b4fd03 Makefile: work with shasum -a as well as sha256sum.
At least on my Ubuntu box, they're compatible. If they're not, we need
to disable regeneration altogether.

Fixes: #4075
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 14:34:42 +02:00
Christian Decker
d9ba39d2cb pytest: Add test for key-value style postgresql DSNs 2020-09-23 20:20:40 +09:30
Christian Decker
1d9e7cf079 db: Add support for key-value pair DSNs in postgresql
These are simple space-separated key-value pair sets of options instead of the
URI style DSNs, but they are also much more flexible allowing the user to
specify client SSL certificates, server certificates, compression and
encryption levels, and much more (see [1] for more information)

[1]: https://www.postgresql.org/docs/9.1/libpq-connect.html

Changelog-Added: db: Added support for key-value DSNs for postgresql, allowing for a wider variety of configurations and environments.
2020-09-23 20:20:40 +09:30
Antoine Poinsot
15adcc915f Remove varint typedef for bigsize
It's not part of the spec anymore

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-23 16:30:36 +09:30
Rusty Russell
7db8680530 lightningd: don't hand around partially-assembled channel_info.
wallet_commit_channel would fill in the old_remote_per_commit and
fee_states, which is weird since the caller doesn't care.

Make the caller set all the channel_info fields, so wallet_commit_channel
is a simple consumer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 15:51:13 +09:30
Rusty Russell
58f6e316cc lightningd: separate fee_states out of struct channel_info.
It was always kind of weird in there anyway.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-23 15:51:13 +09:30
niftynei
cf2a807d3c dual_open: don't release uc here, it gets freed elsewhere 2020-09-23 15:51:13 +09:30
niftynei
db2b013990 dual-fund: call daemon_shutdown only
daemon_shutdown calls common_shutdown internally; calling both frees the
same context twice
2020-09-23 15:51:13 +09:30
niftynei
56b3711bbc dual_open: alignment nit 2020-09-23 15:51:13 +09:30
niftynei
ed9f661e07 dual_open: memleak problem with fee_states 2020-09-23 15:51:13 +09:30
niftynei
7d836a5945 dual_open_control: nit, reword comment 2020-09-23 15:51:13 +09:30
niftynei
aa6420cb58 dual_open: split logic for deserializing a changed vs signed PSBT
We need to run different checks on these, so split them out for finer
grained control
2020-09-23 15:51:13 +09:30
niftynei
1f165c00ae psbt_txid: it's possible a psbt may already have the finalized scriptsig
If we've already got a scriptSig field filled out for a PSBT input, we
use that instead of 'deriving' the scriptSig from the redeemscript
(finalizing a PSBT removes the redeemscript field)
2020-09-23 15:51:13 +09:30
niftynei
c5f748e716 dualopend: tiny billboard capitalization change 2020-09-23 15:51:13 +09:30
niftynei
5f6a2c05bd dualopend: check that we're not exceeding max allowable capacity 2020-09-23 15:51:13 +09:30
niftynei
8761468f42 dualopend: use status failed instead of peer failed 2020-09-23 15:51:13 +09:30
Christian Decker
668debb74f pyln: Rename LightningNode.fund_channel to fundchannel
It is inconsistent with the RPC method, which was bothering me for some time.
2020-09-23 14:45:12 +09:30
Christian Decker
8ecb157af6 pyln: Add type annotations to lightning.py 2020-09-23 14:45:12 +09:30
Christian Decker
49ec800a07 pyln: Add type-annotations to plugin.py
This should help users that have type-checking enabled.
2020-09-23 14:45:12 +09:30
Christian Decker
d27da4d152 pyln: Fix two formatting string not matching parameters 2020-09-23 14:45:12 +09:30
Christian Decker
748caf91d3 pyln-client: Add support to monkey patch the JSONEncoder
Several times we had issues with plugins not being able to re-encode an RPC
result because they forgot to use the custom encoder class. This allows us to
patch the JSONEncoder when we start the RPC or the plugin and automagically
support classes that provide a `to_json` method.
2020-09-23 14:45:12 +09:30
Christian Decker
382230509b pyln: Add stubs to generate documentation for pyln-client 2020-09-23 14:45:12 +09:30