Commit Graph

513 Commits

Author SHA1 Message Date
Michael Schmoock
c8198d9820 pyln-testing: add scids param to wait_for_htlcs 2021-02-04 12:02:15 +10:30
Michael Schmoock
d76ca6ed35 pyln-testing: improve wait_for a bit
This 'fixes' the `wait_for` helper by removing a pointless final
`time.sleep()`, thus potentially making the method return quicker.

The old code could have had three final states:
 - success() := True
 - Timeout and success() := True
 - Timeout and success() := False

The new code has just two final state:
 - success() := True
 - Timeout and success() := False

It ensures the final `time.sleep()` is just the right amount before timeout.
And more importantly making it more readable :-)

Changelog-None
2021-02-04 12:02:15 +10:30
Rusty Russell
3c5502426b lightningd: addgossip API to inject gossip messages.
Importantly, this is synchronous, so pay will be able to use it
reliably.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-02 13:44:01 +01:00
Christian Decker
a5f16ab5b1 pyln: Catch OSError when cleaning up test directories 2021-01-29 10:29:09 +10:30
Christian Decker
ea67710e01 pyln: Pretty print RPC calls in the testing framework
We are printing `repr(obj)` which is not pretty-printed, hard to read,
and can't even be copied and inspected to JSON tools. We now print the
JSONified and indented calls and responses for easier debugging based
on solely the logs (useful for CI!).

Changelog-Added: pyln-testing: The RPC client will now pretty-print requests and responses to facilitate log-based debugging.
2021-01-29 10:29:09 +10:30
Christian Decker
cd9aa267b4 pyln: Adjust maximum load allowed by the throttler 2021-01-29 10:29:09 +10:30
Christian Decker
4c3ee04bb7 pyln: Use a fair FS lock to throttle node startups
We were getting a couple of starvations, so we need a fair filelock. I
also wasn't too happy with the lock as is, so I hand-coded it quickly.

Should be correct, but the overall timeout will tell us how well we
are doing on CI.
2021-01-29 10:29:09 +10:30
Christian Decker
fc677e331a pyln: Update dependencies for all pyln packages
We were getting a number of incompatibility warning due to the
dependencies being expressed too rigidly. This losens the requirement
definitions to being compatible with a known good version, and while
we're at it we also bump all outdated requirements.
2021-01-29 10:29:09 +10:30
Christian Decker
b447944285 gci: Add a tester Dockerfile 2021-01-29 10:29:09 +10:30
Christian Decker
6bed85bac2 pyln: Bump pyln version to 0.9.3 before publishing on PyPI
Let's just keep pyln synched up with the c-lightning version
2021-01-27 11:29:56 +01:00
Christian Decker
483579f8b6 pyln: Pin the mypy dependency to 0.790 since 0.800 has a regression 2021-01-26 15:45:13 +01:00
nicolas.dorier
3105a7a204 Fix dockerfiles 2021-01-26 15:03:35 +01:00
niftynei
ea1895fc1e pyln-tests: tweak fundbalancedchannel to assume peer will match
we got rid of push_msats for dual funded channels. this assumes that hte
peer will match an equal amount of sats as ours (the df_accepter.py
plugin will do this)
2021-01-10 13:44:04 +01:00
niftynei
f4b7904ce5 tests: don't enforce daemon name on opening logline 2021-01-10 13:44:04 +01:00
niftynei
3c1d90086f pyln-testing: add 'config' method to a node
Query for a config's value. If not set, returns 'none'
2021-01-10 13:44:04 +01:00
Christian Decker
a91254de11 pyln: Add ammag key to onion keyset
This was missing, and is required to wrap error responses.
2021-01-08 19:28:30 +01:00
Rusty Russell
1a8978100e invoice: add ctlv option.
This is required if we want to create a "bouncer" plugin (in my copious free time!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now takes an optional `cltv` parameter.
2021-01-08 10:45:56 +01:00
Christian Decker
b6650425b9 pyln: Add notifications support to LightningRpc
Changelog-Added: pyln-client: Added support for command notifications to LightningRpc via the `notify` context-manager.
2021-01-07 08:32:40 +01:00
Christian Decker
84b3653606 pyln: Add command notification support for plugins
Changelog-Added: pyln-client: Plugin methods can now report progress or status via the `Request.notify` function
2021-01-07 08:32:40 +01:00
Christian Decker
a1d33c3781 pytest: Add a test for listinvoices
We now have some more logic in the query, so let's test it
exhaustively.
2021-01-06 18:48:06 +01:00
Rusty Russell
d41b383cf7 pyln-testing: suppress mypy on FileLock
3.0.12 here:
```
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-06 12:15:39 +01:00
niftynei
e3a1d1a7f3 pyln-client: to_whole_satoshi returns the rounded up satoshi value
A fractional satoshi value isn't really useful; rounding up loses
precision but that's why you called "whole satoshi", wasn't it?

Changelog-Changed: pyln-client: Millisatoshi has new method, `to_whole_satoshi`; *rounds value up* to the nearest whole satoshi
2021-01-05 19:12:00 +01:00
niftynei
cce0020568 pyln-client: add missing min_witness_weight to fundpsbt/utxopsbt
We need to use it for the 'df_accepter' plugin, so we get the feerate
correct.

Changelog-Added: pyln-client: `fundpsbt`/`utxopsbt` now support `min_witness_weight` param
2021-01-05 19:12:00 +01:00
Christian Decker
dbb83a1bc3 pyln: Do not rstrip() the return value of .append()
This was causing the following error

```
Exception in thread Thread-553:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/cirrus-ci-build/contrib/pyln-testing/pyln/testing/utils.py", line 232, in tail
    self.err_logs.append(line.rstrip().decode('UTF-8', 'replace')).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'

[gw5] [ 33%] FAILED tests/test_misc.py::test_bitcoin_failure
```

Notice the second call to `.rstrip()` on the return value of `.append()`
2021-01-04 11:25:50 +01:00
Christian Decker
1ff57f07f8 pyln: Check that the stderr line is not none in pyln-testing tail
I stumbled over this in a test run and it seems benign.
2021-01-04 11:25:50 +01:00
Christian Decker
7e867e5ee6 pytest: Add throttler to limit load on the test system
Both my machine and apparently the CI tester machines regularly run
into issues with load on the system, causing timeouts (and
unresponsiveness). The throttler throttles the speed with which new
instances of c-lightning get started to avoid overloading. Since the
plugin used for parallelism when testing spawns multiple processes we
need to lock on the fs. Since we have that file open already, we'll
also write a couple of performance metics to it.
2021-01-04 11:25:50 +01:00
Christian Decker
f2a0a4abfc pyln: Add logging to bitcoind RPC calls
Useful if we want to debug a bit better
2021-01-04 11:25:50 +01:00
Christian Decker
8b09ac38aa pyln: Log mempool when we generate blocks in BitcoinD fixture
We sometimes have very specific sequences of tx broadcasts and blocks
being generated to confirm them. If the confirmation is missed the
test can completely get out of sync. Make debugging this easier by
logging what we confirmed.
2021-01-04 11:25:50 +01:00
Karol Hosiawa
4a3dec660f Added spent option to listfunds in pyln-client 2020-12-22 13:00:02 +01:00
Rusty Russell
77478408f9 pyln: add RpcException for finer method failure control.
Allows caller to set code and exact message to be returned.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln-client: plugins can now raise RpcException for finer control over error returns.
2020-12-16 12:37:14 +01:00
Rusty Russell
d971e3de98 Plugin: support extra args to "start".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `start` command can now take plugin-specific parameters.
2020-12-15 09:28:56 +10:30
Rusty Russell
8a9976c4c1 plugins: support concatenation of multiple args.
"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
2020-12-15 09:28:56 +10:30
Michael Schmoock
0a01111395 pytest: check millisatoshi by float raises
This only adds a test that currently makes sure its not possible
to init a Millisatoshi by a floating number.

Discussion:
As @gallizoltan points out, initialization with a float should be possible:
https://github.com/ElementsProject/lightning/pull/4273#discussion_r540369093

> Millisatoshi(5) / 2 currently works, and removes the half msat.
> So, I think Millisatoshi(5 / 2) should be the same.
2020-12-14 17:11:04 +01:00
Michael Schmoock
bd13726db7 pytest: test mul and div units for Millisatoshi 2020-12-14 17:11:04 +01:00
Michael Schmoock
fffc343dd7 pyln: fix Millisatoshi div with msat itself
Before this patch this fails: `Millisatoshi(42) / Millisatoshi(2)`
This is an operation that should return the ratio between the two
operands as a float number. Same goes for __floordiv__ operator `//`.

Changelog-None
2020-12-14 17:11:04 +01:00
Christian Decker
f7cdf1dd98 pytest: Debug test_lockup_drain 2020-12-11 16:55:55 -06:00
Christian Decker
68d08fc7d7 pyln: Add TOR and SOCKS5 support in pyln.proto.wire.connect
I wanted to talk to TOR-based nodes, so here comes TOR support :-)
2020-12-11 11:39:18 +10:30
Christian Decker
eacc54646f pyln: Remove deprecated txprepare variant
We promised to deprecate the old "destination satoshi feerate utxos"
variant some time on mid-2020 so let's do just that. `txprepare`
doesn't parse it either anymore.

Changelog-Changed: pyln: `txprepare` no longer supports the deprecated `destination satoshi feerate utxos` call format.
2020-12-09 06:56:21 +10:30
Michael Schmoock
362284981d plyn: use math.floor for msat mul and div 2020-12-08 13:48:14 +01:00
Michael Schmoock
565dc95a8e pyln: fix msat float multiplication 2020-12-08 13:48:14 +01:00
Michael Schmoock
2678cfb364 pyln: extend msat floating testcases
This adds two more xfail'ing testcases to show that the current way of
parsing Millisatoshi decimals is not yet optimal.

Changelog-None
2020-12-08 13:48:14 +01:00
Christian Decker
76124eb800 pyln: Replace undecodeable symbols when tailing logs
Logs may contain non-ASCII and non-UTF8 symbols, which crashes the
tailer. It's better to replace them with a glyph representing
undecodeable symbols instead, and handle the issue further up the
call-chain.
2020-12-04 18:34:12 -06:00
Christian Decker
c9d24b1de2 pyln: Set the prefix in TailableProc
We rely on it when logging, but didn't set it in the class, rather we
did in the sub-classes. This proved to be rather annoying to search
since it'd just fail silently and not log any output.
2020-12-04 18:34:12 -06:00
Rusty Russell
e4950db9a3 pytest: recreate wallet on bitcoind restart.
Doesn't seem to stick in master.  Andy Chow suggested we
simply turn off wallet on older versions, and always create/load.

```
[gw8] [ 40%] FAILED tests/test_misc.py::test_bitcoind_goes_backwards

============================================================= FAILURES ==============================================================
___________________________________________________ test_bitcoind_goes_backwards ____________________________________________________
[gw8] linux -- Python 3.8.5 /usr/bin/python3

node_factory = <pyln.testing.utils.NodeFactory object at 0x7f931859a760>
bitcoind = <pyln.testing.utils.BitcoinD object at 0x7f931865eee0>

    def test_bitcoind_goes_backwards(node_factory, bitcoind):
        """Check that we refuse to acknowledge bitcoind giving a shorter chain without explicit rescan"""
        l1 = node_factory.get_node(may_fail=True, allow_broken_log=True)

        bitcoind.generate_block(10)
        sync_blockheight(bitcoind, [l1])
        l1.stop()

        # Now shrink chain (invalidateblock leaves 'headers' field until restart)
        bitcoind.rpc.invalidateblock(bitcoind.rpc.getblockhash(105))
        # Restart without killing proxies
        bitcoind.rpc.stop()
        TailableProc.stop(bitcoind)
        bitcoind.start()

        # Will simply refuse to start.
        with pytest.raises(ValueError):
            l1.start()

        # Nor will it start with if we ask for a reindex of fewer blocks.
        l1.daemon.opts['rescan'] = 3

        with pytest.raises(ValueError):
            l1.start()

        # This will force it, however.
        l1.daemon.opts['rescan'] = -100
        l1.start()

        # Now mess with bitcoind at runtime.
>       bitcoind.generate_block(6)

tests/test_misc.py:1307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:399: in generate_block
    return self.rpc.generatetoaddress(numblocks, self.rpc.getnewaddress())
contrib/pyln-testing/pyln/testing/utils.py:322: in f
    return proxy._call(name, *args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bitcoin.rpc.RawProxy object at 0x7f93184f6a30>, service_name = 'getnewaddress', args = ()
postdata = '{"version": "1.1", "method": "getnewaddress", "params": [], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -18, 'message': 'No wallet is loaded. Load a wallet using loadwallet or create a new one with createwallet. (Note: A default wallet is no longer automatically created)'}, 'id': 1, 'result': None}
```
2020-12-03 11:35:30 +01:00
Michael Schmoock
7bfb5f10c7 pyln: failing test msat from float str
We were not able to create pyln Millisatoshi from floats, e.g.:
 - "0.01btc"
 - "0.1sat"
 - ...

This adds a test that makes sure this won't happen again.
2020-12-02 10:37:31 +10:30
Michael Schmoock
83a21138b8 pyln: fix msat from float str
Changelog-fixed: pyln: parsing msat from a float string
2020-12-02 10:37:31 +10:30
Michael Schmoock
bdad9fcdd9 pyln: def amount 10**6 for fund- and openchannel
Changelog-None
2020-12-01 11:15:30 +10:30
Antoine Poinsot
a561360334 pytest: create a wallet for bitcoind if none is present
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Rusty Russell
fa006fd2f7 pyln-testing: require bitcoin v0.20 for PSBT handling, and create wallet for v0.21
With older bitcoind, PSBTs fail:

```
    def test_utxopsbt(node_factory, bitcoind, chainparams):
...
>       psbt = bitcoind.rpc.decodepsbt(funding['psbt'])

tests/test_wallet.py:561:
...
self = <bitcoin.rpc.RawProxy object at 0x7f4ec602e100>, service_name = 'decodepsbt'
args = ('cHNidP8BADMCAAAAAaoMihSVXlpdBHGcJePiroqtwq/b1zu09j8IkTG4OKs7AQAAAAD9////AGYAAAAAAQDeAgAAAAABAefqB6BkZE1/AqXaf36T02a7.../7Stf971PEgvUXgvASECXPTIO6tIVxDih6tfKy6suj6WJhhjycwoaTeuso/AQ8llAAAAAQEfQEIPAAAAAAAWABQB+tkKvNZml+JZIWRyLeSpXr7hZQA=',)
postdata = '{"version": "1.1", "method": "decodepsbt", "params": ["cHNidP8BADMCAAAAAaoMihSVXlpdBHGcJePiroqtwq/b1zu09j8IkTG4OKs7AQ...gvUXgvASECXPTIO6tIVxDih6tfKy6suj6WJhhjycwoaTeuso/AQ8llAAAAAQEfQEIPAAAAAAAWABQB+tkKvNZml+JZIWRyLeSpXr7hZQA="], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -22, 'message': 'TX decode failed PSBT is not sane.: iostream error'}, 'id': 1, 'result': None}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: *** Requires bitcoind v0.20.1 or above ***
2020-11-09 20:05:30 -06:00
Rusty Russell
e16ed0e207 pyln: add support for dependent hooks.
And use that to add simple tests.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-09 15:22:33 -06:00