In the case where you want a PSBT and also want the output to be added
as a change address, use `excess_as_change` = true.
Generates a change address to use. If you want to pay the excess
elsewhere, you will have to add separately.
Changelog-Added: JSON-RPC: Add new parameter `excess_as_change` to fundpsbt+utxopsbt
Since we turned many errors into warnings, we want our tests to fail
when they happen unexpectedly. We make WARNING clear in the strings
we print, too, to help out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This 'fixes' the `wait_for` helper by removing a pointless final
`time.sleep()`, thus potentially making the method return quicker.
The old code could have had three final states:
- success() := True
- Timeout and success() := True
- Timeout and success() := False
The new code has just two final state:
- success() := True
- Timeout and success() := False
It ensures the final `time.sleep()` is just the right amount before timeout.
And more importantly making it more readable :-)
Changelog-None
We are printing `repr(obj)` which is not pretty-printed, hard to read,
and can't even be copied and inspected to JSON tools. We now print the
JSONified and indented calls and responses for easier debugging based
on solely the logs (useful for CI!).
Changelog-Added: pyln-testing: The RPC client will now pretty-print requests and responses to facilitate log-based debugging.
We were getting a couple of starvations, so we need a fair filelock. I
also wasn't too happy with the lock as is, so I hand-coded it quickly.
Should be correct, but the overall timeout will tell us how well we
are doing on CI.
We were getting a number of incompatibility warning due to the
dependencies being expressed too rigidly. This losens the requirement
definitions to being compatible with a known good version, and while
we're at it we also bump all outdated requirements.
we got rid of push_msats for dual funded channels. this assumes that hte
peer will match an equal amount of sats as ours (the df_accepter.py
plugin will do this)
This is required if we want to create a "bouncer" plugin (in my copious free time!)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `invoice` now takes an optional `cltv` parameter.
3.0.12 here:
```
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: error: Cannot find implementation or library stub for module named 'filelock'
pyln/testing/utils.py:9: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A fractional satoshi value isn't really useful; rounding up loses
precision but that's why you called "whole satoshi", wasn't it?
Changelog-Changed: pyln-client: Millisatoshi has new method, `to_whole_satoshi`; *rounds value up* to the nearest whole satoshi
We need to use it for the 'df_accepter' plugin, so we get the feerate
correct.
Changelog-Added: pyln-client: `fundpsbt`/`utxopsbt` now support `min_witness_weight` param
This was causing the following error
```
Exception in thread Thread-553:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/tmp/cirrus-ci-build/contrib/pyln-testing/pyln/testing/utils.py", line 232, in tail
self.err_logs.append(line.rstrip().decode('UTF-8', 'replace')).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'
[gw5] [ 33%] FAILED tests/test_misc.py::test_bitcoin_failure
```
Notice the second call to `.rstrip()` on the return value of `.append()`
Both my machine and apparently the CI tester machines regularly run
into issues with load on the system, causing timeouts (and
unresponsiveness). The throttler throttles the speed with which new
instances of c-lightning get started to avoid overloading. Since the
plugin used for parallelism when testing spawns multiple processes we
need to lock on the fs. Since we have that file open already, we'll
also write a couple of performance metics to it.
We sometimes have very specific sequences of tx broadcasts and blocks
being generated to confirm them. If the confirmation is missed the
test can completely get out of sync. Make debugging this easier by
logging what we confirmed.
Allows caller to set code and exact message to be returned.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln-client: plugins can now raise RpcException for finer control over error returns.
"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
This only adds a test that currently makes sure its not possible
to init a Millisatoshi by a floating number.
Discussion:
As @gallizoltan points out, initialization with a float should be possible:
https://github.com/ElementsProject/lightning/pull/4273#discussion_r540369093
> Millisatoshi(5) / 2 currently works, and removes the half msat.
> So, I think Millisatoshi(5 / 2) should be the same.
Before this patch this fails: `Millisatoshi(42) / Millisatoshi(2)`
This is an operation that should return the ratio between the two
operands as a float number. Same goes for __floordiv__ operator `//`.
Changelog-None
We promised to deprecate the old "destination satoshi feerate utxos"
variant some time on mid-2020 so let's do just that. `txprepare`
doesn't parse it either anymore.
Changelog-Changed: pyln: `txprepare` no longer supports the deprecated `destination satoshi feerate utxos` call format.
Logs may contain non-ASCII and non-UTF8 symbols, which crashes the
tailer. It's better to replace them with a glyph representing
undecodeable symbols instead, and handle the issue further up the
call-chain.
We rely on it when logging, but didn't set it in the class, rather we
did in the sub-classes. This proved to be rather annoying to search
since it'd just fail silently and not log any output.
This will change the current hardwired PostgreSQL binary lookup mechanism to
utilizing the `pg_config --bindir` utility that should work better for most
distributions. The old method caused issues at least on Archlinux.
Changelog-None
We also sanity check that response id matches our request.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: pyln.client handles and can send progress notifications.
- Adds bitarray filling so mesages of any length can be encoded, instead of forcing the encoding to be of messages with length multiple of 5.
- Adds checks for encoding / decoding and raises expections if the inputs are not as expected.
- Flags functions that are supposed to be internal as "private".
My bad, i comitted them as part of the LSAN suppressions while this data
race could have and had been fixed.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
`openchannel_signed` and `openchannel_update` which allow a user to
continue a openchannel or kick off the completion of a openchannel.
`openchannel_update` should be called until it returns with
`commitments_secured`.
Finally, extends the 'close_to' functionality up to the flagship 'open a
channel' command.
Changelog-Added: JSON-API `fundchannel` now accepts an optional 'close_to' param, a bitcoin address that the channel funding should be sent to on close. Requires `opt_upfront_shutdownscript`
We had a couple of instances where a plugin would be killed by `lightningd`
because we were returning a result of an exception twice, and it was hard to
trace down the logic error in the user plugin that caused that. This patch
adds a traceback the first time we return a result/exception, and raise an
exception with a stacktrace of the first termination when a second one comes
in.
This can still terminate the plugin, but the programmer gets a clear
indication where the result was set, and can potentially even recover from it.
Changelog-Added: pyln: Plugin method and hook requests prevent the plugin developer from accidentally setting the result multiple times, and will raise an exception detailing where the result was first set.
It is often pretty usefuk to use the builtin logging module to debug things,
including libraries that a plugin may use. This adds a simple
`PluginLogHandler` that maps the python logging levels to the `lightningd`
logging levels, and formats the record in a way that it doesn't clutter up the
`lightningd` logs (no duplicate timestamps and levels).
This allow us to tweak the log level that is reported to `lightningd` simply
using the following
```python3
import logging
logging.basicConfig(level=logging.DEBUG)
```
Notice that in order for the logs to be displayed on the terminal or the
logfile, both the logging level in the plugin _and_ the `--log-level`
`lightningd` is running need to be adjusted (the python logging level only
controls which messages get forwarded to `lightningd`, it does not have the
power to overrule `lightningd` about what to actually display).
I chose `logging.INFO` as the default, since libraries have a tendency to spew
out everything in `logging.DEBUG` mode
Changelog-Added: pyln: Plugins have been integrated with the `logging` module for easier debugging and error reporting.
Since we start a new instance of postgres for each test we may end up swamped
and the startup can take a bit longer. So let's loop until we get a success.
It was really flaky, especially under `test_mpp_interference_2`, most likely
due to multiple calls to `fund_channel`. This commit looks for the specific
txids in the `listfunds` output and the `getrawmempool` output, avoiding
strange artifacts from multiple calls.
For performance reasons we were starting one for each session, which caused
the same postgres DB to be re-used for multiple tests (all test run in the
same worker process), but this could lead to interactions if there is a
timeout or a test happens to touch the `db_provider`. It turns out that we
were only saving about 15 seconds on a 1250 second run anyway, which is a
small cost for increased test isolation.
We were not removing the base test directory if we had other files in there,
which was the case for postgres runs. This now explicitly check for `test_*`
directories which are an indicator of a failed test.
We had a couple of issues with workers dying and attempting to re-initialize
the database while it was already initialized. This will look for a free
directory and just start the DB in there, allowing workers to be better
isolated.
Several times we had issues with plugins not being able to re-encode an RPC
result because they forgot to use the custom encoder class. This allows us to
patch the JSONEncoder when we start the RPC or the plugin and automagically
support classes that provide a `to_json` method.
The next patch perturbed things enough that we suddenly started
getting (with --track-origins=yes):
Valgrind error file: valgrind-errors.120470
==120470== Use of uninitialised value of size 8
==120470== at 0x14EBD5: htable_val (htable.c:150)
==120470== by 0x14EC3C: htable_firstval_ (htable.c:165)
==120470== by 0x14F583: htable_del_ (htable.c:349)
==120470== by 0x11825D: pointer_referenced (memleak.c:65)
==120470== by 0x118485: scan_for_pointers (memleak.c:121)
==120470== by 0x118500: memleak_remove_region (memleak.c:130)
==120470== by 0x118A30: call_memleak_helpers (memleak.c:257)
==120470== by 0x118A8B: call_memleak_helpers (memleak.c:262)
==120470== by 0x118A8B: call_memleak_helpers (memleak.c:262)
==120470== by 0x118B25: memleak_find_allocations (memleak.c:278)
==120470== by 0x10EB12: closing_dev_memleak (closingd.c:584)
==120470== by 0x10F3E2: main (closingd.c:783)
==120470== Uninitialised value was created by a heap allocation
==120470== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==120470== by 0x1604E8: allocate (tal.c:250)
==120470== by 0x160AA9: tal_alloc_ (tal.c:428)
==120470== by 0x119BE0: new_per_peer_state (per_peer_state.c:24)
==120470== by 0x11A101: fromwire_per_peer_state (per_peer_state.c:95)
==120470== by 0x10FB7C: fromwire_closingd_init (closingd_wiregen.c:103)
==120470== by 0x10ED15: main (closingd.c:626)
==120470==
This is because there is uninitialized padding at the end of struct
peer_state.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently ``LightningConnection.remote_pubkey`` is set to ``None`` if the node is not the handshake initiator. This sets it to ``rs`` in act three from the receiver side
Since pyln-bolt* specify the 0.8.4 version which we didn't upload, and the
requirements.txt specify ==0.8.4, we need to backfill that version, even if we
could just bump it directly to 0.9.1.
Hooks do not tolerate failures at all. If we return a JSON-RPC error to a hook
call the only thing the main daemon can really do is to crash. This commit
adds a mapping of error to a safe fallback result, including a warning to the
node operator that this should be addressed in the plugin. The warning is
reported as a `**BROKEN**` message, and should therefore fail any testing done
on the plugin.
Changelog-Fixed: pyln: Fixed HTLCs hanging indefinitely if the hook function raises an exception. A safe fallback result is now returned instead.
Fixes: #2679
Changelog-Added: JSON-RPC: New `multiwithdraw` command to batch multiple onchain sends in a single transaction. Note it shuffles inputs and outputs, does not use BIP69.
Re-write start_ln such that we can create up to 10 nodes locally for
testing. Useful for scenarios where more than two nodes are needed
Changelog-Changed: contrib: startup_regtest.sh `startup_ln` now takes a number of nodes to create as a parameter
This is an extra safety check for dual funding, where we only want to sign
the inputs we provided!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `signpsbt` takes an optional `signonly` array to limit what inputs to sign.
Required for dual funding where the opener sets it.
Changelog-Added: JSON-RPC: `fundpsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows plugins to choose how to present things in getmanifest.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: plugins: `getmanifest` may now include "allow-deprecated-apis" boolean flag.
Changelog-Deprecated: plugins: `getmanifest` without any parameters; plugins should accept any parameters for future use.
And when it's set, and we're SLOW_MACHINE, simply disable valgrind.
Since Travis (SLOW_MACHINE=1) only does VALGRIND=1 DEVELOPER=1 tests,
and VALGRIND=0 DEVELOPER=0 tests, it was missing tests which needed
DEVELOPER and !VALGRIND.
Instead, this demotes them to non-valgrind tests for SLOW_MACHINEs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can query all the txids at once, rather than one at a time.
Doesn't make any measurable difference to full runtime testing here
though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We might have funds prior to calling join_nodes(), so testing that
we've all seen the block is better.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is what fund_channel() does, which is more thorough than what
we were doing. But since the order of the logs is undefined, we need
to be a little careful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reduces VALGRIND=1 node_factory.line_graph(5) time on my laptop from 42s to 36s.
This is simply because forking all the subdaemons just to check the
version is very expensive under valgrind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fundpsbt forces the caller to manually add their weight * feerate
to the satoshis they ask for. That means no named feerates.
Instead, create a startweight parameter and do the calc for them
internally, and return the feerate we used (and, while we're at it,
the estimated final weight).
This API change is best done now, as it would otherwise have to
be appended as a parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: new call `signpsbt` which will add the wallet's signatures to a provided psbt
Changelog-Added: JSON-RPC: new call `sendpsbt` which will finalize and send a signed PSBT
Reserve and unreserve wallet UTXOs using a PSBT which includes those
inputs.
Note that currently we unreserve inputs everytime the node restarts.
This will be addressed in a future commit.
Changelog-Added: JSON-RPC: Adds two new rpc methods, `reserveinputs` and `unreserveinputs`, which allow for reserving or unreserving wallet UTXOs
Now they look like 1.0.1.137, so you can explicitly depend on a csv change
(without caring about a textual change).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. version was 0.0.2 in setup.py, which means we didn't get the dist/ files we expected.
2. We need 'bdist_wheel' to make the .whl file.
3. --no-site-packaged was apparently removed in 0.20.0, and was default long before that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This includes some real bugfixes, since it noticed some places we were
being loose with different types!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They're almost entirely autogenerated, and we use symlinks into the
top directory to reduce replication.
They can't be under pyln.spec.message, because a package can't also
be a namespace.
We also add fulltext and desc fields, and exclude our "gen" files from
flake8, since the spec quotes contain weird whitespace.
Changelog-Added: Python: pyln.spec.bolt{1,2,4,7} packages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Remove non-existant pyln.proto.bolts. bolts will have separate setup.py, so we
can rev the versions individually.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This delivers the message contents in a much friendlier form for
manipulation: in particular, it makes it easy to compare two
messages without having to know all the message type internals.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they don't exist, that's OK. These will eventually be going away
from the spec, but there are still some in gossip messages for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Exposing the array types is required for our dummyrunner in the lnprototest suite, since
it wants to be able to generate fake fields.
The set_field is similarly useful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Other changes along the way:
1. In a couple of places we passed None as a dummy for for
`otherfields` where {} is just as good.
2. Turned bytes into hex for errors.
3. Remove nonsensical (unused) get_tlv_by_number() function from MessageNamespace
4. Renamed unrelated-but-overlapping `field_from_csv` and
`type_from_csv` static methods, since mypy thought they should have
the same type.
5. Unknown tlv fields are placed in dict as strings, not ints, for
type simplicity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are autogenerated, but now they export their own
MessageNamespace, as well as the raw csv.
They also expose their SubtypeTypes, MessageTypes and TlvStreamTypes,
though in theory these could clash (they don't for now, and it'd be
kinda awkward if they did).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This contains the CSVs for the current bolts (autogenerated). It's a
separate module because I expect it to be updated alongside the spec.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message.bolts
This will be useful for the next patch, which introduces per-bolt
modules. This makes it easier for them generate variables for each
field type they parse (they don't want to export u16, for example)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of val_to_bin/val_from_bin which deal with bytes, we implement
read and write which use streams. This simplifies the API.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This supports infrasructure for creating messages. In particular, it
can be fed CSV from the spec's `tools/extract-formats.py` and then convert
them all to and from strings and binary formats.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message
Moves the 'daemon'ization from c-lightning to the process level, so that
stderr print messages appear in the terminal. Easier debugging!
Changelog-None
At some point lightning-cli help defaulted to human readable format, the additional -H broke the bash completion.
Changelog-Fixed: bash completion on lightning-cli now works again
contrib/pyln-proto/pyln/proto/bech32.py:120
/home/rusty/devel/cvs/lightning/contrib/pyln-proto/pyln/proto/bech32.py:120: SyntaxWarning: "is not" with a literal. Did you mean "!="?
assert decode(hrp, ret) is not (None, None)
I think this warning is correct (though I don't see the warning once I installed coincurve:
are we suppressing warnings?)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I quickly ran through the comments in lightning.py and saw a few small inconsistencies:
- upper/lower case for the "B" in "Bitcoin" unified (see https://github.com/lnbook/lnbook/pull/98)
- added missing "." after a complete sentence
- removed unnecessary double spaces
secp256k1 Python library is not maintained anymore and coincurve was
already used in the `wire` module.
Changelog-None
Signed-off-by: Michal Rostecki <mrostecki@mailfence.com>
Telling `lightningd` to pass a `-datadir` to `bitcoin-cli` so it doesn't go
snooping where it doesn't belong (i.e., the user's home directory and config).
Changelog-None
Suggested-by: Simon Vrouwe <@SimonVrouwe>
Signed-off-by: Christian Decker <@cdecker>
This was breaking a couple of tests if the pyln version was not synced up with
`lightningd`, so now we just warn (these are then collected when running in
pytest and highlighted).
We only support a very limited number of argument combinations, and apparently
sometimes we trigger a case we aren't handling. This adds a more useful error
message, including the params we didn't match.
So far we've always cleared the node directory when provisioning the node, but
while testing some plugins we noticed that pre-generating some files in that
directory is useful. This just adds yet another flag to `get_node` that
disables deleting any existing node directory.
A CONSERVATIVE/3 target for them.
Some noisy changes to the tests as we had to update the estimatesmartfee
mock.
Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
We kept track of an URGENT, a NORMAL, and a SLOW feerate. They were used
for opening (NORMAL), mutual (NORMAL), UNILATERAL (URGENT) transactions
as well as minimum and maximum estimations, and onchain resolution.
We now keep track of more fine-grained feerates:
- `opening` used for funding and also misc transactions
- `mutual_close` used for the mutual close transaction
- `unilateral_close` used for unilateral close (commitment transactions)
- `delayed_to_us` used for resolving our output from our unilateral close
- `htlc_resolution` used for resolving onchain HTLCs
- `penalty` used for resolving revoked transactions
We don't modify our requests to our Bitcoin backend, as the next commit
will batch them !
Changelog-deprecated: The "urgent", "slow", and "normal" field of the `feerates` command are now deprecated.
Changelog-added: The fields "opening", "mutual_close", "unilateral_close", "delayed_to_us", "htlc_resolution" and "penalty" have been added to the `feerates` command.
pytest captures the output by monkey patching out `sys.stdout`. This may
conflict with our use of `sys.stdout` when configuring logging, resulting in
the "Write to closed file" issue that is spamming the logs. By making the
logging configuration a fixture hopefully we always use the correct
stdout (after pytest has monkey-patched it).
For some reason we fail to remove the test directory in some cases. My
hypothesis is that it is a daemon that is not completely shut down yet, and
still writes to the directory. This commit intercepts the error, prints any
files in the directory and re-raises the error. This should allow us to debug
the reappears.
Some tests may not spawn a node at all, so make sure that our assumption that
the directory exists in the fixture cleanup is correct by creating the
directory.
we loosely enforce that the specified type must be one of the listed
options. you can still cause an error because we're not checking the
default value you're passing in ...
not sure if this is totally necessary, should we jsut let clightning
enforce the input?
if the node fails to start (and we're expecting it to) return to us the
node object anyway
we also signal to collect all of its stderr logs by setting stderr
on the tailableproc that backs the node
ChangeLog-Added: New `getsharedsecret` command, which lets you compute a shared secret with this node knowing only a public point. This implements the BOLT standard of hashing the ECDH point, and is incompatible with ECIES.
Update to v13 as v12 started to raise undefined symbol exceptions agains latest
libsecp256k1.
Note: any version of `pip install coincurve` fails if no libsecp256k1 headers
are installed, should we point this out somewhere/somehow?
Changelog-None
we have 4 venues in which we can add features, 3 of which are unilaterally
controlled (`init`, `node_announcement`, and `invoices`) the
`channel_announcement` is co-signed by both parties, so we can't add
featurebits without additional coordination overhead.
Each location is encoded as a key-value pair in a dict called `featurebits` in
the manifest (omitted if no custom featurebits are set).
We are about to disect a couple of invoices for features, so let's add a class
that can encode and decode invoices from bolt11 strings. This is pretty much
the lnaddr.py file created by @rustyrussell with some minor changes. I'm
planning to clean this up further which is why I'm only exporting the
`Invoice` class for now.
The `generate` has been deprecated since 0.16 and has been removed in 0.18.0
so we better use `generatetoaddress` instead, which is already what we do with
`bitcoind`. So we remove the override here.
Fixes: #3192
Changelog-Added: `waitanyinvoice` now supports a `timeout` parameter, which when set will cause the command to fail when the timeout is reached; can set this to 0 to fail immediately if no new invoice has been paid yet.
As per the spec (https://www.jsonrpc.org/specification#request_object)
```
A rpc call is represented by sending a Request object to a Server. The Request object has the following members:
jsonrpc
A String specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".
```
Changelog-fixed: pyln now includes the "jsonrpc" field to jsonrpc2 requests
This is needed to fully implement handling of blockheight disagreements
between us and payee.
If payee believes the blockheight is higher than ours, then `pay`
should wait for our node to achieve that blockheight.
Changelog-Add: Implement `waitblockheight` to wait for a specific blockheight.
Some Linux OSs impose a length limit on the path a Unix socket may have. This
is not an issue in `lightningd` since we `chdir()` into that directory before
opening the socket, however in pyln this became a problem for some tests,
since we use absolute paths in the testing framework. It's also a rather
strange quirk to expose to users.
This patch introduces a `UnixSocket` abstraction that attempts to work around
these limitations by aliasing the directory containing the socket into
`/proc/self/fd` and then connecting using that alias.
It was inspired by Open vSwitch code here https://github.com/openvswitch/ovs/blob/master/python/ovs/socket_util.py
Signed-off-by: Christian Decker <@cdecker>
We were indiscriminately accessing the `__annotations__` which could cause
issues if the function had been wrapped by some functions such as
`functools.partial`. This just checks that the access is safe before doing it.
Suggested-by: jarret <@jarret>
Signed-off-by: Christian Decker <@cdecker>
it's that time of year (merry xmas!)
enables the ability to push_msat on fundchannel
Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
This will change some `requirements.txt` of pyln-testing in a way that
it does not require different package version i.e. to `tests/requirements.txt`.
The reason for this is that users are not forced to hassle with pyenv
or virtualenv and could just use `--user`.
```bash
pip install --user -r tests/requirements.txt -r contrib/pyln-testing/requirements.txt
```
Changelog-None
Trying to `pip install psycopg2-binary==2.8.3` raised an error for a
long time. Since version `2.8.4` is recent and also seem to work I
suggest updating the requirements so other users dont run into the
following error, where pip tries to use `pg_config` for the BINARY
package:
Collecting psycopg2-binary==2.8.3 (from -r contrib/pyln-testing/requirements.txt (line 6))
Using cached 91911be018/psycopg2-binary-2.8.3.tar.gz
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-zapr0fhs/psycopg2-binary/
Complete output (23 lines):
running egg_info
creating pip-egg-info/psycopg2_binary.egg-info
writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Changelog-None
Thanks to @t-bast, who made this possible by interop testing with Eclair!
Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were hardcoding the chainparams->chain_hash which caused the query to
return an empty result. By parametrizing the test we can make it work on
elements.
addresses issue #2753.
Formatting the JSON with the default parameters will escape the unicode
symbols in a way that c-lightning won't allow, leading to an exception.
Changelog-Fixed: `pylightning` now handles unicode characters in JSON-RPC requests and responses correctly.
In the c-lightning tests we have `tests/conftest.py` which annotates test
function with the outcome. If we use pyln-testing outside of the c-lightning
tree we cannot rely on that annotation being there, so we assume it passed.
Using the psycopg2-binary package means that the apropriate compiled binary
for the user platform will be shipped alongside the python binaries. Otherwise
the python bindings and the C shims would be shipped which would then require
the postgres development packages as well.
This just makes things easier, since we don't require the build dependencies.
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Spaces just make life a little harder for everyone.
(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.
Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
A log can have a default node_id, which can be overridden on a per-entry
basis. This changes the format of logging, so some tests need rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.
We were relying heavily on NodeFactory to do some magic before instantiating
the Node with rpc and daemon initialized, that meant that we'd have to replace
all 3 classes when customizing the node to our needs. Moving that
initialization into the node itself means that the LightningNode class now can
be swapped out and customized, without having to wire everything else through.
`DEVELOPER=1` assumes that the binary has been compiled with developer set to
true, which might not be the case for plugin developers. Setting this to 0 by
default has no effect in c-lightning since we always at least set it in
`config.vars` but may prevent some issues outside.
We'll rewrite the tests to use this infrastructure in the next commit.
Changelog-Added: The new `pyln-testing` package now contains the testing infrastructure so it can be reused to test against c-lighting in external projects
This should not affect any consumer of the API since we just shift the actual
implementation from one side to the other, and keep aliases in place so
scripts don't break.
We also bump the version number from 0.0.7.3 to 0.7.4 which allows us to be in
sync with c-lightning itself, and remove the superfluous `0` in front.
Takes advantage of upfront-shutdown-script to permit users to
specify the close-to address for a channel at open, by adding
a `close_to` field to `fundchannel_start`.
Note that this only is in effect if `fundchannel_start` returns
with `close_to` set -- otherwise, peer doesn't
support `option_upfront_shutdown_script`.
Command format: close id [unilateraltimeout] [destination]
Close the channel with peer {id}, forcing a unilateral
close after {unilateraltimeout} seconds if non-zero, and
the to-local output will be sent to {destination}. If
{destination} isn't specified, the default is the address
of lightningd.
Also change the pylightning:
update the `close` API to support `destination` parameter
* Add compatibility with the new shortchannelid syntax with x as separator
* Add an error message in case the parse failed, instead of receveing an unrelated error from the bitcoin-cli
This is the first step to transition to a better organized python module
structure. Sadly we can't reuse the `pylightning` module as a namespace module
since having importable things in the top level of the namespace is not
allowed in any of the namespace variants [1], hence we just switch over to the
`pyln` namespace. The code the was under `lightning` will now be reachable
under `pyln.client` and we add the `pyln.proto` module for all the things that
are independent of talking to lightningd and can be used for protocol testing.
[1] https://packaging.python.org/guides/packaging-namespace-packages/
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Allow a user to select the utxo set that will be added to a
transaction, via the `utxos` parameter. Optional.
Format for utxos should be of the form ["txid:vout","..."]
___________________________________________________________________________
/ The most important plugin. Ever. Much thanks to @practicalswift and @jb55 \
\ for their improvements! /
---------------------------------------------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In contrib/bootstrap-node.sh line 7:
if type lightning-cli >/dev/null 2>&1; then
^-- SC2039: In POSIX sh, 'type' is undefined.
In contrib/startup_regtest.sh line 41:
type lightning-cli || return
^-- SC2039: In POSIX sh, 'type' is undefined.
In contrib/startup_regtest.sh line 42:
type lightningd || return
^-- SC2039: In POSIX sh, 'type' is undefined.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call. `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.
Timing out JSON calls is generally deprecated: that's the job of the
client. And the semantics of this are confusing, even to me! A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.
The transition code is awkward, but we'll manage for the three
releases until we can remove it.
The new defaults are to unilaterally close after 48 hours.
Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
