This includes some real bugfixes, since it noticed some places we were
being loose with different types!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They're almost entirely autogenerated, and we use symlinks into the
top directory to reduce replication.
They can't be under pyln.spec.message, because a package can't also
be a namespace.
We also add fulltext and desc fields, and exclude our "gen" files from
flake8, since the spec quotes contain weird whitespace.
Changelog-Added: Python: pyln.spec.bolt{1,2,4,7} packages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Remove non-existant pyln.proto.bolts. bolts will have separate setup.py, so we
can rev the versions individually.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This delivers the message contents in a much friendlier form for
manipulation: in particular, it makes it easy to compare two
messages without having to know all the message type internals.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If they don't exist, that's OK. These will eventually be going away
from the spec, but there are still some in gossip messages for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Exposing the array types is required for our dummyrunner in the lnprototest suite, since
it wants to be able to generate fake fields.
The set_field is similarly useful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Other changes along the way:
1. In a couple of places we passed None as a dummy for for
`otherfields` where {} is just as good.
2. Turned bytes into hex for errors.
3. Remove nonsensical (unused) get_tlv_by_number() function from MessageNamespace
4. Renamed unrelated-but-overlapping `field_from_csv` and
`type_from_csv` static methods, since mypy thought they should have
the same type.
5. Unknown tlv fields are placed in dict as strings, not ints, for
type simplicity.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are autogenerated, but now they export their own
MessageNamespace, as well as the raw csv.
They also expose their SubtypeTypes, MessageTypes and TlvStreamTypes,
though in theory these could clash (they don't for now, and it'd be
kinda awkward if they did).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This contains the CSVs for the current bolts (autogenerated). It's a
separate module because I expect it to be updated alongside the spec.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message.bolts
This will be useful for the next patch, which introduces per-bolt
modules. This makes it easier for them generate variables for each
field type they parse (they don't want to export u16, for example)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of val_to_bin/val_from_bin which deal with bytes, we implement
read and write which use streams. This simplifies the API.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This supports infrasructure for creating messages. In particular, it
can be fed CSV from the spec's `tools/extract-formats.py` and then convert
them all to and from strings and binary formats.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message
Moves the 'daemon'ization from c-lightning to the process level, so that
stderr print messages appear in the terminal. Easier debugging!
Changelog-None
At some point lightning-cli help defaulted to human readable format, the additional -H broke the bash completion.
Changelog-Fixed: bash completion on lightning-cli now works again
contrib/pyln-proto/pyln/proto/bech32.py:120
/home/rusty/devel/cvs/lightning/contrib/pyln-proto/pyln/proto/bech32.py:120: SyntaxWarning: "is not" with a literal. Did you mean "!="?
assert decode(hrp, ret) is not (None, None)
I think this warning is correct (though I don't see the warning once I installed coincurve:
are we suppressing warnings?)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I quickly ran through the comments in lightning.py and saw a few small inconsistencies:
- upper/lower case for the "B" in "Bitcoin" unified (see https://github.com/lnbook/lnbook/pull/98)
- added missing "." after a complete sentence
- removed unnecessary double spaces
secp256k1 Python library is not maintained anymore and coincurve was
already used in the `wire` module.
Changelog-None
Signed-off-by: Michal Rostecki <mrostecki@mailfence.com>
Telling `lightningd` to pass a `-datadir` to `bitcoin-cli` so it doesn't go
snooping where it doesn't belong (i.e., the user's home directory and config).
Changelog-None
Suggested-by: Simon Vrouwe <@SimonVrouwe>
Signed-off-by: Christian Decker <@cdecker>
This was breaking a couple of tests if the pyln version was not synced up with
`lightningd`, so now we just warn (these are then collected when running in
pytest and highlighted).
We only support a very limited number of argument combinations, and apparently
sometimes we trigger a case we aren't handling. This adds a more useful error
message, including the params we didn't match.
So far we've always cleared the node directory when provisioning the node, but
while testing some plugins we noticed that pre-generating some files in that
directory is useful. This just adds yet another flag to `get_node` that
disables deleting any existing node directory.
A CONSERVATIVE/3 target for them.
Some noisy changes to the tests as we had to update the estimatesmartfee
mock.
Changelog-Changed: We now use a higher feerate for resolving onchain HTLCs and for penalty transactions
We kept track of an URGENT, a NORMAL, and a SLOW feerate. They were used
for opening (NORMAL), mutual (NORMAL), UNILATERAL (URGENT) transactions
as well as minimum and maximum estimations, and onchain resolution.
We now keep track of more fine-grained feerates:
- `opening` used for funding and also misc transactions
- `mutual_close` used for the mutual close transaction
- `unilateral_close` used for unilateral close (commitment transactions)
- `delayed_to_us` used for resolving our output from our unilateral close
- `htlc_resolution` used for resolving onchain HTLCs
- `penalty` used for resolving revoked transactions
We don't modify our requests to our Bitcoin backend, as the next commit
will batch them !
Changelog-deprecated: The "urgent", "slow", and "normal" field of the `feerates` command are now deprecated.
Changelog-added: The fields "opening", "mutual_close", "unilateral_close", "delayed_to_us", "htlc_resolution" and "penalty" have been added to the `feerates` command.
pytest captures the output by monkey patching out `sys.stdout`. This may
conflict with our use of `sys.stdout` when configuring logging, resulting in
the "Write to closed file" issue that is spamming the logs. By making the
logging configuration a fixture hopefully we always use the correct
stdout (after pytest has monkey-patched it).
For some reason we fail to remove the test directory in some cases. My
hypothesis is that it is a daemon that is not completely shut down yet, and
still writes to the directory. This commit intercepts the error, prints any
files in the directory and re-raises the error. This should allow us to debug
the reappears.
Some tests may not spawn a node at all, so make sure that our assumption that
the directory exists in the fixture cleanup is correct by creating the
directory.
we loosely enforce that the specified type must be one of the listed
options. you can still cause an error because we're not checking the
default value you're passing in ...
not sure if this is totally necessary, should we jsut let clightning
enforce the input?
if the node fails to start (and we're expecting it to) return to us the
node object anyway
we also signal to collect all of its stderr logs by setting stderr
on the tailableproc that backs the node
ChangeLog-Added: New `getsharedsecret` command, which lets you compute a shared secret with this node knowing only a public point. This implements the BOLT standard of hashing the ECDH point, and is incompatible with ECIES.
Update to v13 as v12 started to raise undefined symbol exceptions agains latest
libsecp256k1.
Note: any version of `pip install coincurve` fails if no libsecp256k1 headers
are installed, should we point this out somewhere/somehow?
Changelog-None
we have 4 venues in which we can add features, 3 of which are unilaterally
controlled (`init`, `node_announcement`, and `invoices`) the
`channel_announcement` is co-signed by both parties, so we can't add
featurebits without additional coordination overhead.
Each location is encoded as a key-value pair in a dict called `featurebits` in
the manifest (omitted if no custom featurebits are set).
We are about to disect a couple of invoices for features, so let's add a class
that can encode and decode invoices from bolt11 strings. This is pretty much
the lnaddr.py file created by @rustyrussell with some minor changes. I'm
planning to clean this up further which is why I'm only exporting the
`Invoice` class for now.
The `generate` has been deprecated since 0.16 and has been removed in 0.18.0
so we better use `generatetoaddress` instead, which is already what we do with
`bitcoind`. So we remove the override here.
Fixes: #3192
Changelog-Added: `waitanyinvoice` now supports a `timeout` parameter, which when set will cause the command to fail when the timeout is reached; can set this to 0 to fail immediately if no new invoice has been paid yet.
As per the spec (https://www.jsonrpc.org/specification#request_object)
```
A rpc call is represented by sending a Request object to a Server. The Request object has the following members:
jsonrpc
A String specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".
```
Changelog-fixed: pyln now includes the "jsonrpc" field to jsonrpc2 requests
This is needed to fully implement handling of blockheight disagreements
between us and payee.
If payee believes the blockheight is higher than ours, then `pay`
should wait for our node to achieve that blockheight.
Changelog-Add: Implement `waitblockheight` to wait for a specific blockheight.
Some Linux OSs impose a length limit on the path a Unix socket may have. This
is not an issue in `lightningd` since we `chdir()` into that directory before
opening the socket, however in pyln this became a problem for some tests,
since we use absolute paths in the testing framework. It's also a rather
strange quirk to expose to users.
This patch introduces a `UnixSocket` abstraction that attempts to work around
these limitations by aliasing the directory containing the socket into
`/proc/self/fd` and then connecting using that alias.
It was inspired by Open vSwitch code here https://github.com/openvswitch/ovs/blob/master/python/ovs/socket_util.py
Signed-off-by: Christian Decker <@cdecker>
We were indiscriminately accessing the `__annotations__` which could cause
issues if the function had been wrapped by some functions such as
`functools.partial`. This just checks that the access is safe before doing it.
Suggested-by: jarret <@jarret>
Signed-off-by: Christian Decker <@cdecker>
it's that time of year (merry xmas!)
enables the ability to push_msat on fundchannel
Changelog-Added: RPC: `fundchannel` and `fundchannel_start` can now accept an optional parameter, `push_msat`, which will gift that amount of satoshis to the peer at channel open.
This will change some `requirements.txt` of pyln-testing in a way that
it does not require different package version i.e. to `tests/requirements.txt`.
The reason for this is that users are not forced to hassle with pyenv
or virtualenv and could just use `--user`.
```bash
pip install --user -r tests/requirements.txt -r contrib/pyln-testing/requirements.txt
```
Changelog-None
Trying to `pip install psycopg2-binary==2.8.3` raised an error for a
long time. Since version `2.8.4` is recent and also seem to work I
suggest updating the requirements so other users dont run into the
following error, where pip tries to use `pg_config` for the BINARY
package:
Collecting psycopg2-binary==2.8.3 (from -r contrib/pyln-testing/requirements.txt (line 6))
Using cached 91911be018/psycopg2-binary-2.8.3.tar.gz
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zapr0fhs/psycopg2-binary/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-zapr0fhs/psycopg2-binary/
Complete output (23 lines):
running egg_info
creating pip-egg-info/psycopg2_binary.egg-info
writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Changelog-None
Thanks to @t-bast, who made this possible by interop testing with Eclair!
Changelog-Added: Protocol: can now send and receive TLV-style onion messages.
Changelog-Added: Protocol: can now send and receive BOLT11 payment_secrets.
Changelog-Added: Protocol: can now receive basic multi-part payments.
Changelog-Added: RPC: low-level commands sendpay and waitsendpay can now be used to manually send multi-part payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were hardcoding the chainparams->chain_hash which caused the query to
return an empty result. By parametrizing the test we can make it work on
elements.
addresses issue #2753.
Formatting the JSON with the default parameters will escape the unicode
symbols in a way that c-lightning won't allow, leading to an exception.
Changelog-Fixed: `pylightning` now handles unicode characters in JSON-RPC requests and responses correctly.
In the c-lightning tests we have `tests/conftest.py` which annotates test
function with the outcome. If we use pyln-testing outside of the c-lightning
tree we cannot rely on that annotation being there, so we assume it passed.
Using the psycopg2-binary package means that the apropriate compiled binary
for the user platform will be shipped alongside the python binaries. Otherwise
the python bindings and the C shims would be shipped which would then require
the postgres development packages as well.
This just makes things easier, since we don't require the build dependencies.
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Spaces just make life a little harder for everyone.
(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.
Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
A log can have a default node_id, which can be overridden on a per-entry
basis. This changes the format of logging, so some tests need rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.
We were relying heavily on NodeFactory to do some magic before instantiating
the Node with rpc and daemon initialized, that meant that we'd have to replace
all 3 classes when customizing the node to our needs. Moving that
initialization into the node itself means that the LightningNode class now can
be swapped out and customized, without having to wire everything else through.
`DEVELOPER=1` assumes that the binary has been compiled with developer set to
true, which might not be the case for plugin developers. Setting this to 0 by
default has no effect in c-lightning since we always at least set it in
`config.vars` but may prevent some issues outside.
We'll rewrite the tests to use this infrastructure in the next commit.
Changelog-Added: The new `pyln-testing` package now contains the testing infrastructure so it can be reused to test against c-lighting in external projects
This should not affect any consumer of the API since we just shift the actual
implementation from one side to the other, and keep aliases in place so
scripts don't break.
We also bump the version number from 0.0.7.3 to 0.7.4 which allows us to be in
sync with c-lightning itself, and remove the superfluous `0` in front.
Takes advantage of upfront-shutdown-script to permit users to
specify the close-to address for a channel at open, by adding
a `close_to` field to `fundchannel_start`.
Note that this only is in effect if `fundchannel_start` returns
with `close_to` set -- otherwise, peer doesn't
support `option_upfront_shutdown_script`.
Command format: close id [unilateraltimeout] [destination]
Close the channel with peer {id}, forcing a unilateral
close after {unilateraltimeout} seconds if non-zero, and
the to-local output will be sent to {destination}. If
{destination} isn't specified, the default is the address
of lightningd.
Also change the pylightning:
update the `close` API to support `destination` parameter
* Add compatibility with the new shortchannelid syntax with x as separator
* Add an error message in case the parse failed, instead of receveing an unrelated error from the bitcoin-cli
This is the first step to transition to a better organized python module
structure. Sadly we can't reuse the `pylightning` module as a namespace module
since having importable things in the top level of the namespace is not
allowed in any of the namespace variants [1], hence we just switch over to the
`pyln` namespace. The code the was under `lightning` will now be reachable
under `pyln.client` and we add the `pyln.proto` module for all the things that
are independent of talking to lightningd and can be used for protocol testing.
[1] https://packaging.python.org/guides/packaging-namespace-packages/
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Allow a user to select the utxo set that will be added to a
transaction, via the `utxos` parameter. Optional.
Format for utxos should be of the form ["txid:vout","..."]
___________________________________________________________________________
/ The most important plugin. Ever. Much thanks to @practicalswift and @jb55 \
\ for their improvements! /
---------------------------------------------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In contrib/bootstrap-node.sh line 7:
if type lightning-cli >/dev/null 2>&1; then
^-- SC2039: In POSIX sh, 'type' is undefined.
In contrib/startup_regtest.sh line 41:
type lightning-cli || return
^-- SC2039: In POSIX sh, 'type' is undefined.
In contrib/startup_regtest.sh line 42:
type lightningd || return
^-- SC2039: In POSIX sh, 'type' is undefined.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call. `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.
Timing out JSON calls is generally deprecated: that's the job of the
client. And the semantics of this are confusing, even to me! A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.
The transition code is awkward, but we'll manage for the three
releases until we can remove it.
The new defaults are to unilaterally close after 48 hours.
Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The helpme plugin is more comprensive, but this at least connects to a
few random nodes, and doesn't require python libraries in path or anything.
I selected the nodes from helpme.py, eliminating ones I couldn't reach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We recently noticed that the way we unpack the call arguments for hooks and
notifications in pylightning breaks pretty quickly once you start changing the
hook and notification params. If you add params they will not get mapped
correctly causing the plugin to error out.
This can be fixed by adding a `VAR_KEYWORD` argument to the calbacks, i.e., by
adding a single `**kwargs` argument at the end of the signature. This commit
adds a check that such a catch-all argument exists, and emits a warning if it
doesn't.
It also fixes up the plugins that we ship ourselves.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Add an RPC method (not working at the moment) called
`fundchannel_continue` that takes as its parameters a
node_id and a txid for a transaction (that ostensibly has an output
for a channel)
The next commit breaks it: `if b' }\n' not in buff:` is always true since
we're about to clean up our JSON so there won't be a space. I could have
hacked the space in our JSON, but 6 months is long enough anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Tries to return the approxmost posible string of a Millisatoshi amount using
various unit representations. The function will round to an effective
number of digits. Default: 3.
```
>>> Millisatoshi("100000sat").to_approx_str()
'0.001btc'
>>> Millisatoshi("100msat").to_approx_str()
'0.1sat'
>>> Millisatoshi("10000000sat").to_approx_str()
'0.1btc'
```
We had a bit of a hand-woven mess in there, trying to inject the extra
arguments in the correct places. We now instead treat positional and keyword
calls separately and can go back to using the builtin argument binding again.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The old codes if % 1000 statement logic was simply inverted
and produced the opposite output of the intention behin it.
Before Fix:
- Millisatoshi('42sat').to_btc_str() => 0.00000042000btc
- Millisatoshi('42001msat').to_btc_str() => 0.00000042btc
After Fix:
- Millisatoshi('42sat').to_btc_str() => 0.00000042btc
- Millisatoshi('42001msat').to_btc_str() => 0.00000042001btc
Currently, when a multiplication operator is invoked that
does not result in an even integer result but a floating result,
the pylightning code will raise an exception:
Millisatoshi must be string with msat/sat/btc suffix or int
This is because the internal float result will be used as
contructor argument like this: return Millisatoshi(10000.5)
This happens especially on fee calculations where small uneven amounts
are calculated.
Millisatoshi's inner representation expected to be an int, otherwise unwanted exceptions could occur
The following example raises: TypeError: __int__ returned non-int (type decimal.Decimal)
from lightning import Millisatoshi
one_sat = Millisatoshi("1sat")
two_sats = one_sat * 2
make it easier to fire up a local test environment to try out
c-lightning.
requires bitcoind to be installed. to use, you have to run it
via `source contrib/startup_regtest.sh`, so that the aliases
are set correctly.
* Improved plugin install in docker
- All files generated by 'make install' are copied
- Fixes issues with incomplete installation.
- Example: New executables created by build but are missing in docker.
With the preceeding UTF-8 fix, I'd like to detect UTF-8 support. But
AFAICT Python doesn't have a standard way of doing version exposure.
So I added __version__, but now we need to make sure it matches. I
used the hackiest possible method.
[ Christian Decker fixed version to be sane, so previous comment no longer
applies! --RR ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. We need to read in as a byte string, then decode into utf8 once we
have a marker. Otherwise we seem to mangle it horribly, and we
might have a bad utf8 string anyway.
2. We need to suppress the JSON \u escapes on output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than using LightningJSONDecoder's implicit "field name and
value ends in msat, try converting to Millisatoshi", we do it to
parameters using type annotations.
If you had a parameter which was an array or dict itself, we don't
delve into that, but that's probably OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I originally converted input JSON naively into Millisatoshi, and the
result was a strange failure in Millisatoshi.__eq__.
It seems this is because inspect._empty.__eq__(Millisatoshi) raises
NotImplemented, and so it tries Millisatoshi.__eq__(inspect._empty)
which doesn't like it.
'is' is the correct test here, AFAICT, and doesn't suffer from these
problems.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some JSON functions want a *class*, not just a hook, so provide one.
To make it clear that we want an encoding *class* and a decoding *object*,
rename the UnixDomainSocketRpc encode parameter to encode_cls.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we can't marshall an object into JSON, the exception causes a deadlock
and we don't get any results.
Instead of deadlocking, our failure now is:
lightning.lightning.RpcError: RPC call failed: method: echo, payload: {'msat': 17msat}, error: Error while processing echo: TypeError("Object of type 'Millisatoshi' is not JSON serializable",)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I tried annotating the plugin-millisatoshis.py plugin, and it failed like so:
plugin-millisatoshis.py Killing plugin: "getmanifest" result is not an object: {"jsonrpc": "2.0", "id": 1, "error": "Error while processing getmanifest: ValueError(\'Function has keyword-only parameters or annotations, use getfullargspec() API which can support them\',)"}'
So, let's do that!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the same deprecation, but one level up. For the moment, we
still support invoices with a `h` field (where description will be
necessary) but that will be removed once this option is removed.
Note that I just changed pylightning without backwards compatibility,
since the field was unlikely to be used, but we could do something
more complex here?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this the RPC will fail to continue buffering if the response does not
fit in the first read, and if we don't switch over to the non-compat
mode. This was introduced by our mitigation of the UTF-8 misalignment, but I
missed this path.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
These weren't checked by CI yet, and they are really short so I just added
them to the check-python target.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This indicates that the method or hook will accepts a request
parameter, and will use that to return the result or raise an
exception instead of returning the return value. This allows the hook
or method to stash the incomplete request or pass it around, without
blocking the JSON-RPC interface.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This isn't a problem for now since we don't support multithreading,
and only allow synchronous calls, but eventually this'll become
important.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We well need this in the next commit to be able to return from an
asynchronous call. We also guard stdout access with a reentrant lock
since we are no longer guaranteed that all communication happens on
the same thread.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Sending around unnamed tuples is bound to cause some issues sooner or
later, so we just create a quick class that holds all the information
about a plugin method.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Little point having users handle the postfixes manually, this
translates them, and also allows Millisatoshi to be used wherever an
'int' would be previously.
There are also helpers to create the formatting in a way c-lightning's
JSONRPC will accept.
All standard arithmetic operations with integers work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We read a JSON message from the buffer, after converting it from raw bytes to
UTF-8, and returning the remainder of the byte array back to the
caller. However the return value of `raw_decode` refers to symbols in the
UTF-8 decoded string, not the raw bytes underlying byte-array, which means
that if we have multi-byte encoded UTF-8 symbols in the byte-array we end up
with a misaligned offset and will return part of the message as
remainder. This would then end up being interpreted as the result of the next
call.
This could not be exploited currently since we use a socket only for a single
JSON-RPC call and will close the connection afterwards, but since we want to
eventually recycle connections for multiple calls, this could have been very
dangerous.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Corné Plooy <@bitonic-cjp>
The next patch wants to decorate the methods with a compulsory
'usage' option, which doesn't make sense for init. So I wanted
to change the init to its own decoration.
Made-to-work-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was failing the docker hub builds, since the git-config retains
an absolute path to the worktree location when cloning. Copying it
over from the host system means that this path now points to a
non-existent location, which then interfered with the submodule
initialization.
This fixes it by not using the copy directly, but rather it creates a
clean clone from the copied location, including a submodule init.
Signed-off-by: Christian Decker <@cdecker>
Logging an empty line (without newline character) would raise an
Exception due to out of bounds check.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Valgrind seems to be slowing the pay-plugin down enough for the 10
seconds timeout to get triggered on a semi-regular basis.
Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Instead of creating a new map I opted to re-use the Plugin.methods
map, since the semantics are really similar and we don't allow
duplicates. The only difference is in how they are announced to
lightningd, so we use an enum to differentiate rpcmethods from hooks,
since only the former will get added to the JSON-RPC dispatch table in
lightningd.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
After this code change people can use `plugin.rpc` from anywhere in
their plugin code this is much nicer than going this way:
```
@plugin.method("init")
def init(options, configuration, plugin):
global rpc
basedir = plugin.lightning_dir
rpc_filename = plugin.rpc_filename
path = os.path.join(basedir, rpc_filename)
rpc = LightningRpc(path)
```
or similarly that way:
```
@plugin.method("init")
def init(options, configuration, plugin):
global rpc
basedir = configuration['lightning-dir']
rpc_filename = configuration['rpc-file']
path = os.path.join(basedir, rpc_filename)
rpc = LightningRpc(path)
```
Also the imports have been sorted alphabetically
Co-authored-by: Rene Pickhardt <rene@rene-pickhardt.de>
Co-authored-by: Christian Decker <decker.christian@gmail.com>
If the `request` or `plugin` parameter that are injected by the
framework where before or inbetween positional arguments we'd be
injecting them incorrectly, i.e., we'd be providing them both as
`args` (and mismapping another argument) as well as `kwargs`.
This is a better way to map arguments, which takes advantage of the
fact that JSON-RPC calls are either all positional or named arguments.
I also included a test for various scenarios, that hopefull cover the
most common cases.
Reported-by: Rene Pickhardt <@renepickhardt>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The example code had the `plugin` argument as the last argument. this disallows arguments that have a standard value. As far as I understand the dispatching code the order of arguments does not matter since it is the name `plugin` that is relevant. Therefor I changed the order so that newbe's don't have to read the entire code and can easily add optional arguments
Just like we added the RPC methods, the notification handlers can also
be registered using a function decorator, and we auto-subscribe when
asked for a manifest.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was causing `listchannels` to be incredibly slow. The response is
several megabyte in size, and we were only buffering 1Kb on each
iteration.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It's flask inspired with the Plugin instance and decorators to add
methods to the plugin description.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Fixes some lint errors with unused variables:
contrib/plugins/fail/failtimeout.py:48:5:
F841 local variable 'e' is assigned to but never used
contrib/plugins/helloworld.py:86:5:
F841 local variable 'e' is assigned to but never used
Signed-off-by: William Casarin <jb55@jb55.com>
This tells the plugin both the `lightning-dir` as well as the
`rpc-filename` to use to talk to `lightningd`. Prior to this they'd
had to guess.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We inadvertently broke the compatibility between the python library
and the binary when switching to \n\n-delimiters. This reintroduces
the old inefficient parsing, and dynamically upgrades to the faster
version if it detects the \n\n-delimiter.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Both of these plugins will fail in interesting ways, and we should
still handle them correctly.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This doesn't make a performance difference, but even better, it
simplifies the code.
We hacked test_multirpc to send 200x as many commands, and timed the
pytest over 20 runs:
Before:
=================== 1 passed, 136 deselected in 8.550000-9.400000(9.0045+/-0.2) seconds ===================
After:
=================== 1 passed, 136 deselected in 8.540000-9.370000(8.97286+/-0.16) seconds ===================
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to keep the remaining buffer, and we need to try to parse it
before we read the next. I first tried keeping it in the object, but
its lifetime is that of the *socket*, which we actually reopen for
every command.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We also make `--help` a non-early arg so it allows for the plugins to
register their options before printing the help message. The options
themselves are stored in a separate struct inbetween them being
registered and them being forwarded to the plugin. Currently only
supports string options.
Signed-off-by: Christian Decker <@cdecker>
And there's a difference between no description and "" as a description:
for no description, listpayments doesn't show the field at all. So fix
that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
While not strictly necessary it's certainly a good idea to test
against the latest one and not encourage users to use old versions.
Reported-by: Jonas Nick <@jonasnick>
Signed-off-by: Christian Decker <@cdecker>
And, reluctantly, default to bitcoind style.
"It's wrong to be right too soon."
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No semantical change but when using the python lib for the rpc-api it is confusing that my developing environment suggests that I should fund a channel and pass a channel_id when in fact I want to pass a node_id
This is useful mainly in the case where bitcoind is not giving estimates,
but can also be used to bias results if you want.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Manipulate fees via fake-bitcoin-cli. It's not quite the same, as
these are pre-smoothing, so we need a restart to override that where
we really need an exact change. Or we can wait until it reaches a
certain value in cases we don't care about exact amounts.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They're much more useful being programatically-accessible, AFAICT.
The string stays the same so they're backwards compatible.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`getinfo` has been providing the blockheight for a good while and doesn't
require the `DEVELOPER=1` flag during compilation, so it should be the preferred
method to retrieve the blockchain height.
Strictly speaking not needed for the build and test, but if we want to use the
image also for packaging this is required.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Also report tx and txid, and whether we closed unilaterally or
bilaterally, if we could close the channel.
Also make a manpage.
Fixes: #1207Fixes: #714Fixes: #622
We can have more than one; eg we might offer both bech32 and a p2sh
address, and in future we might offer v1 segwit, etc.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
* Modifies invoice command to have the following format
invoice <msatoshi> <label> <desc> <?expiry> <?fallbackaddr>
* Adds support for Segwit bcrt1 addresses for withdraw
* Add test case for fallback address in invoice creation
* Create a common json_tok_address_scriptpubkey to be used
by invoice and withdraw commands.
* Fix dev_setfees to set slow and normal fees correctly.
Due to a bug def_setfees(100, slow=100) would instead set immediate and
normal fees to 100. This behavior has been updated to set fees to
correct values, make the values truly optional as per documentation and
unit test this behavior.
* Fix pay() to set msatoshi, description and risk factor properly
Due to a bug pay(invoice, description='1000') resulted in payment of
1000 msatoshi instead. This was fixed and covered with tests.
* Fix named args in listpayments, listpeers and connect
* Do not pass None to methods where it is default value
* Make description on invoice and pay match.
Suggested-by: @ZmnSCPxj
* Fix dev_setfees to set slow and normal fees correctly.
Due to a bug def_setfees(100, slow=100) would instead set immediate and
normal fees to 100. This behavior has been updated to set fees to
correct values, make the values truly optional as per documentation and
unit test this behavior.
* Fix pay() to set msatoshi, description and risk factor properly
Due to a bug pay(invoice, description='1000') resulted in payment of
1000 msatoshi instead. This was fixed and covered with tests.
* Fix named args in listpayments, listpeers and connect
* Do not pass None to methods where it is default value
* Make description on invoice and pay match.
Suggested-by: @ZmnSCPxj
'rhash' is the old terminology, but 'payment_preimage' and
'payment_hash' were decided on for the BOLTs, so we should fix that here.
We still use rhash internally, but that's much easier to fix.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With the previous commit this enables py.test on travis, which should
give us some better way of hunting down bugs on travis.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was causing intermittent `rawtransactiondecode` errors see
ElementsProject/lightning#332
Reported-by: @achow101
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Also, we split the more sophisticated json_add helpers to avoid pulling in
everything into lightning-cli, and unify the routines to print struct
short_channel_id (it's ':', not '/' too).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The Dockerfile is now stored in contrib and built using the Docker
Hub. This allows us to simply pull in the finished image from the hub
instead of having to build it ourself. Should shave off about 2
minutes from the build time.
I also switched to running the individual build and check steps in
their own containers, but on the same volume, so travis can group the
commands and run them independently.
We intend to ultimately no longer use the legacy `daemon/lightningd`
and instead use `lightningd/lightningd`, so I grouped the new RPC and
the legacy RPC and implemented stubs for the new daemon.
