addresses issue #2753.
Formatting the JSON with the default parameters will escape the unicode
symbols in a way that c-lightning won't allow, leading to an exception.
Changelog-Fixed: `pylightning` now handles unicode characters in JSON-RPC requests and responses correctly.
In the c-lightning tests we have `tests/conftest.py` which annotates test
function with the outcome. If we use pyln-testing outside of the c-lightning
tree we cannot rely on that annotation being there, so we assume it passed.
Using the psycopg2-binary package means that the apropriate compiled binary
for the user platform will be shipped alongside the python binaries. Otherwise
the python bindings and the C shims would be shipped which would then require
the postgres development packages as well.
This just makes things easier, since we don't require the build dependencies.
Changelog-changed: .lightningd plugins and files moved into <network>/ subdir
Changelog-changed: WARNING: If you don't have a config file, you now may need to specify the network to lightning-cli
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Spaces just make life a little harder for everyone.
(Plus, fix documentation: it's 'jsonrpc' not 'json' subsystem).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplifies our tests, too, since we don't need a magic option to
enable io logging in subdaemons.
Note that test_bad_onion still takes too long, due to a separate minor
bug, so that's marked and left dev-only for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Printed form is always "[<nodeid>-]<prefix>: <string>"
2. "jcon fd %i" becomes "jsonrpc #%i".
3. "jsonrpc" log is only used once, and is removed.
4. "database" log prefix is use for db accesses.
5. "lightningd(%i)" becomes simply "lightningd" without the pid.
6. The "lightningd_" prefix is stripped from subd log prefixes, and pid removed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-changed: Logging: formatting made uniform: [NODEID-]SUBSYSTEM: MESSAGE
Changelog-removed: `lightning_` prefixes removed from subdaemon names, including in listpeers `owner` field.
A log can have a default node_id, which can be overridden on a per-entry
basis. This changes the format of logging, so some tests need rework.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Quite a few of the things in the LightningNode class are tailored to their use
in the c-lightning tests, so I decided to split those customizations out into
a sub-class, and adding one more fixture that just serves the class. This
allows us to override the LightningNode implementation in our own tests, while
still having sane defaults for other users.
We were relying heavily on NodeFactory to do some magic before instantiating
the Node with rpc and daemon initialized, that meant that we'd have to replace
all 3 classes when customizing the node to our needs. Moving that
initialization into the node itself means that the LightningNode class now can
be swapped out and customized, without having to wire everything else through.
`DEVELOPER=1` assumes that the binary has been compiled with developer set to
true, which might not be the case for plugin developers. Setting this to 0 by
default has no effect in c-lightning since we always at least set it in
`config.vars` but may prevent some issues outside.
We'll rewrite the tests to use this infrastructure in the next commit.
Changelog-Added: The new `pyln-testing` package now contains the testing infrastructure so it can be reused to test against c-lighting in external projects
This should not affect any consumer of the API since we just shift the actual
implementation from one side to the other, and keep aliases in place so
scripts don't break.
We also bump the version number from 0.0.7.3 to 0.7.4 which allows us to be in
sync with c-lightning itself, and remove the superfluous `0` in front.
Takes advantage of upfront-shutdown-script to permit users to
specify the close-to address for a channel at open, by adding
a `close_to` field to `fundchannel_start`.
Note that this only is in effect if `fundchannel_start` returns
with `close_to` set -- otherwise, peer doesn't
support `option_upfront_shutdown_script`.
Command format: close id [unilateraltimeout] [destination]
Close the channel with peer {id}, forcing a unilateral
close after {unilateraltimeout} seconds if non-zero, and
the to-local output will be sent to {destination}. If
{destination} isn't specified, the default is the address
of lightningd.
Also change the pylightning:
update the `close` API to support `destination` parameter
* Add compatibility with the new shortchannelid syntax with x as separator
* Add an error message in case the parse failed, instead of receveing an unrelated error from the bitcoin-cli
This is the first step to transition to a better organized python module
structure. Sadly we can't reuse the `pylightning` module as a namespace module
since having importable things in the top level of the namespace is not
allowed in any of the namespace variants [1], hence we just switch over to the
`pyln` namespace. The code the was under `lightning` will now be reachable
under `pyln.client` and we add the `pyln.proto` module for all the things that
are independent of talking to lightningd and can be used for protocol testing.
[1] https://packaging.python.org/guides/packaging-namespace-packages/
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Allow a user to select the utxo set that will be added to a
transaction, via the `utxos` parameter. Optional.
Format for utxos should be of the form ["txid:vout","..."]
___________________________________________________________________________
/ The most important plugin. Ever. Much thanks to @practicalswift and @jb55 \
\ for their improvements! /
---------------------------------------------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In contrib/bootstrap-node.sh line 7:
if type lightning-cli >/dev/null 2>&1; then
^-- SC2039: In POSIX sh, 'type' is undefined.
In contrib/startup_regtest.sh line 41:
type lightning-cli || return
^-- SC2039: In POSIX sh, 'type' is undefined.
In contrib/startup_regtest.sh line 42:
type lightningd || return
^-- SC2039: In POSIX sh, 'type' is undefined.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call. `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.
Timing out JSON calls is generally deprecated: that's the job of the
client. And the semantics of this are confusing, even to me! A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.
The transition code is awkward, but we'll manage for the three
releases until we can remove it.
The new defaults are to unilaterally close after 48 hours.
Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The helpme plugin is more comprensive, but this at least connects to a
few random nodes, and doesn't require python libraries in path or anything.
I selected the nodes from helpme.py, eliminating ones I couldn't reach.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We recently noticed that the way we unpack the call arguments for hooks and
notifications in pylightning breaks pretty quickly once you start changing the
hook and notification params. If you add params they will not get mapped
correctly causing the plugin to error out.
This can be fixed by adding a `VAR_KEYWORD` argument to the calbacks, i.e., by
adding a single `**kwargs` argument at the end of the signature. This commit
adds a check that such a catch-all argument exists, and emits a warning if it
doesn't.
It also fixes up the plugins that we ship ourselves.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Add an RPC method (not working at the moment) called
`fundchannel_continue` that takes as its parameters a
node_id and a txid for a transaction (that ostensibly has an output
for a channel)
The next commit breaks it: `if b' }\n' not in buff:` is always true since
we're about to clean up our JSON so there won't be a space. I could have
hacked the space in our JSON, but 6 months is long enough anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Tries to return the approxmost posible string of a Millisatoshi amount using
various unit representations. The function will round to an effective
number of digits. Default: 3.
```
>>> Millisatoshi("100000sat").to_approx_str()
'0.001btc'
>>> Millisatoshi("100msat").to_approx_str()
'0.1sat'
>>> Millisatoshi("10000000sat").to_approx_str()
'0.1btc'
```
We had a bit of a hand-woven mess in there, trying to inject the extra
arguments in the correct places. We now instead treat positional and keyword
calls separately and can go back to using the builtin argument binding again.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The old codes if % 1000 statement logic was simply inverted
and produced the opposite output of the intention behin it.
Before Fix:
- Millisatoshi('42sat').to_btc_str() => 0.00000042000btc
- Millisatoshi('42001msat').to_btc_str() => 0.00000042btc
After Fix:
- Millisatoshi('42sat').to_btc_str() => 0.00000042btc
- Millisatoshi('42001msat').to_btc_str() => 0.00000042001btc
Currently, when a multiplication operator is invoked that
does not result in an even integer result but a floating result,
the pylightning code will raise an exception:
Millisatoshi must be string with msat/sat/btc suffix or int
This is because the internal float result will be used as
contructor argument like this: return Millisatoshi(10000.5)
This happens especially on fee calculations where small uneven amounts
are calculated.
Millisatoshi's inner representation expected to be an int, otherwise unwanted exceptions could occur
The following example raises: TypeError: __int__ returned non-int (type decimal.Decimal)
from lightning import Millisatoshi
one_sat = Millisatoshi("1sat")
two_sats = one_sat * 2
make it easier to fire up a local test environment to try out
c-lightning.
requires bitcoind to be installed. to use, you have to run it
via `source contrib/startup_regtest.sh`, so that the aliases
are set correctly.
* Improved plugin install in docker
- All files generated by 'make install' are copied
- Fixes issues with incomplete installation.
- Example: New executables created by build but are missing in docker.
With the preceeding UTF-8 fix, I'd like to detect UTF-8 support. But
AFAICT Python doesn't have a standard way of doing version exposure.
So I added __version__, but now we need to make sure it matches. I
used the hackiest possible method.
[ Christian Decker fixed version to be sane, so previous comment no longer
applies! --RR ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. We need to read in as a byte string, then decode into utf8 once we
have a marker. Otherwise we seem to mangle it horribly, and we
might have a bad utf8 string anyway.
2. We need to suppress the JSON \u escapes on output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than using LightningJSONDecoder's implicit "field name and
value ends in msat, try converting to Millisatoshi", we do it to
parameters using type annotations.
If you had a parameter which was an array or dict itself, we don't
delve into that, but that's probably OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I originally converted input JSON naively into Millisatoshi, and the
result was a strange failure in Millisatoshi.__eq__.
It seems this is because inspect._empty.__eq__(Millisatoshi) raises
NotImplemented, and so it tries Millisatoshi.__eq__(inspect._empty)
which doesn't like it.
'is' is the correct test here, AFAICT, and doesn't suffer from these
problems.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some JSON functions want a *class*, not just a hook, so provide one.
To make it clear that we want an encoding *class* and a decoding *object*,
rename the UnixDomainSocketRpc encode parameter to encode_cls.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we can't marshall an object into JSON, the exception causes a deadlock
and we don't get any results.
Instead of deadlocking, our failure now is:
lightning.lightning.RpcError: RPC call failed: method: echo, payload: {'msat': 17msat}, error: Error while processing echo: TypeError("Object of type 'Millisatoshi' is not JSON serializable",)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I tried annotating the plugin-millisatoshis.py plugin, and it failed like so:
plugin-millisatoshis.py Killing plugin: "getmanifest" result is not an object: {"jsonrpc": "2.0", "id": 1, "error": "Error while processing getmanifest: ValueError(\'Function has keyword-only parameters or annotations, use getfullargspec() API which can support them\',)"}'
So, let's do that!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the same deprecation, but one level up. For the moment, we
still support invoices with a `h` field (where description will be
necessary) but that will be removed once this option is removed.
Note that I just changed pylightning without backwards compatibility,
since the field was unlikely to be used, but we could do something
more complex here?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this the RPC will fail to continue buffering if the response does not
fit in the first read, and if we don't switch over to the non-compat
mode. This was introduced by our mitigation of the UTF-8 misalignment, but I
missed this path.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
These weren't checked by CI yet, and they are really short so I just added
them to the check-python target.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This indicates that the method or hook will accepts a request
parameter, and will use that to return the result or raise an
exception instead of returning the return value. This allows the hook
or method to stash the incomplete request or pass it around, without
blocking the JSON-RPC interface.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This isn't a problem for now since we don't support multithreading,
and only allow synchronous calls, but eventually this'll become
important.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We well need this in the next commit to be able to return from an
asynchronous call. We also guard stdout access with a reentrant lock
since we are no longer guaranteed that all communication happens on
the same thread.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Sending around unnamed tuples is bound to cause some issues sooner or
later, so we just create a quick class that holds all the information
about a plugin method.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Little point having users handle the postfixes manually, this
translates them, and also allows Millisatoshi to be used wherever an
'int' would be previously.
There are also helpers to create the formatting in a way c-lightning's
JSONRPC will accept.
All standard arithmetic operations with integers work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We read a JSON message from the buffer, after converting it from raw bytes to
UTF-8, and returning the remainder of the byte array back to the
caller. However the return value of `raw_decode` refers to symbols in the
UTF-8 decoded string, not the raw bytes underlying byte-array, which means
that if we have multi-byte encoded UTF-8 symbols in the byte-array we end up
with a misaligned offset and will return part of the message as
remainder. This would then end up being interpreted as the result of the next
call.
This could not be exploited currently since we use a socket only for a single
JSON-RPC call and will close the connection afterwards, but since we want to
eventually recycle connections for multiple calls, this could have been very
dangerous.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Reported-by: Corné Plooy <@bitonic-cjp>
The next patch wants to decorate the methods with a compulsory
'usage' option, which doesn't make sense for init. So I wanted
to change the init to its own decoration.
Made-to-work-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was failing the docker hub builds, since the git-config retains
an absolute path to the worktree location when cloning. Copying it
over from the host system means that this path now points to a
non-existent location, which then interfered with the submodule
initialization.
This fixes it by not using the copy directly, but rather it creates a
clean clone from the copied location, including a submodule init.
Signed-off-by: Christian Decker <@cdecker>
Logging an empty line (without newline character) would raise an
Exception due to out of bounds check.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Valgrind seems to be slowing the pay-plugin down enough for the 10
seconds timeout to get triggered on a semi-regular basis.
Reported-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Instead of creating a new map I opted to re-use the Plugin.methods
map, since the semantics are really similar and we don't allow
duplicates. The only difference is in how they are announced to
lightningd, so we use an enum to differentiate rpcmethods from hooks,
since only the former will get added to the JSON-RPC dispatch table in
lightningd.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
