Requires us to update to latest lnproto which is now using the most up
to date python-bitcoinlib, as well as updating our python lock files
(which pin the grpcio deps, because of locking problems h/t @cdecker)
This happens with deprecated-apis and listconfigs, breaking some
python plugins!
Fixes: #5546Fixes: #5563
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
First, how we record "our_funds" and then apply pushes vs lease_fees
(for liquidity ad buys/sales) was exactly opposite.
For pushes we were reporting the total funded into the channel, with the
push representing how much we'd later moved to the peer.
For lease_fees we were rerporting the total in the channel, with the
push representing how much was already moved to the peer.
We fix this (from a view perspective) by re-adding lease fees to what's
reported in the channel funding totals. Since this is now new behavior
(for leased channel values), we added new fields so we can take the old
field names thru a deprecation cycle.
We also make it possible to differentiate btw a push and a lease_fee
(before they were all the same), by adding to new fields to `listpeers`:
`fee_paid_msat` and `fee_rcvd_msat`.
This allows us to avoid math in the bookkeeper, instead we just pick
the numbers out directly and record them.
Fixes#5472
Changelog-Added: JSON-RPC: `listpeers` now has a few new fields for `funding` (`remote_funds_msat`, `local_funds_msat`, `fee_paid_msat`, `fee_rcvd_msat`).
Changelog-Deprecated: JSON-RPC: `listpeers`.`funded` fields `local_msat` and `remote_msat` are now deprecated.
Some tests need to inspect it, but most don't, and I suspect I'm missing some
error messages due to this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The CLN API is rather strict about the fact that we should skip
providing a field whenever it is empty. Checking for `is_none` would
still include empty arrays.
Changelog-Fixed cln-rpc: Optional empty arrays will no longer be serialized in requests
This allows us to re-use existing tests (assuming the call and fields
are covered by `cln-rpc` and `cln-grpc`) to test the full roundtrip
from test over the grpc interface to the json-rpc interface and back
again.
You can switch to the grpc interface by setting the `CLN_TEST_GRPC`
environment variable to 1, but for now only very few shims are
implemented (due to the non-generated nature of LightningRpc).
To test the grpc interface we'll want to emulate the JSON-RPC
interface as best we can, hence when talking to the grpc interface we
want to convert back into a parsed JSON format as LightningRpc would
have returned it. This is just the simplest way of closing the loop
here:
```
pyln-testing --grpc-> cln-grpc --grpc2json
^ |
| v
| JSON-RPC
| |
TEST v
^ CLN
| |
| v
pyln-testing <-grpc2py-- cln-grpc <- json2grpc
```
These may eventually end up in pyln-client, as they allow talking to
the GRPC interface exposed by cln-grpc, however for now they are used
for testing only. Once we have sufficient API and test coverage we can
move them and leave imports in their place.
We were overriding the name right when loading, which is bad since in
some languages we use the method name as tag in the requests, thus
renaming causes us to call something that isn't defined.
Changelog-Fixed: cln-rpc: Fixed a naming mismatch for `ConnectPeer` causing `connectpeer` to be called on the JSON-RPC
This was introduced to allow creating a shared secret, but it's better to use
makesecret which creates unique secrets. getsharedsecret being a generic ECDH
function allows the caller to initiate conversations as if it was us; this
is generally OK, since we don't allow untrusted API access, but the commando
plugin had to blacklist this for read-only runes explicitly.
Since @ZmnSCPxj never ended up using this after introducing it, simply
remove it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSONRPC: `getsharedsecret` API: use `makesecret`
This has been replaced with better rust bindings that can then be
consumed via pyo3, consolidating the C interface in a portable
wrapper.
Changelog-Removed: libhsmd: Removed the `libhsmd_python` wrapper as it was unused
Since we now log directly, we don't prepend the prefix ourselves, making it really
hard to tell *which* lightningd the log applies to!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
routing.c now flags rate-limited gossip as it enters the gossip_store but
makes use of it in updating the routing graph. Flagged gossip is not
rebroadcast to gossip peers.
Changelog-Changed: gossipd: now accepts spam gossip, but squelches it for
peers.
This will be used to decouple internal use of gossip from what is
passed to gossip peers. Updates GOSSIP_STORE_VERION to 10.
Changelog-Changed: gossip_store updated to version 10.
I had to parse quite a few of these files debugging zeroconf, so I
thought it might be nice to have direct access here.
Changelog-Added: pyln-testing: Added utilities to read and parse `gossip_store` file for nodes.
This will eventually enable us to specify 0 for zeroconf channels.
Changelog-Added: JSON-RPC: Added `mindepth` argument to specify the number of confirmations we require for `fundchannel` and `multifundchannel`
Ok this should be fixed the following stack trace
```
2022-06-29T14:19:41.183Z DEBUG lightningd: Command returned result after jcon close
------------------------------- Valgrind errors --------------------------------
Valgrind error file: valgrind-errors.55581
==55581== Syscall param statx(file_name) points to unaddressable byte(s)
==55581== at 0x4B0188E: statx (statx.c:29)
==55581== by 0x1133481: std::sys::unix::fs::try_statx (weak.rs:178)
==55581== by 0x11265E0: std::fs::buffer_capacity_required (fs.rs:851)
==55581== by 0x112675B: <std::fs::File as std::io::Read>::read_to_string (fs.rs:644)
==55581== by 0x10DACA8: num_cpus::linux::Cgroup::param (linux.rs:214)
==55581== by 0x10DAB39: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==55581== by 0x10DA9C2: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==55581== by 0x10DA5A1: num_cpus::linux::load_cgroups (linux.rs:149)
==55581== by 0x10DA23D: num_cpus::linux::init_cgroups (linux.rs:129)
==55581== by 0x10DCDC8: core::ops::function::FnOnce::call_once (function.rs:227)
==55581== by 0x10DC749: std::sync::once::Once::call_once::{{closure}} (once.rs:276)
==55581== by 0x21EE89: std::sync::once::Once::call_inner (once.rs:434)
==55581== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==55581==
==55581== Syscall param statx(buf) points to unaddressable byte(s)
==55581== at 0x4B0188E: statx (statx.c:29)
==55581== by 0x1133481: std::sys::unix::fs::try_statx (weak.rs:178)
==55581== by 0x11265E0: std::fs::buffer_capacity_required (fs.rs:851)
==55581== by 0x112675B: <std::fs::File as std::io::Read>::read_to_string (fs.rs:644)
==55581== by 0x10DACA8: num_cpus::linux::Cgroup::param (linux.rs:214)
==55581== by 0x10DAB39: num_cpus::linux::Cgroup::quota_us (linux.rs:203)
==55581== by 0x10DA9C2: num_cpus::linux::Cgroup::cpu_quota (linux.rs:188)
==55581== by 0x10DA5A1: num_cpus::linux::load_cgroups (linux.rs:149)
==55581== by 0x10DA23D: num_cpus::linux::init_cgroups (linux.rs:129)
==55581== by 0x10DCDC8: core::ops::function::FnOnce::call_once (function.rs:227)
==55581== by 0x10DC749: std::sync::once::Once::call_once::{{closure}} (once.rs:276)
==55581== by 0x21EE89: std::sync::once::Once::call_inner (once.rs:434)
==55581== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==55581==
--------------------------------------------------------------------------------
Leaving base_dir /tmp/ltests-hzt9ppqp intact, it still has test sub-directories with failure details: ['test_peers_1']
```
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Even though we generally wait until a node has seen the gossip,
that doesn't mean that connectd has processed it! This means when
we connect it may still send us "old" gossip.
So we set the OPT_GOSSIP_QUERIES bit, which means don't send until we
ask. But now it sends us WIRE_QUERY_CHANNEL_RANGE, so everyone needs
to filter that out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I saw another "only_one()" fail on alias checking: it's not
entirely clear to me with the more aggressive sending of
own gossip, that we necessarily process in order, so we might
not have actually seen all channels just because we saw
the farthest one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Get it to log direct to stdout, so we see what's happening *as it
happens* rather than as we read it. We could restore the thread we
were using before, but that added more problems than it solved.
This means that we need the hsm password prompts in the log though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some flakes are caused by weird races in this code. Plus, if we
get things to write straight to files, we might see things in
there on post-mortem which happen after the python runner exits.
It's a bit less efficient, but much simpler. Let's see if it helps!
Some tests need a rework now, since we don't get a failure (except
eventual timeout), but they're simpler.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use ephemeral_port_reserve to grab ports, but this can fail when we
restart a node, since the port can be reallocated at that point.
Attempt to overcome this using a global reserved list (is there a
neater way?).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This happens under CI, but it's not informative. Sleep and retry.
Also, "except (OSError, Exception)" does not seem to do what you'd think:
this clause never gets run.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Over time, it has cost us more developer cycles than it has gained.
It has hidden intermittant bugs, and allowed cruft to accumulate:
when we eventually tried to figure out what was going wrong, the
actual change which caused it was now stale and forgotten.
This was a particular bane during the connectd rewrite, and I
worked through some issues which had occurred before, but were not
more likely.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
So this was quite a journey:
- We want relative depdendencies (using the `path` argument) whenever
developing locally. Otherwise we would have to install each
dependency every time we change a single character, which
undoubtedly would cause us to waste time trying to debug an issue
just because we forgot to install.
- When publishing however we want to rely on the version number,
since the repo context gets lost upon publishing, and path
dependencies cause failures.
The solution then it seems is to use `dev-dependencies` (not that
surprising once you find it) with relative paths, so that `poetry
install` uses these over the normal dependencies (no idea how they
dedup them) and use `dependencies` when publishing. The paths are
still in there when publishing, but `pip install` ignores them.
I checked that `poetry install` from an unrelated project doesn't
accidentally use the path dependencies, even when adding them as
dev-dependencies. This should hopefully also allow installing them
as a repo link, though I can't test that right now.
We should be using amount_msat always. Many tests were not. Plus,
deprecating it simplifies the code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSONRPC: `sendpay` `route` elements `msatoshi` (use `amount_msat`)
This prepares for when they start being u64, not strings with msat appended.
This has a strange side effect on our schema: despite the name,
decodepay's `fee_base_msat` is actually a u64, which we now convert to
msat on decode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is consistent with our output changes, and increases consistency.
It also keeps future sanity checks happy, that we only use JSON msat
helpers with '_msat' fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `invoice`, `sendonion`, `sendpay`, `pay`, `keysend`, `fetchinvoice`, `sendinvoice`: `msatoshi` argument is now called `amount_msat` to match other fields.
Changelog-Deprecated: JSON-RPC: `invoice`, `sendonion`, `sendpay`, `pay`, `keysend`, `fetchinvoice`, `sendinvoice` `msatoshi` (use `amount_msat`)
Changelog-Added: msggen: introduce chain of responsibility pattern to make msggen extensible
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Switching to poetry, and deprecating python 3.6, made things a bit
more tricky. Sadly we'll not be able to build jammy, as its support is
missing in the tag tarball, but it'll be there for the next release.
Mostly comments and docs: some places are actually paths, which
I have avoided changing. We may migrate them slowly, particularly
when they're user-visible.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `pay` has `description` parameter, will be required if bolt11 only has a hash.
Changelog-Deprecated: JSON-RPC: `pay` for a bolt11 which uses a `description_hash`, without setting `description`.
This is what LND does, and it's better for upper layers than trying to
twist our maxfeepercent / exemptfee heuristics to suit.
(I don't remember who complained about this, sorry!)
I'm doing this now because I want to add YA parameter next!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `pay` has new parameter `maxfee` for setting absolute fee (instead of using `maxfeepercent` and/or `exemptfee`)
It's schema definition is weirdly asymmetric, with variants dependent
on another fields' value. Need to decide if we want to either
hand-code a superset or make a more complex decoding, but definitely
not something we'd want the generator to be able to do.
Types are fixed, in particular:
* rename "OutputDesc" to more consistent "outputdesc".
* rename "utxo" to more consistent "outpoint".
* it's "boolean" not "bool".
* "number" means int or float, usually it should be u32.
Specific commands:
* close `id` can be by channel id, scid.
* close `feerange` is a feerate type.
* datastore/deldatastore/listdatastore `key` can be singleton.
* delexpiredinvoice: `maxexpirytime` is not required, is a u64.
* invoice/delinvoice/listinvoice `label` can be an integer
* fundpsbt: many fields are u32 not number (JSON for int or float).
* invoice: `msatoshi` can be "any".
* invoice: `expiry` has a type (now must be numeric).
* invoice: `exposeprivatechannels` can be bool or array of scids.
* invoice: `deschashonly` added
* keysend: there's no "float" type, use "number" or "u32" etc.
* keysend: `routehints` is a valid arg, as is `extratlvs` (EXPERIMENTAL_FEATURES)
* listdatastore: `key` is not required.
* newaddr: `addresstype` can be "all"
* pay: `exemptfee` is "msat", new fields `locaofferid` and `exclude`
* sendonion: was mis-formatted, missed `localofferid` and `groupid` fields.
* sendpay: add `localofferid` and `groupid` params.
* signpsbt: add `signonly` param.
* txprepare "outptus" typo.
* waitsendpay: add `groupid` and fix `partid` type.
* withdraw: `destination` is a bitcoin address, not a pubkey.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are useful for requests:
1. "outpoint": <txid>:<outnum>
2. "feerate": strings or a number
3. "outputdesc": bitcoin-style addresses-as-keys
4. "msat_or_all": amount or "all"
4. "msat_or_any": amount or "any"
5. "short_channel_id_dir": scid with /0 or /1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make it always a number; this makes the JSON request specification
simpler. We allowed a number since v0.10.1.
(reserve=True is the default anyway, so usually it can be omitted:
reserve=False becomes reserve=0).
Changelog-Deprecated: JSON-RPC: `fundpsbt`/`utxopsbt` `reserve` must be a number, not bool (for `true` use 72/don't specify, for `false` use 0). Numbers have been allowed since v0.10.1.
This is likely inherited from bitcoind, and a bit awkward for us, so
we parse it into a classic struct, but serialize it back into the
bitcoind format when talking to the RPC.
The CI fails with the error
```
EnvCommandError
Command ['/usr/bin/python3', '-m', 'pip', 'install', '--no-deps', '-U', '/root/.cache/pypoetry/artifacts/07/6f/ab/ca33bde7c6751a5ad8d13495b766891cd70e61786112885733ce9b0562/cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl'] errored with the following return code 1, and output:
ERROR: cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.
at ~/.local/lib/python3.8/site-packages/poetry/utils/env.py:1195 in _run
1191│ output = subprocess.check_output(
1192│ cmd, stderr=subprocess.STDOUT, **kwargs
1193│ )
1194│ except CalledProcessError as e:
→ 1195│ raise EnvCommandError(e, input=input_)
1196│
1197│ return decode(output)
1198│
1199│ def execute(self, bin, *args, **kwargs):
```
The solution is to upgrade the pip version as suggested in https://github.com/python-poetry/poetry/issues/2688#issuecomment-937837619
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>x
Means that field is now optional in JSON output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `delinvoice` has a new parameter `desconly` to remove description.
LNURL wants this so they can include images etc in descriptions.
Replaces: #4892
Changelog-Added: JSON-RPC: `invoice` has a new parameter `deschashonly` to put hash of description in bolt11.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Either because lightningd tells us it wants to talk, or because the peer
says something about a channel.
We also introduce a behavior change: we disconnect after a failed open.
We might want to modify this later, but we it's a side-effect of openingd
not holding onto idle connections.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Turns out that the pyln-proto dependency in the bolt packages is only
needed for testing, not for production. Making it a dev-dependency
means it isn't considered in resolution anymore.
Since the bolt, testing and client packages are to be used outside
from the project we can't use relative dependencies either, so make
then dependent on the version on PyPI. This also means we had to push
a couple of updated to PyPI.
Changelog-None
Suggested by @m-schmook, I realized that if we append it later I'll
never get it right: I expect parameters min and max, not max and min!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: you can now alter the `htlc_minimum_msat` and `htlc_maximum_msat` your node advertizes.
Based on setchannelfee, but expanded to allow setting max htlc amount (and others
in future?).
The main differences:
1. It doesn't change values which are not specified (that would be hard to
add fields to!)
2. It says exactly what all values are in any potentially changed channels.
Changelog-Added: JSON-RPC: new `setchannel` command generalizes `setchannelfee`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I thought about fixing them up, but really these should be in
lnprototest anyway. Turns out they're from the spec, so we should
actually fix them up there.
I moved the vector files into contrib/pyln-proto, since that still
needs them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We are inferring the field numbers on the fly, which isn't really
compatible with the way GRPC field numbers work, i.e., they must be
stable while the IDL file evolves. So far when a field was added in
the middle of a struct or removed all subsequent fields would get
renumbered, essentially breaking any client that was using the old
scheme.
We now add a meta file `.msggen.json` that keeps track of the numbers
assigned so far, so they can be reused, and new ones can be generated
not to conflict with existing ones. This file is intentionally kept
generic, so other generators can add more information that has to be
managed across runs.
Changelog-None
We must use `None` for default arguments since otherwise they aren't
filtered out when serializing the request. In addition default
arguments to functions are initialized once and, if mutable, could
persist internal changes across function calls.
Changelog-None
`listpeers` is a rather deeply nested structure which has a couple of
caveats, namely that we use the same enum multiple times, which causes
naming clashes. So we truncate the state_changes[]. We can later map
them if needed, but it'll get much easier once we have an abstract
model description that isn't JSON schema, which unrolls all types,
causing us to generate those enums multiple times.
There is at least one clash with a built-in for the grpc server trait,
namely `connect` so we add support for renaming a method when
generating the scaffolding
The server doesn't do much more than unwrapping the request from its
grpc envelope, convert it into the matching JSON-RPC binding struct,
initiate the RPC connection (until we have connection pooling), and
then forwards the converted request. The inverse then happens for the
result.
Unfortunately we can't do any smart parsing here since
wiregen does not support switch/type cases for different
substructure unions yet. So just give us a pointer we can use.
But this requires a watch-only wallet, and python-bitcoinlib doesn't support
multiple wallets, so we need to unload the original one, but then we need
to generate a block, so that can't generate a new address, so we need
an address arg to generate_block.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We build an in-memory model of what the API should look like, which
will later be used to generate a variety of bindings. In this PR we
will use the model to build structs corresponding to the requests and
responses for the various methods.
The JSON-RPC schemas serve as ground-truth, however they are missing a
bit of context: methods, and the request-response matching (as well as
a higher level grouping we'll call a Service). I'm tempted to create a
new document that describes this behavior and we could even generate
the rather repetitive JSON schemas from that document. Furthermore
it'd allow us to add some required metadata such as grpc field
numbering once we generate those bindings.
Changelog-Added: JSON-RPC: A new `msggen` library allows easy generation of language bindings for the JSON-RPC from the JSON schemas
If we fund a channel between two nodes, then mine all the blocks to
announce it, any other nodes may see the announcement before the
blocks, causing CI to complain about "bad gossip":
```
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Ignoring future channel_announcment for 113x1x1 (current block 112)
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 113x1x1/0
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_CHANNEL_UPDATE before announcement 113x1x1/1
lightningd-4: 2022-01-25T22:33:25.468Z DEBUG 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e-gossipd: Bad gossip order: WIRE_NODE_ANNOUNCEMENT before announcement 032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e
```
Add a new helper for this case, and use it where there are more than 2 nodes.
Cleans up test_routing_gossip and a few other places which did this manually.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This seems to trigger now, especially on PostgresQL (maybe it's faster
to process blocks?).
e.g. test_closing_simple() hangs in close(), because the close is unilateral
because the HTLC timed out, so it's waiting for a block (other lines removed):
```
lightningd-1: 2022-01-12T00:33:46.258Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#1: peer_out WIRE_COMMITMENT_SIGNED
lightningd-1: 2022-01-12T00:33:46.278Z DEBUG lightningd: close_command: timeout = 172800
2022-01-12T01:03:36.9757201Z lightningd-2: 2022-01-12T00:33:46.384Z DEBUG lightningd: Adding block 104: 73ffa19d27d048613b2731e1682b4efff0dc226807d8cc99d724523c2ea58204
2022-01-12T01:03:36.9759053Z lightningd-2: 2022-01-12T00:33:46.396Z DEBUG lightningd: Adding block 105: 44fd06ed053a0d0594abcfefcfa69089351fc89080826799fb4b278a68fe5c20
2022-01-12T01:03:36.9760865Z lightningd-2: 2022-01-12T00:33:46.406Z DEBUG lightningd: Adding block 106: 0fee2dcbd1376249832642079131275e195bba4fb49cc9968df3a899010bba0f
2022-01-12T01:03:36.9762632Z lightningd-2: 2022-01-12T00:33:46.418Z DEBUG lightningd: Adding block 107: 7f24f2d7d3e83fe3256298bd661e57cdf92b058440738fd4d7e1c8ef4a4ca073
2022-01-12T01:03:36.9773411Z lightningd-2: 2022-01-12T00:33:46.429Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: peer_in WIRE_REVOKE_AND_ACK
2022-01-12T01:03:36.9794707Z lightningd-2: 2022-01-12T00:33:46.437Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: Commits outstanding after recv revoke_and_ack
2022-01-12T01:03:36.9788197Z lightningd-2: 2022-01-12T00:33:46.433Z DEBUG lightningd: Adding block 108: 283b371fb5d1ef42980ea10ab9f5965a179af8e91ddf31c8176e79820e1ec54d
2022-01-12T01:03:36.9799347Z lightningd-2: 2022-01-12T00:33:46.439Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-channeld-chan#1: HTLC 0[REMOTE] => RCVD_REMOVE_REVOCATION
2022-01-12T01:03:36.9808057Z lightningd-2: 2022-01-12T00:33:46.447Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Peer permanent failure in CHANNELD_NORMAL: Fulfilled HTLC 0 RCVD_REMOVE_REVOCATION cltv 109 hit deadline
```
This is because `pay` returns from l1 when it has the preimage, not
when the HTLC is fully resolved. Add a helper for this, and call it
at the end of the pay test helper. We might need this elsewhere
though!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This loads up 20MB of plugins temporarily; we seem to be getting OOM
killed under CI and I wonder if this is contributing.
Doesn't significantly reduce runtime here, but I have lots of memory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When downloading a python package from the PyPI repository the links
where pointing to a non-existent parent directory, thus breaking the
packages. The files don't ever change, and are really simple, so let's
just materialize them.
This was measured as a 95th percentile in our rough testing, thanks to
all the volunteers who monitored my channels.
Fixes: #4761
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setchannelfee` gives a grace period (`enforcedelay`) before rejecting old-fee payments: default 10 minutes.
This is best-practice (to ensure prototypes match up), but there were a
few places we didn't (at least, directly). Make it a requirement,
either of form "foo.h" or <dir/foo.h>.
The noise is the change to our print templates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
On my machine it should have produced:
/tmp/ltests-1p0v5z1j/.locks/lock-1631090663.1066182
but instead it produced:
/private/tmp/ltests-1p0v5z1j/.locks/lock-1631090663.105848
The mismatch resulted in a hang inside flock.
Path .resolve() seems to be causing problems for others as well. It appears the library was not meant to handle complex path situations and isn't maintained as such (see link reference).
Since we have already built a full path here anyway, the call to .resolve() appears redundant.
Tested on python 3.9.6 on my Mac OS X 11.4 (20F71), Xcode 12.5.1 (12E507)
Relevant discussion: https://discuss.python.org/t/pathlib-absolute-vs-resolve/2573
This affects the range we offer even without quick-close, but it's
more critical for quick-close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSONRPC: `close` now takes a `feerange` parameter to set min/max fee rates for mutual close.
This only happens when a deletion is added by a running gossipd, so
we put a deletion at the end of the store to test it.
mypy noticed that this code was nonsensical, so clearly untested.
The testing noticed that making a nodeid from a string was also buggy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Mainly fixing type annotations, but some real fixes:
1. GossmapHalfchannel.from_str() should be a classmethod.
2. update_channel had weird, unusable default values (fields can't be NULL,
since we use it below).
[ There was one more occurence where isinstance should be used above
type() == xyz comparison. -- MS ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Do not mix bytes and GossmapNodeId when accessing Gossmap.nodes dicts.
Therefore the definion got GossmapNodeId also needed to be pulled to the
beginning of the file.
This reads the `gossip_store_channel_amount` that always follows the
`channel_announcement` messages. Therefore it uses an internal variable
_last_scid to know what channel has been added last time.
This is more efficient than converting them all to Pubkeys: about 3.8
seconds vs 5.4 seconds. Usually treating them as raw bytes is what we
want anyway.
[ Fixup by Michael Schmoock <michael@schmoock.net> ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It doesn't do much work, but it does parse the gossmap file and extract
nodes and channels.
[ Fixup by Michael Schmoock <michael@schmoock.net> ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As suggested in this issue https://github.com/python/mypy/issues/7484#issuecomment-529363083 we skip following import becuase with the recent version of mypy the __init__.py file make confusione inside the analysis (in the python issue it is unclear the main motivation of this issue. At list unclear to me).
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Without this, we cannot pip install pylightning and pyln-client in the
same environment anymore, as it tries to pull in an incompatible version
of pyln-client.
Changelog-None
The issue is that the new keyword `force_lease_closed` was being set
even if the user didn't specify it, which results in breakage if this
new pyln version talks to older c-lightning nodes that don't have this
keyword yet. By setting the default to `None` it gets filtered out if
the user has not explicitly set it, but still retains the `True` /
`False` values if they did.
Changelog-None Issue is not present in released versions
Since we now use 'compact_lease' to gate an open (if the rates have
changed, we fail), we no longer need to rely on query rates for figuring
things out, so we make it dev-only.
Changelog-Changed: JSON-API: queryrates is now developer only
We need to know what the lease we're expecting is. To do this
we pass around the hex encoded portion of the wire format.
We can use this passed in expected lease rates to confirm that the peer
is, in fact, using the same rates as what we have currently.
Changelog-Added: JSON-RPC: fundchannel, multifundchannel, and openchannel_init now accept a 'compact_lease' for any requested funds
By default, we won't close a channel that we leased to a peer.
You can override this with the `force_lease_closed` flag.
Changelog-Added: JSON-RPC: close now has parameter to force close a leased channel (option_will_fund)
Using a 'feestep' is more restrictive than you'd want, instead we
enforce that the next feerate must be at least 1/64th more than the
last, but put no upper limit on it
Includes update to lnprototest changes
Contributed-By: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-EXPERIMENTAL: Protocol: Replaces init_rbf's `fee_step` for RBF of v2 opens with `funding_feerate_perkw`, breaking change
We would fail even if a process exited cleanly after having the line we
were looking for, because we checked if it died before reading the logs.
Co-Authored-by: Daniela Brozzoni <daniela@revault.dev>
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
It sometimes fails because the two race, and sometimes because there's
randomness, but it generally works (and doesn't fail systemically).
We remove this before the final merge.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It sometimes fails because the two race, and sometimes because there's
randomness, but it generally works (and doesn't fail systemically).
We remove this before the final merge.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It sometimes fails because the two race, and sometimes because there's
randomness, but it generally works (and doesn't fail systemically).
We remove this before the final merge.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This test takes 695 seconds, because fundwallet waits for the wallet to
notice the tx, which takes 60 seconds if not DEVELOPER. Do all the waiting
at once, and this speeds the test up to 153 seconds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds our first (basic) schema, and sews support into pyln-testing
so it will load schemas for any method for doc/schemas/{method}.schema.json.
All JSON responses in a test run are checked against the schema (if any).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not an API break: reserve=true|false still works for fundpsbt and utxopsbt,
but we also allow a raw number in there.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Tests that will only run when !EXPERIMENTAL_DUAL_FUND:
@pytest.marker.openchannel('v1')
def test_...()
Tests that will only run when EXPERIMENTAL_DUAL_FUND:
@pytest.marker.openchannel('v2')
def test_...()
Users are more upset recently with the cost of unilateral closes
than they are the risk of being cheated. While we complete our
anchor implementation so we can use low fees there, let's
get less aggressive (we already have 34 or 18 blocks to close
in the worst case).
The changes are:
- Commit transactions were "2 CONSERVATIVE" now "6 ECONOMICAL".
- HTLC resolution txs were "3 CONSERVATIVE" now "6 ECONOMICAL".
- Penalty txs were "3 CONSERVATIVE" now "12 ECONOMICAL".
- Normal txs were "4 ECONOMICAL" now "12 ECONOMICAL".
There can be no perfect levels, but we have had understandable
complaints recently about how high our default fee levels are.
Changelog-Changed: Protocol: channel feerates reduced to bitcoind's "6 block ECONOMICAL" rate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We will eventually start emitting and dispatching custom notifications
from plugins just like we dispatch internal notifications. In order to
get reasonable error messages we need to make sure that the topics
plugins are asking for were correctly registered. When doing this we
don't really care about whether the plugin that registered the
notification is still alive or not (it might have died, but
subscribers should stay up and running), so we keep a list of all
topics attached to the `struct plugins` which gathers global plugin
information.
You still shouldn't do this (you could get some transient failures),
but at least you have a decent chance if you reinstall over a running
daemon, instead of getting confusing internal errors if message
formats have changed.
Changelog-Added: lightningd: we now try to restart if subdaemons are upgraded underneath us.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #4346
e.g. in test_closing_id we can get a spend from the first (closed) channel
in the same block as the open of the second. Half the time, we'll choose
the wrong one as scid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This centralizes the setup.py file, and parametrizes it so it can
auto-detect which bolt we are building. It also uses trick 3 from [1]
to avoid importing the package itself during the manifest creation,
which'd cause an import error due to missing dependencies.
[1] https://packaging.python.org/guides/single-sourcing-package-version/
This would lead to errors about missing dependencies when attempting
to install using `pyhon setup.py install`. This is because the
`setup.py` file effectively is the manifest file used to discover
which dependencies are needed, so when using it to detect dependencies
we obviously don't have them yet.
See https://packaging.python.org/guides/single-sourcing-package-version/
