There are quite some things we want to generate from the schema
definitions, both inside and outside of CLN itself. By bundling the
schemas into the library we can make use of the tooling without
running in the CLN source tree. This is in part used to generate some
of the interfaces in Greenlight.
Changelog-None
We still want to test non-anchor channels, as we still support them, but
we've made it non-experimental. To test non-anchor channels, we
use dev-force-features: -23.
Changelog-Added: Protocol: `option_anchors_zero_fee_htlc_tx` enabled, no longer experimental.
Changelog-Changed: Config: `experimental-anchors` now does nothing (it's enabled by default).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Header from folded patch 'fixup!_options__make_anchors_enabled_by_default,_ignore_experimental-anchors.patch':
fixup! options: make anchors enabled by default, ignore experimental-anchors.
The `amount` field is intended to be either a native unit (`msat`,
`sat` or `btc`) or it can also be a non-native unit that needs to be
converted into native when creating the invoice. As such limiting the
`amount` to only be native is very restrictive.
We weakened this progressively over time, and gossip v1.5 makes spam
impossible by protocol, so we can wait until then.
Removing this code simplifies things a great deal!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer ratelimit gossip messages by channel, making our code far simpler.
We never enabled it, because we seemed to be eliminating valid
channels. We discard zombie-marked records on loading.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If $BITCOIN_BIN is empty and bitcoin-cli or bitcoind are not found
startup_regtest will shutdown and display a corresponding error message.
Similarly if lightning-cli and lightningd are not found.
This isuseful to find completely dead channels which are worthy of
closing.
Changelog-Added: JSON-RPC: `listpeerchannels` field `last_stable_connection` showing when we last held an established channel for a minute or more.
Changelog-Added: JSON-RPC: `listclosedchannels` field `last_stable_connection` showing when we last held an established channel for a minute or more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Let's tell the caller what channel_type they got!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel`, `fundchannel_start` and `openchannel_init`: new field `channel_type`.
And add a request schema for multifundchannel.
Changelog-Added: JSON-RPC: `fundchannel` and `multifundchannel` now take an optional `channel_type` parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add request schemas for openchannel_init and fundchannel_start.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel_start` and `openchannel_init` now take an optional `channel_type` parameter.
It fails because this is an array of enum values, which is a
combination we hadn't seen before. Replacing this with a manually
managed enum, since it is likely we'll want to reuse it in the future.
Type-checking in Python is so loose you could already do this, but this updates the
mypy type annotations.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Updating this every release was just busywork, and it turns out we don't actually
care: if something is marked deprecated we want to make it optional, whenever
it is for, and the only real test is if it was added before our lowest-supported
version we can consider it non-optional.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Seriously, it's taproot time, let's get rid of p2sh wrapped segwit.
Changelog-Removed: wallet: removal of p2sh-segwit addresses; newaddr won't issue them, we won't watch them for new funds (deprecated in *23.02*)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
One limitation of the `call_typed`-method was that it could not be used
with types defined outside of this crate. (See dependency on
`IntoRequest`)
This is useful for types that are not (yet) defined in this crate. A
possible case is an rpc-method that is defined in a plug-in that is
external to core-lightning or any method which isn't yet a part of the
`msggen`-script.
I've implemented a `TypedRequest` trait to make it work.
PS: This change is breaking. Users of `call_typed` must import
`cln_rpc::models::TypedRequest` instead of
`cln_rpc::models::IntoRequest`
We used to use the zip archive, which comes with some baggage,
especially for some of the submodule-based dependencies. Using `git
clone` ensures that we have a clean snapshot, based on the latest
commit, and we can skip some of the wildcard operations on zip files.
Changelog-None
Now _msat fields are all integers (last conversion 23.08) we can simply
leave them alone, rather than trying to convert them.
And for turning Millisatoshi into JSON, we simply globally replace the
default encoding function to try ".to_json()" on items, which allows
anything to be marshalled.
The global replacement was interfering with other uses of JSON, such
as the clnrest plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: pyln-client: no longer autoconverts _msat field to Millisatoshi class (leaves as ints).
Using `pyln-testing` with `python3.11` results in the following warning.
```
.../venv/lib/python3.11/site-packages/cheroot/__init__.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
```
I've updated the `cheroot`-dependency.
However, the changes will not take effect immediately because we are
facing an internal conflict.
This conflict comes from the following requirements
- `//contrib/pyln-testing/pyproject.toml` requires `cheroot`. We can pick
the requried version
- `./external/lnprototest/pyproject.toml` requires `pyln="^0.12"` requires
`cheroot="^8"
It appears we have to do a multi-stage upgrade here.
**Step 1**
This commit configures `./contrib/pyln-testing/pyproject.toml`
to require `cheroot=">=8 <=10`. This allows users of `pyln-testing`
to start using a new release of `cheroot` immediately.
However, we still require `cheroot="^8"` because of `lnprototests`.
Even after running `poetry install` in the project root the warning
will remain.
**Step 2**
Not a part of this commit.
Publish a new release of `pyln-testing` on PyPI.
Once this release is finished we can update
`./external/lnprototests/pyproject.toml` to use the new
version of `pyln-testing` and the warning will disappear.
This breaks our tests a bit, which assumed we can always see ourselves
even if we don't have a proper channel.
Usually we would deprecate this first, but it's unlikely to break
anyone since it's a bit obscure that this worked at all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `listnodes` no longer shows private (peer) nodes: use listpeers
This is redundant if it's a public channel, but vital if it's not. Publishing unconditionally makes
it easier for gossmap: we create a local modification all the time, even if redundant (and we can
have the actual capacity ceiling accurate in this case, since we know it for local channels).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listpeerchannels` now shows gossip update contents (even if channel unannounced).
This allows use of clnrest + websocket in the regtest environment.
If clnrest starts up, a rune is generated and provided along with
the connection info.
Also uses the snap installed bitcoind path if appropriate.
Format of the start_ln output is:
Commands:
l1-cli, l1-log,
l2-cli, l2-log,
bt-cli, stop_ln, fund_nodes
Node Info:
l1 rest: https://127.0.0.1:3110 rune: "lgkWSp0PQK-pkbElLpjcNVQnX7yfEjouJaJHjLuW8w89OA=="
l2 rest: https://127.0.0.1:3111 rune: "sUqGqkevGYG2r1e_JUiz8Me00GhtTv5-IuGk4o9Beyc9OQ=="
Changelog-None
Now that 'developer' isn't a compile time flag, let's always use it
when using the startup regtest nodes.
Should always make gossip + polling bitcoind fast
We were using similar variables for different things (directories vs binary).
Update this to separate them out, while also adding ability to use a different directory
for lightning nodes
Changelog-Changed: startup_regtest.sh PATH_TO_LIGHTNING + PATH_TO_BITCOIN are no more. Use LIGHTNING_BIN and BITCOIN_DIR
We need to upgrade the rust version of the ubuntu docker images as
time-core v0.1.2 needs a version newer than 1.67.0
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
The fedora image installs a setuptools version that can not be upgraded
via pip. A fix is to remove the files manually and reinstall it via pip.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
From the spec:
Once peers are ready to exchange commitment signatures, they must remember
the details of the funding transaction to allow resuming the signatures
exchange if a disconnection happens.
Basically this means we add channels to the database before we've gotten
commitments for them; it's nice that there's now a state for commitments
recevied but we now save the channel prior to that.
This commit makes it possible to track the pre-commit-rcvd but not quite
open-init state.
startup_regtest now checks if wallet directory exists before creating
the default wallet, then it will check if wallet is loaded before
attempting to load it. This prevents unnecessary errors during the
execution of this bash script.
[ Squashed fixup: `regtest: restore double quotes on PATH_TO_BITCOIN` -- RR ]
Put an assertion inside db.c, and run every command we do (in testing) through
a `check` variant.
I inserted a deliberate bug (made addpsbtoutput call wallet_get_newindex()
before returning when running `check`, and indeed, backtrace as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`[False]` is considered True in Python, so this flake fix didn't work.
```
>>> if [False]:
... print('x')
...
x
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an alternative version to [1] that
update only the necessary information around our codebase
to upgrade the bitstring package.
This upgrade is necessary for the user of our package that
will find conflics in their codebase about packages version.
[1] https://github.com/ElementsProject/lightning/pull/6585
Co-Developed-by: @dni
Chanegelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We truncate the file on stop(), but don't re-created it on start().
We didn't notice it before, but the net
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes the pay helper function being able to route
a payment using the optional `route` paramter that defaults to `False`.
I added this, as some plugins maintained their own version of `pay` that
needed routed payment helper.
Changelog-None
Without `developer` option, after compiling CLN and sourcing
`contrib/startup_regtest.sh`, if we try to start 2 nodes on regtest
running the command `start_ln` we get the following error:
```
lightningd: Config file /tmp/l1-regtest/config line 6: dev-fast-gossip: requires --developer
...
lightningd: Config file /tmp/l2-regtest/config line 6: dev-fast-gossip: requires --developer
```
Changelog-None
Also added splice_out tests that use the new PSBT command.
ChangeLog-Added: New `addpsbtoutput` command for creating a PSBT that can receive funds to the on-chain wallet.
This is the variant of DUALOPEND_OPEN_INIT which you see once
the channel is in the db: we'll be adding it next, but to reduce
clutter the docs are added as a separate commit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you previously configured with `--enable-developer` we turn that into `--enable-debugbuild`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: build: `--enable-developer` arg to configure (and DEVELOPER variables): use `./configure --enable-debugbuild` and `developer` setting at runtime.
We were using `number` instead of `u32` which caused it to be
serialized to string with a decimal, which then would be rejected by
lightningd.
Changelog-Fixed: proto: Fixed a wrong number type being used in routes
This was recommended by @t-bast: if the final spec commits to something
compatible, we can simply advertize and accept both features, but if it
does change in incompatible ways we won't cause problems for nodes
who implement the official spec.
(I split this, so first, we remove the OPT_SPLICE entirely, to make
sure we caught them all. --RR)
Suggested-by: @t-bast
Changelog-None
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.
Implement the channeld splicing protocol leveraging the interactivetx protocol.
Implement lightningd’s channel_control to support channeld in its splicing efforts.
Changelog-Added: Added the features to enable splicing & resizing of active channels.
Using the -n flag on echo is non-standard for zsh. printf ‘%s’ accomplishes the equivilent thing.
Changelog-Added: Small fix for Mac OS building
# Conflicts:
# plugins/Makefile
The fields were missing because they weren't annotated with a type and
a description. Adding those fixes them.
Changelog-Fixed: msggen: `listpays` now includes the missing `amount_msat` and `amount_sent_msat` fields
No-schema-diff-check: fields were always there, just undocumented!
If you miss a wait event, you can catch up by doing listinvoices and
getting the max of these fields. It's also a good debugging clue.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we have defined ordering, we can add a start param.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listinvoices` has `index` and `start` parameters for listing control.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `setchannel` adds a new `ignorefeelimits` parameter to allow peer to set arbitrary commitment transaction fees on a per-channel basis.
We also document how the grpc test mode works, and why it currently
lacks coverage.
Changelog-Changed: pyln-testing: The grpc dependencies are now optional.
People running master notice that calling listconfigs fails, because
we don't handle objects called xxx_msat correctly (see
d348554ff4). This makes it painful
to test, until we release a pyln-client version.
Fortunately, the three changes in master are all fully backwards compatible,
so we can simply cut a release now and upload to pipy.org.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This silences a compilation warning about accessing a deprecated
field. This was triggering on each build and we'll notice when they go
away, so why upset users with it?
Having the grpc bindings in the pyln-testing package was always a bit
strange, however it came with additional issues. Due to the way that
protos are handled by protobuf, any name clash, independently of where
we import the protos, would cause an import error. This usually
happens in diamond-pattern dependencies, and so pull out the generated
files into their own package that everyone else can rely on.
Changelog-None
Updating bionic, focal and jammy cl-repro builds:
- Rust version from 1.62 to 1.65
- Removed git clone, only mount option
- Removed sha256sums generation
When the `--daemon` flag is used, `lightningd` requires that `--log-file`
is used as well. By consequence, the `lightningd.service` didn't work
out-of-the-box for me.
This changes also sends the logs to `journald`.
The new approach is consistent with the `bitcoind.service` file in the bitcoin-core repository.
I prefer this approach because it comes with automatic log-rotation.
This is needed when we know we're *opening* an anchor channel, to
override the "do we already have an anchor channel open?" logic.
Also, document the nonwrapped arg added in v23.02.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundpsbt` and `utxopsbt` new parameter `opening_anchor_channel` so lightningd knowns it needs emergency reserve for anchors.
We use parameterization here. The old `anchor_expected()` was for
non-zero-fee anchors, and have bitrotted so there are some other
changes as well.
Unfortunately, all the anchor accounting seems to be broken, but I
cannot understand these tests at all. I had to simply disable them
for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we can CPFP, we don't have to track the feerate as closely. But
it still needs to get in the mempool, so we use 10 sat/byte, or the
100 block estimate if that is higher.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `feerates` has new fields `unilateral_anchor_close` to show the feerate used for anchor channels (currently experimental), and `unilateral_close_nonanchor_satoshis`.
Changelog-Changed: JSON-RPC: `feerates` `unilateral_close_satoshis` now assumes anchor channels if enabled (currently experimental).
I noted in some PR that we are failing during the diff of the autogenerated files.
So this will generate the last version of them!
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Due to a security report by github, we should increase
our cryptography lib version.
This may impact potential another version that is stuck
with a cryptography version.
Link: https://github.com/ElementsProject/lightning/issues/6164
Reported-by: @dni
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Deprecated: Plugins: `default` no longer accepted on `flag` type parameters (it was silently ignored, so just don't set it).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We leave the code in contrib/pyln-client/pyln/client/lightning.py to handle
msat null fields for now, though, for a bit more compatibility.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I need to update gprcio-tools, but it needs protobuf v4. Christian
added this restriction in a99509db36 to
"Update protobuf dependency to silence dependabot", but perhaps it's
time to actually update?
```
$ poetry update
Updating dependencies
Resolving dependencies... (1.0s)
Because pyln-testing (23.05rc2) @ file:///home/rusty/devel/cvs/lightning/contrib/pyln-testing depends on protobuf (>=3.20.3,<4)
and grpcio-tools (1.54.0) depends on protobuf (>=4.21.6,<5.0dev), pyln-testing (23.05rc2) @ file:///home/rusty/devel/cvs/lightning/contrib/pyln-testing is incompatible with grpcio-tools (1.54.0).
So, because cln-meta-project depends on both grpcio-tools (1.54.0) and pyln-testing (23.05rc2) @ file:///home/rusty/devel/cvs/lightning/contrib/pyln-testing, version solving failed.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Build: all experimental features are now runtime-enabled; no more ./configure --enable-experimental-features
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were using per-type overrides which caused some asymmetries, where
conversions could end up dropping fields as we went along. Essentially
each conversion would need to override a superset of the previous one,
which then caused issues when attempting to close the loop. By
overriding on the model level we ensure that all representations are
equivalent and convertible into one another, at the expense of
overriding a bit more aggressively, which should be fine anyway.
We use overrides that omit fields in some cases, which makes the
conversion lossy. This also means that until we complete the mapping
we can't reconvert back.
Now we've set everything up, the replacement code is quite simple.
Some tests now have to deal with RBF though, and our rbf tests need work
since they look for the old onchaind messages.
In particular, when we can't afford the fee we want, we back off to
the next blockcount estimate, rather than spending all on fees
(necessarily). So test_penalty_rbf_burn no longer applies.
Changelog-Changed: Protocol: spending unilateral close transactions now use dynamic fees based on deadlines (and RBF), instead of fixed fees.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `feerates`: added `floor` field for current minimum feerate bitcoind will accept
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And consolidate descriptions into lightning-feerates().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` now allow "minimum" and NN"blocks" as `feerate` (`feerange` for `close`).
Drop try_get_feerate() in favor of explicit feerate_for_deadline() and
smoothed_feerate_for_deadline().
This shows us everywhere we deal with old-style feerates by names.
`delayed_to_us` and `htlc_resolution` will be moving to dynamic fees,
so deprecate those.
Note that "penalty" is still used for generating penalty txs for
watchtowers, and "unilateral_close" still used until we get zero-fee
anchors.
Changelog-Added: JSON-RPC: `feerates` `estimates` array shows fee estimates by blockcount from underlying plugin (usually *bcli*).
Changelog-Changed: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) value *slow* is now 100 block-estimate, not half of 100-block estimate.
Changelog-Deprecated: JSON-RPC: `close`, `fundchannel`, `fundpsbt`, `multifundchannel`, `multiwithdraw`, `txprepare`, `upgradewallet`, `withdraw` `feerate` (`feerange` for `close`) expressed as, "delayed_to_us", "htlc_resolution", "max_acceptable" or "min_acceptable". Use explicit block counts or *slow*/*normal*/*urgent*/*minimum*.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than have specific-purpose levels, have an array of
[blockcount, feerate], and rebuild the specific-purpose levels
for now on top.
We also keep a *separate* smoothed feerate, so you can ask for that
explicitly.
Since all the plugins used the same formula to derive the different
named fee levels, we apply the reverse to return to the underlying
estimates: updating the interface comes next.
This is ugly for now, but various specific-purpose levels will be
going away, as we shift to deadline-driven fees.
This temporarily breaks the floor calculation, so that test is
disabled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Adding a new field with `added` fails:
```
AssertionError: Field Feerates.perkb.estimates[] does not have an `added` annotation
```
Looks like this assertion is wrong: we should get an added from the field itself or
from the .msggen.json file.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using single tuples in Python is ugly, so:
1. Rename wait_for_onchaind_tx to wait_for_onchaind_txs.
2. Make it take tuples explicitly.
3. Make wait_for_onchaind_tx a simpler wrapper/unwrapper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can no longer grab the tx in one line as we did with
wait_for_onchaind_broadcast, we need to track the broadcast from
lightningd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a visitor that ensures every new field has at least an `added`
field, and that we don't change the `added` or `deprecated` annotation
after the fact.
