The modern style is to assert that all messages have tlvs, but many
are currently empty. In particular,
c4c5a8e5fb30b1b99fa5bb0aba7d0b6b4c831ee5 added "update_add_htlc_tlvs"
without adding an explicit field of that type to "update_add_htlc".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are no feature-dependent fields left in the spec. (I've also
made a PR for the spec to remove the tooling for it there).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes a bug in installer registration where executable is evaluated
before entrypoints and other details are added.
***RECKLESS STDERR***
Traceback (most recent call last):
File lightning/tools/reckless, line 382, in <module>
INSTALLERS['nodejs'].add_entrypoint('{name}')
KeyError: 'nodejs'
Reported by @ksedgwic
Changelog-None
When enabling or disabling a plugin, the entrypoint is inferred
from the user provided name. A canonical name should be used, which
the installer entrypoint formats help to determine (this generally strips
the file extension if one is provided.)
Also adds a timeout when testing a plugin. Previously the behavior
of pyln-client was relied upon to exit if not communicating with
lightningd, however, this behavior is not universal.
Changlelog-Changed: reckless now installs node.js plugins
Also removes support for pip editable install using pyproject.toml
`pip install -e .` This was a fallback method when a requirements
file was not present, but was hacky and often failed anyway.
reckless: remove installation via pyproject.toml
This method relied on pip install in editable mode (hacky) and often
failed to complete anyhow. We should instead encourage a requirements
file to be created/used for user installation.
You still need to actually make a rune when lightningd starts, as
commando (for safety) won't work unless you actually generate a rune
(that it knows of!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: hsmtool: `makerune` command to make a master rune for a node.
1. Rename get_hsm_secret to get_unencrypted_hsm_secret.
2. Create a common helper for fetching full file contents.
3. Create new routine to decrypt if necessary: get_hsm_secret().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This should have been added earlier as @cdecker suggested, but is needed
to enable CI testing.
Changelog-Changed: Reckless - added support for networks beyond bitocoin and regtest
Also, fix the case where we didn't use --network with EXPOSE_TCP,
as reported by @theborakompanioni:
```
I get Wrong network! Our Bitcoin backend is running on 'regtest', but we expect 'main'. with LIGHTNINGD_NETWORK := regtest when param --network is not provided.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We only handle top-level objects with an array of objects:
make sure it is one before we call the routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we will document deprecations and additions, rather than just
pretending they've always been that way!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The switch to logging enabled verbose output regardless of the option
flag. Here the functionality is restored.
Changelog-Fixed: reckless verbosity properly applied.
Ubuntu clang 15.0.2-1 complains:
```
wire/peer_exp_wiregen.c:257:14: error: variable 'i' set but not used [-Werror,-Wunused-but-set-variable]
for (size_t i = 0; *plen != 0; i++) {
^
wire/peer_exp_wiregen.c:1373:14: error: variable 'i' set but not used [-Werror,-Wunused-but-set-variable]
for (size_t i = 0; *plen != 0; i++) {
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`urllib3` does not ship as built-in with any of the recent python
releases, whereas `urllib` does, and for the uses we have, they are
pretty much identical.
The requests package is preferred, but until installation of python user
dependencies is implemented, sticking with standard modules allows a
frictionless experience.
And make errors gcc-style, so emacs can jump through the automatically.
```
In devtools/reduce-includes.sh line 21:
echo -n "-$LINE"
^-- SC3037 (warning): In POSIX sh, echo flags are undefined.
In devtools/reduce-includes.sh line 25:
echo -n "."
^-- SC3037 (warning): In POSIX sh, echo flags are undefined.
In tools/rel.sh line 6:
prefix=$(printf '%s\n' "${from#$common}" | sed 's@[^/][^/]*@..@g')
^-----^ SC2295 (info): Expansions inside ${..} need to be quoted separately, otherwise they match as patterns.
Did you mean:
prefix=$(printf '%s\n' "${from#"$common"}" | sed 's@[^/][^/]*@..@g')
In tools/rel.sh line 7:
printf '%s\n' "$prefix/${to#$common}"
^-----^ SC2295 (info): Expansions inside ${..} need to be quoted separately, otherwise they match as patterns.
Did you mean:
printf '%s\n' "$prefix/${to#"$common"}"
For more information:
https://www.shellcheck.net/wiki/SC3037 -- In POSIX sh, echo flags are undef...
https://www.shellcheck.net/wiki/SC2295 -- Expansions inside ${..} need to b...
make: *** [Makefile:553: check-shellcheck] Error 123
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reckless was failing to install multiple plugins due to git not
appreciating the cwd being a now removed dir after the first plugin
tmp files were cleaned up.
While loading the appropriate lightningconfig file, it is now checked
against the active config file in lightningd. Because a deviation from the
default file structure would not be possible, a -conf option is also added
to explicitly pass the lightningd config file into reckless.
This also simplifies dynamic enable/disable by catching the exception
raised when the cli is unable to connect to RPC (lightningd offline or
misconfigured relative to reckless).
This change makes it easier to follow retrieval of parent directories.
Additional os.path operations replaced with their pathlib.Path equivalents
to keep module usage consistent.
The goal was to support passing a list to install, enable, etc. in order
to improve performance. Passing lists to most of the functions was less
practical than iterating through the items from the top level.
`reckless help <cmd>` previously called the function docstring. This could
be updated to use the subparser help, but would require a strict naming
convention or a dictionary. Providing a hint to use the built-in
contextual help via the option flag is hopefully sufficient.
Regtest environments commonly use explicit definition of the config
file for lightningd. This can be queried and utilized by default,
saving redundant definitions between lightning and reckless.
A simple standalone python executable to track plugin repositories, clone
to /tmp, install requirements, test plugin runs, then install and enable in
lightningd and in the config.
Changelog-Added: Reckless - a Core Lightning plugin manager
Mainly, field name changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Protocol: Support for forwarding blinded payments (as per latest draft)
This allows GDB to print values, but also allows us to use them in
'case' statements. This wasn't allowed before because they're not
constant terms.
This also made it clear there's a clash between two error codes,
so move one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: Error code from bcli plugin changed from 400 to 500.
This broke with COPTFLAGS="-flto -O3", and so I took a look (it
complains more than normal because main isn't there). We should never
be running update-mocks except on programs expected to compile: in
this case, that's tools/test/run-test-wire.c.
Remove the code which tries to run this, which also means
non-developers won't be running update-mocks!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: `hsmtool`: hsm_secret (ignored) on cmdline for dumponchaindescriptors (deprecated in v0.9.3)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If there's only a single underscore, lowdown ignores it, but if there are multiple
(see min_final_cltv_expiry) it decides we're trying to highlight part of the word.
Reported-by: @wtogami
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You can't start a list without a paragraph separator.
```diff
--- /tmp/before 2022-07-20 22:02:23.485372596 +0930
+++ /tmp/after 2022-07-20 22:02:33.745528456 +0930
@@ -21,12 +21,16 @@
On startup of the daemon, no autoclean is set up.
RETURN VALUE
- On success, an object is returned, containing: - enabled (boolean):
- whether invoice autocleaning is active
+ On success, an object is returned, containing:
- If enabled is true: - expired_by (u64): how long an invoice must be ex‐
- pired (seconds) before we delete it - cycle_seconds (u64): how long an
- invoice must be expired (seconds) before we delete it
+ • enabled (boolean): whether invoice autocleaning is active
+
+ If enabled is true:
+
+ • expired_by (u64): how long an invoice must be expired (seconds) be‐
+ fore we delete it
+ • cycle_seconds (u64): how long an invoice must be expired (seconds)
+ before we delete it
AUTHOR
ZmnSCPxj <ZmnSCPxj@protonmail.com> is mainly responsible.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are no definition lists in Markdown, and lists get mangled if
they follow immediately: they need a line between them.
So use bullets for options, and use an indent so the text gets in the
line below.
Here's a before-and-after example:
```diff
--- /tmp/after 2022-07-20 21:55:54.355487769 +0930
+++ /tmp/after2 2022-07-20 21:58:17.305642576 +0930
@@ -10,38 +10,71 @@
lightning-cli sends commands to the lightning daemon.
OPTIONS
- --lightning-dir=DIR Set the directory for the lightning daemon we're talking to; defaults to $HOME/.lightning.
+ • --lightning-dir=DIR
- --conf=PATH Sets configuration file (default: lightning-dir/config ).
+ Set the directory for the lightning daemon we're talking to; defaults to $HOME/.lightning.
- --network=network --mainnet --testnet --signet Sets network explicitly.
+ • --conf=PATH
- --rpc-file=FILE Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
+ Sets configuration file (default: lightning-dir/config ).
- --keywords/-k Use format key=value for parameters in any order
+ • --network=network
- --order/-o Follow strictly the order of parameters for the command
+ • --mainnet
- --json/-J Return result in JSON format (default unless help command, or result contains a format-hint field).
+ • --testnet
- --raw/-R Return raw JSON directly as lightningd replies; this can be faster for large requests.
+ • --signet
- --human-readable/-H Return result in human-readable output.
+ Sets network explicitly.
- --flat/-F Return JSON result in flattened one-per-line output, e.g. { "help": [ { "command": "check" } ] } would become
+ • --rpc-file=FILE
+
+ Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
+
+ • --keywords/-k
+
+ Use format key=value for parameters in any order
+
+ • --order/-o
+
+ Follow strictly the order of parameters for the command
+
+ • --json/-J
+
+ Return result in JSON format (default unless help command, or result contains a format-hint field).
+
+ • --raw/-R
+
+ Return raw JSON directly as lightningd replies; this can be faster for large requests.
+
+ • --human-readable/-H
+
+ Return result in human-readable output.
+
+ • --flat/-F
+
+ Return JSON result in flattened one-per-line output, e.g. { "help": [ { "command": "check" } ] } would become
help[0].command=check. This is useful for simple scripts which want to find a specific output field without parsing
JSON.
- --notifications/-N=LEVEL If LEVEL is 'none', then never print out notifications. Otherwise, print out notifications of
- LEVEL or above (one of io, debug, info (the default), unusual or broken: they are prefixed with # .
+ • --notifications/-N=LEVEL
+
+ If LEVEL is 'none', then never print out notifications. Otherwise, print out notifications of LEVEL or above (one of
+ io, debug, info (the default), unusual or broken: they are prefixed with # .
+
+ • --help/-h
+
+ Pretty-print summary of options to standard output and exit. The format can be changed using -F, -R, -J, -H etc.
+
+ • --version/-V
- --help/-h Pretty-print summary of options to standard output and exit. The format can be changed using -F, -R, -J, -H
- etc.
+ Print version number to standard output and exit.
- --version/-V Print version number to standard output and exit.
+ • allow-deprecated-apis=BOOL
- allow-deprecated-apis=BOOL Enable deprecated options. It defaults to true, but you should set it to false when testing to
- ensure that an upgrade won't break your configuration.
+ Enable deprecated options. It defaults to true, but you should set it to false when testing to ensure that an upgrade
+ won't break your configuration.
COMMANDS
lightning-cli simply uses the JSON RPC interface to talk to lightningd, and prints the results. Thus the commands avail‐
@@ -60,13 +93,13 @@
this is not encouraged.
EXAMPLES
- 1. List commands
+ 1. List commands:
- lightning-cli help
+ • lightning-cli help
- 2. Fund a 10k sat channel using uncomfirmed outputs
+ 2. Fund a 10k sat channel using uncomfirmed outputs:
- lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0
+ • lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0
BUGS
This manpage documents how it should work, not how it does work. The pretty printing of results isn't pretty.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This gives a nice way to ensure your secret is the correct one.
Also, we don't need to suppress VALGRIND for this test, now the output
races are fixed.
Changelog-Added: `hsmtool`: new command `checkhsm` to check BIP39 passphrase against hsm_secret.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows us to re-use existing tests (assuming the call and fields
are covered by `cln-rpc` and `cln-grpc`) to test the full roundtrip
from test over the grpc interface to the json-rpc interface and back
again.
You can switch to the grpc interface by setting the `CLN_TEST_GRPC`
environment variable to 1, but for now only very few shims are
implemented (due to the non-generated nature of LightningRpc).
Most unexpected ones are still 1, but there are a few recognizable error codes
worth documenting.
Rename the HSM ones to put ERRCODE_ at the front, since we have non-HSM ones
too now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Callers were supposed to call "tlv_fields_valid" after fromwire_tlv,
but few did. Make this the default, and call the underlying function
directly where we want to be more flexible (one place).
This loses the ability to allow misordered fields, or to pass through
*any* even fields. We restore that for special cases in the next
patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Requiring the caller to allocate them is ugly, and differs from
other types.
This means we need a context arg if we don't have one already.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we actually put bolt12 fields (.e.g tlv_invoice) in onion messages,
that code will try to call printwire_tlv_invoice(), so expose it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This works better in general: let printwire_x do the work of figuring
out how to demarshal x. This is particularly important for TLVs, which
require a call to tlv_x_new() first.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We make them return bool, and always use names `cursor` and `plen` in
callers, for simplicity.
Also, `...` means "loop until finished" not "loop this many bytes".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No more "towire_offer", but "towire_tlv_offer".
This means we double-up on the unfortunately-named `tlv_payload` inside
the onion, but we should rename that in the spec when we remove
old payloads.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also has to fix up tests.
Changelog-Fixed: cli doesn't required anymore to confirm the password if the `hsm_secret` is already encrypted.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Suggested-by: Rusty Russell
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Changed: Support hsm specific error error code in lightning-cli
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).
config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This include the following commits:
- review 1/2: move from tab to space, and remove the exp. prop from doc;
- review 2/2: remove experimental features;
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We were really strict with the version check, requiring an exact
match. We now check that we are running at least with the version we
were compiled with (distro upgrades continue to work, and repro builds
are built off of an unupdated installation matching this minimum
requirement), and a major version match (since major versions can and
will introduce breaking changes).
Changelog-Fixed: sqlite3: Relaxed the version match requirements to be at least a minimum version and a major version match
Before:
Ten builds, laptop -j5, no ccache:
```
real 0m36.686000-38.956000(38.608+/-0.65)s
user 2m32.864000-42.253000(40.7545+/-2.7)s
sys 0m16.618000-18.316000(17.8531+/-0.48)s
```
Ten builds, laptop -j5, ccache (warm):
```
real 0m8.212000-8.577000(8.39989+/-0.13)s
user 0m12.731000-13.212000(12.9751+/-0.17)s
sys 0m3.697000-3.902000(3.83722+/-0.064)s
```
After:
Ten builds, laptop -j5, no ccache: 8% faster
```
real 0m33.802000-35.773000(35.468+/-0.54)s
user 2m19.073000-27.754000(26.2542+/-2.3)s
sys 0m15.784000-17.173000(16.7165+/-0.37)s
```
Ten builds, laptop -j5, ccache (warm): 1% faster
```
real 0m8.200000-8.485000(8.30138+/-0.097)s
user 0m12.485000-13.100000(12.7344+/-0.19)s
sys 0m3.702000-3.889000(3.78787+/-0.056)s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is best-practice (to ensure prototypes match up), but there were a
few places we didn't (at least, directly). Make it a requirement,
either of form "foo.h" or <dir/foo.h>.
The noise is the change to our print templates.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We make it a first-class citizen internally, even though we won't use
it over the wire (at least, non-experimental builds). This scheme
follows the latest draft, in which features are flagged compulsory.
We also add several helper functions.
Since uses the *even* bits (as per latest spec), not the *odd* bits,
we have some other fixups.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to use this to handle the simple description for channel_type.
It also needs to handle variable-size types (just like subtypes).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Updates from output of latest version of shellcheck as per SC2268
“Avoid x-prefix in comparisons as it no longer serves a purpose”
https://www.shellcheck.net/wiki/SC2268
1. listpeers has a deprecated `"closer": null`, which we need
to handle in the schema, while trying not to damage our
documentation too much.
2. Don't print a condition if there are no fields to print.
3. Allow a special "untyped" marker for multifundchannel which returns
arbitrary JSON in a field.
4. Allow a single field return (for 'stop').
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I did this by copying the updated bech32 code, and then re-patching in
our minor changes:
1. Headers modded (we need size_t)
2. Explicit length for bech32_encode/decode (not 90).
3. Exposing and bech32_ prefix for convert_bits, charset, charset_rev.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This matters for:
- common/wallet.h vs wallet/wallet.h
- common/gossip_store.h vs gossipd/gossip_store.h
- common/json.h vs lightningd/json.h
- common/ping.h vs lightningd/ping.h
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If you do update-mocks in a dirty tree, the recursive make that it
uses will try to rebuild things! Suppress that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We should actually be including this (as it may define _GNU_SOURCE
etc) before any system headers. But where we include <assert.h> we
often didn't, because check-includes would complain that the headers
included it too.
Weaken that check, and include config.h in C files before assert.h.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes it's v0.9.3 and sometimes it's 0.9.3, and I always get it
wrong. This is painful when you have to create a signed tag after
committing to one of the two only to find out you chose wrong and have
to redo the tagging.
This would cause a segfault on the default network parameter for
`dumponchaindescriptors`.
Introduced in 1513a2d07e
Changelog-Fixed: hsmtool: fix a segfault on `dumponchaindescriptors` without network parameter
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This makes use of the constant defined in the previous commits to more
accurately detect plaintext, encrypted, and invalid seeds. We now error
on invalid seeds.
Changelog-changed: hsmd: we now error at startup on invalid hsm_secret
Changelog-changed: hsmtool: all commands now error on invalid hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This avoids duplication of both logic and error-prone values, such as
the salt. Grouping all hsm encryption logic into a public API will also
allow us to fuzz it.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Changelog-changed: lightningd: the `--encrypted-hsm` now asks you to confirm your password when first set
Changelog-changed: hsmtool: the `encrypt` now asks you to confirm your password
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Fixes#4302
Changelog-fixed: hsmtool: the `generatehsm` command now generates an appropriately-sized hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This slightly breaks the API, but still accept the input: we just don't
take it into account anymore.
For `dumponchaindescriptors`, we have to still take the old place of the
`network` parameter into account to not entirely break the API.
Changelog-Added: hsmtool: password must now be entered on stdin. Password passed on the command line are discarded.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This is vital for calculating merkle trees; I previously used
towire+fromwire to get this!
Requires generation change so we can magic the ARRAY_SIZE var (the C
pre-processor can't uppercase things).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Actually, it's more complex to translate the xpub descriptor to
testnet because of the descriptor checksum.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This adds a command which outputs the two output descriptors
corresponding to our onchain wallet.
This can be useful for an external service to monitor / send fund to our
wallet.
Further, an "xpriv" version of such descriptors could be used to import
onchain funds on a new wallet.
Changelog-Added: lightning-hsmtool: a new command was added to hsmtool for dumping descriptors of the onchain wallet
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
tools: Add `generatehsm` method to hsmtool to derivate BIP32 seeds from a
mnemonic using the BIP39 standard.
The new method uses libwally for the BIP39 to BIP32 derivation. It also
fails if an hsm_secret file already exists, so we do not overwrite
someone else's wallet without noticing.
It allows the use of passphrases, the ECHO mode in the terminal is
disable for higher security.
It currently supports "en", "es", "fr", "it", "jp", "zhs", "zht".
Changelog-Added: hsmtool: `hsm_secret` generation from a seed-phrase following BIP39.
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
