Commit graph

519 commits

Author SHA1 Message Date
ShahanaFarooqui
f72b0fdabd docs: Updates script and fixed test to generate lightning-sql
- Updated doc/Makefile for generating schemas/lightning-sql.json
    - Read the lightning-sql template from schemas/lightning-sql-template.json
    - Generate sql tables data and merge it with json object read from above template
    - Save final merged object in schemas/lightning-sql.json
- Updated doc/Makefile to auto generate lightning-sql.7.md and lightning-sql.7
- Deleted schemas/lightning-sql.json and adding it into .gitignore
- Added lightning-sql specific titles in the fromschema.py script
- Fixed test_sql by changing the sequence for listpeerchannels `reestablished` field
- Ignoring lightning-sql.json for msggen schema.json because it is auto generated by sql plugin and lightning-sql-template.json.
2024-03-19 14:58:59 +10:30
ShahanaFarooqui
418ecf3e42 script: Update scripts
- Update `make doc-all` script
- `fromschema.py` script
    - Updated to generate whole doc/lightning-*.md via doc/schemas/lightning-*.json file
    - Updated to print request params
    - Added `oneOfMany` condition for request params
    - Added `pairedWith` condition for request params
    - Added `dependentUpon` condition for request params
    - Updated for pre and post_return_value for response
    - Hiding `hidden` fields
    - `descriptions` are array now
    - Unified gaps between titles
    - Added default key-value pair

- script: msggen, sql-schema, listconfig and pyln-testing script updates

This does not add created markdowns for cleaner review. We will add the markdown files in a separate commit.
2024-03-19 14:58:59 +10:30
Rusty Russell
fdfffdc232 common: add routing test using real data which crashes.
The amount is set not to crash by default, but run
"common/test/run-route-infloop 8388607" and you'll see a crash.

Sorry about the 7MB blob, but this testing was quite revealing and
I consider it worth adding.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-03-07 14:09:14 +01:00
Christian Decker
8418989f9b release: Unbreak the Fedora build
We have installation instructions that tell the user to use `poetry`
and then we ourselves think we're clever and install only a known
subset? It was only a matter of time until we broke this.

Changelog-None
2024-02-28 14:38:10 +01:00
Alex Myers
5c475067b8 reckless: populate local submodules for searching
Git ls-tree does not report submodule contents which was not previously
accounted for.
2024-02-21 14:33:11 +01:00
Alex Myers
538b4c850e reckless: prefer to search a local copy of a remote source
If it has already been cloned, this is less problematic than
using the github rest API.
2024-02-21 14:33:11 +01:00
Alex Myers
bfb29aaef0 reckless: Clone github sources when API access fails
Due to the API ratelimit, this allows cloning a github repo and searching
the result rather than searching via the REST API.  If a source has
already been cloned, it is fetched and the default branch checked out.

Fixes a failure reported by @farscapian

Changelog-Fixed: Reckless no longer fails on github API ratelimit.
2024-02-21 14:33:11 +01:00
Alex Myers
ba9ec412c7 reckless: set relative path of local git repo directories 2024-02-21 14:33:11 +01:00
Alex Myers
47c81995e3 reckless: initialize repos with submodules and other cleanup 2024-02-21 14:33:11 +01:00
Alex Myers
61bbd70bc7 reckless: reduce github API calls
Github API calls are now ratelimited to 60 per hour.  Searching through
the subdirectory contents of lightningd/plugins will hit this limit after
two searches or installs.  The simple answer is to look for the directory
rather than verifying a valid entrypoint in a suitably-named directory
prior to cloning the repository.

Also corrects a bug that was flagging submodules as files while
populating directory contents.
2024-02-21 14:33:11 +01:00
Alex Myers
3879d7b9bd reckless: update timeouts 2024-02-07 13:24:34 +01:00
Alex Myers
90a20ff6e3 reckless: create virtual environments for python plugins
This uses pip to install to a venv when the dependencies are specified
with requirements.txt and poetry when it's present on the system and the
plugin has a pyproject.toml.

The directory structure will be:
reckless/
    source/                (original plugin code here)
        plugin_entrypoint  (original entrypoint)
    plugin_name            (symlink or wrapper to activate venv)
    .metadata              (installation information)
    .venv/                 (python virtual environment)

The wrapper matches the naming of the original plugin entrypoint. The
shebang is modified to use the virtual environment's python binary,
then the original plugin is imported as a module and executed as though
it was run directly.

Changelog-Changed: reckless installs python plugins in virtual environments
2024-02-07 13:24:34 +01:00
Alex Myers
03e78ce6d9 reckless: add option to check out a specific commit
... using the syntax reckless install <plugin_name>@<commit hash>
2024-02-07 13:24:34 +01:00
Alex Myers
d5ed7b7b5c reckless: add installation metadata
The metadata includes an original retrieval source, timestamp, and commit
hash.  This will allow a future update command to more easily evaluate the
status of the existing installation.
2024-02-07 13:24:34 +01:00
Alex Myers
2b502ebcbc reckless: add staging directory and symlink
This creates a separate staging directory from the one used to clone
the source.  A symlink is then added to the plugin's entrypoint which is
now in the source directory.
2024-02-07 13:24:34 +01:00
Alex Myers
c043bf2255 reckless: properly remove entry from reckless sources
Fixes a bug introduced in 6163138420 which
avoided gratuitous rewrites of the lightningd config
2024-02-07 13:24:34 +01:00
Rusty Russell
7f5fe52320 gossipd: remove online gossip_store compaction.
It was an obscure dev command, as it never worked reliably.
It would be much easier to re-implement once this is done.

This turned out to reveal a tiny leak on
tests/test_gossip.py::test_gossip_store_load_amount_truncated where we
didn't immedately free chan_ann if it was dangling.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-02-04 09:24:44 +10:30
Lagrang3
28c28e4ea9 add --show-secrets option for hsmtool dumponchain 2024-01-31 13:02:51 +01:00
Lagrang3
a1c02e8098 hsmtool: add feature to dump private keys 2024-01-31 13:02:51 +01:00
Rusty Russell
dadbdf488c schemas: deprecated is now a range.
Don't assume removal is +6 months, but have a start deprecation/end support range.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-01-26 10:30:22 +10:30
Christian Decker
44b7fe7cc6 meta: Fix the tools/build-release.sh script to match repro instructions 2023-12-29 00:35:45 +01:00
ShahanaFarooqui
78518344f4 docker: Creating docker latest image
- Create docker multiarch image with/without push
- Create docker latest image and publish if asked

Changelog-None
2023-12-12 16:31:44 +01:00
Peter Neuroth
9d40a165a0 hsmtool: Fix argc check on getcodexsecret
This caused a segmentation fault when calling `getcodexsecret` without
id.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-Fixed: Hsmtool: Fix segmentation fault when calling `getcodexsecret` without id.
2023-11-28 11:01:26 +01:00
Aditya Sharma
8d36b7c438 tools/hsmtool.c: Add to hsmtools to get content of emergency.recover in bech32 format.
Changelog-Added: `hsmtool`: new command `getemergencyrecover` to extract emergency.recover in bech32 format (clnemerge1...)
2023-10-27 16:05:28 +10:30
Rusty Russell
64495cf166 tools: make sure we don't use optimization when building mocks.
Don't want it to optimize something out!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-10-23 15:48:50 +10:30
ShahanaFarooqui
b2fe1bc1e5 tools: Fixed release verify typo
Removed `bin-` while copying Fedora sums from release captain's file. Sums are saved as `clightning-$VERSION-Fedora-28-amd64.tar.gz` not as `clightning-$VERSION-bin-Fedora-28-amd64.tar.gz`

Changelog-None.
2023-10-02 15:18:00 +10:30
Rusty Russell
573ce14509 tools: remove DEVELOPER.
We can now always enable developer using --developer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell
64ab7c0c1d lightningd: add --developer runtime option.
Currently it just defaults to the DEVELOPER compile option, but we'll
move over to this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `--developer` enables developer options and changes default to be "disable deprecated APIs".
2023-09-21 20:08:24 +09:30
ShahanaFarooqui
8906e22f52 script: Updated build-release script
- Removed tarball and deb targets
- Added `TARBALL_ARCH` for bitcoin, litecoin image downloads
2023-09-20 14:08:34 +09:30
Peter Neuroth
a3d393a24d tools: Make shellcheck happy
This change proposed by RustyRussel stops a race condition on the
former `echo` line.
2023-09-13 10:42:54 +09:30
Peter Neuroth
8f02678147 tools: Fix accidental override on checksum copy
The former seemed to replace the wrong line with the copied checksum.
We now add read it from the first line and add it on top of our sums.
This expression also seems a fair bit easier to understand now.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-09-13 10:42:54 +09:30
Peter Neuroth
43aeaff1a8 tools: Add sha256sum check for detailed hint
Removing the Fedora fake as it seems to be not necessary. We later copy
the checksum from the release captains checksums anyway.
Also adding the sum check as it gives more details about which file did
not match if so.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-09-13 10:42:54 +09:30
Peter Neuroth
e76c14cb0c tools: Add check if the checksum file exists
This ensures that we have the release captains checksum file where we
expect it to be and gives a little hint where to get it if needed.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-09-13 10:42:54 +09:30
ShahanaFarooqui
e13747bbe6 script: docker build
- With push or load options
- multiarch from single Dockerfile
- Add latest and version manifests too
2023-09-12 13:20:34 +09:30
Rusty Russell
93d8e10786 tools/check-manpage.sh: update undoc to a simple list.
It didn't handle the case where an undocumented option was
was a flag (i.e. didn't end in =), so rework it to be
a simple list and use grep.

Add some more options we don't document, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 20:29:34 +09:30
Rusty Russell
b03358e933 build: have check-manpage.sh ignore options documented as deprecated.
These days we sometimes keep docs for deprecated options, so ignore
those.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 20:29:34 +09:30
Rusty Russell
9085293622 build: fix check-manpage.sh
Our formats changed, so this didn't work any more!

Add a sanity check that in future if we get no options from a command,
we complain.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 20:29:34 +09:30
ShahanaFarooqui
33ab1ce78b docker: Install clnrest dependencies from rquirements.txt
Changelog-None
2023-08-06 12:34:19 +09:30
Rusty Russell
0f6687ec7b codex32: use "cl" instead of "ms" as our HRP.
This was strongly recommended by Russell O'Connor: the "ms" implies that
it's a BIP-32 master secret, and this is CLN specific.

If we changed the hrp to "cln" it would be better, but apparently that
means we no longer fit in a "standard billfold metal wallet" (and
our code assumes a 2-byte prefix anyway).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 16:12:32 +09:30
Rusty Russell
5900742b3a hsmtool: remove threshold arg.
It really has to be 0, since it's the complete secret.  And we didn't handle
it well, (`a` would be treated as 0, for example!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 08:42:50 +09:30
Rusty Russell
4dcd712d6e hsmtool: tighten checks for validity on getcodexsecret
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-02 08:42:50 +09:30
Shahana Farooqui
85332f2bd4 tools: build release docker script update
tools: build release docker script update
2023-08-01 11:35:19 +09:30
niftynei
dd871d9e26 inflights: use ctx for making new ones
Also convert everything to an array thingy
2023-07-31 21:00:22 +09:30
Dusty Daemon
4628e3ace8 channeld: Code to implement splicing
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.

Implement the channeld splicing protocol leveraging the interactivetx protocol.

Implement lightningd’s channel_control to support channeld in its splicing efforts.

Changelog-Added: Added the features to enable splicing & resizing of active channels.
2023-07-31 21:00:22 +09:30
adi2011
6774cc1b13 tools/hsmtool: Add getcodexsecret to get the hsm_secret in bip93 2023-07-31 20:57:40 +09:30
Alex Myers
8704a4b499 reckless: add installation capability for additional sources
Abstracts search and directory traversal. Adds support for installing
from a local git repository, a local directory, or a web hosted git repo
without relying on an api.

Changelog-Changed: Reckless can now install directly from local sources.
2023-07-27 06:53:49 +09:30
Alex Myers
30b873de31 reckless: style cleanups, type hints, simplify create_dir 2023-07-27 06:53:49 +09:30
Shahana Farooqui
94205c094f plugins/clnrest: Update clnrest dependencies in scripts and Dockerfiles
Adding python3-json5 python3-flask python3-gunicorn flask_restx pyln-client
Also run `pip3 install -r plugins/clnrest/requirements.txt`
2023-07-25 19:20:16 +09:30
Shahana Farooqui
2e72387700 lightningd: removing bionic and python 3.7 support
Changelog-Removed: support for python v<=3.7 & Ubuntu bionic has been
removed.

clnrest's flask & gevent libraries require Python v>=3.8.
2023-07-20 11:44:46 +09:30
Rusty Russell
5132fea44e docs: fix up internal references now files have moved.
And tools/check-markdown.sh didn't do anything: remove it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-16 12:59:40 +09:30