Commit Graph

14072 Commits

Author SHA1 Message Date
ShahanaFarooqui
78518344f4 docker: Creating docker latest image
- Create docker multiarch image with/without push
- Create docker latest image and publish if asked

Changelog-None
2023-12-12 16:31:44 +01:00
Christian Decker
a9aa93d822 subd: Do not send feerate updates to non-channeld subds
Turns out we were sending feerate updates to daemons that do not
understand it. Don't do that!

Closes #6932

Changelog-Fixed: channeld: We could crash `closingd` by sending it a `channeld` message
2023-12-12 16:27:56 +01:00
Alex Myers
b0377d2323 misc: Add Alex's gpg key
Changelog-None
2023-12-11 18:46:43 +01:00
ShahanaFarooqui
f89e91332f misc: Add Shahana's gpg key
Changelog-None
2023-12-11 18:46:31 +01:00
Alex Myers
9585b7c238 startup_regtest: test for clnrest dependencies before writing config 2023-12-05 20:31:42 -06:00
Alex Myers
66e2bf2947 startup_regtest: configure clnrest and generate rune
This allows use of clnrest + websocket in the regtest environment.
If clnrest starts up, a rune is generated and provided along with
the connection info.

Also uses the snap installed bitcoind path if appropriate.

Format of the start_ln output is:
Commands:
	l1-cli, l1-log,
	l2-cli, l2-log,
	bt-cli, stop_ln, fund_nodes
Node Info:
	l1 rest: https://127.0.0.1:3110  rune: "lgkWSp0PQK-pkbElLpjcNVQnX7yfEjouJaJHjLuW8w89OA=="
	l2 rest: https://127.0.0.1:3111  rune: "sUqGqkevGYG2r1e_JUiz8Me00GhtTv5-IuGk4o9Beyc9OQ=="

Changelog-None
2023-12-05 20:31:42 -06:00
niftynei
df71f2ffa0 coin_mvt: use the lightning_hrp for *all* coin movement currency
We were (dumbly?) using the `onchain_hrp` for 'chain_mvts' and the
`lightning_hrp` for 'channel_mvts'.

This works fine everywhere *except* for on a signet, where we use
different prefixes.

Since the lightning-hrp set is more diversified (testnet btc
+ signet btc use the same HRP 'onchain'), let's use that.

Should have zero impact on anything other than nodes running on signet.

To preserve your current accounts database without needing to delete,
restart, execute the following: (note preferrably when your node isn't
running).

```
UPDATE chain_events SET currency = 'tbs' WHERE currency = 'tb';
```

Fixes #6534

Changelog-Fixed: `bkpr-listbalances` would crash for nodes on signet with payments in channels, because onchain events were using a different currency than inchannel events.
2023-12-05 20:30:09 -06:00
niftynei
5a29baafdd startup-regtest: fund_nodes make balanced, funded channels
Changelog-Changed: startup_regtest.sh: `fund_nodes` will now make balanced channels
2023-12-05 01:02:43 -06:00
niftynei
a1cecf044c startup-regtest: flag every use as developer
Now that 'developer' isn't a compile time flag, let's always use it
when using the startup regtest nodes.

Should always make gossip + polling bitcoind fast
2023-12-05 01:02:43 -06:00
niftynei
8476a15a44 startup-regtest: add ability to set bitcoin-bin + lightning-dir
We were using similar variables for different things (directories vs binary).

Update this to separate them out, while also adding ability to use a different directory
for lightning nodes

Changelog-Changed: startup_regtest.sh PATH_TO_LIGHTNING + PATH_TO_BITCOIN are no more. Use LIGHTNING_BIN and BITCOIN_DIR
2023-12-05 01:02:43 -06:00
Christian Decker
be52f2f21e ci: Force flaky of tests
We are tracking the flakiness of tests [here][1], so no need to suffer
through fail-looping CI while trying to fix a PR or get it merged.

Changelog-None

[1]: http://35.239.136.52:3170/run
2023-12-04 20:22:04 +01:00
Christian Decker
a96e64ed56 chore(CI): Add manually triggerable PyPI publication
We used to run the publication on each PR, which could cause conflicts
due to the naming convention being based on the commit history length,
and multiple PRs potentially hitting the same name.

Adding a `workflow_dispatch` trigger allows us to still test the
workflow in PRs but only do so manually (when we're watching anyway).

Changelog-None
2023-12-04 20:22:04 +01:00
Christian Decker
66bf0a7bbb chore(CI): Increase the timeout to publish to test.pypi.org 2023-12-04 20:22:04 +01:00
Peter Neuroth
84f22066f2 misc: Add peter's gpg key
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-None
2023-11-29 12:56:45 +01:00
Peter Neuroth
5887be000c CHANGELOG.md: Update for 23.11 final
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-28 19:01:39 +01:00
Peter Neuroth
9ace27e77b doc: Update example to be valid codex32 string
I guess this was to honor Aditya, but it was also a wrong example.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-28 11:01:26 +01:00
Peter Neuroth
3d51f00147 doc: Add getemergencyrecover to the hsmtool doc
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-28 11:01:26 +01:00
Peter Neuroth
9d40a165a0 hsmtool: Fix argc check on getcodexsecret
This caused a segmentation fault when calling `getcodexsecret` without
id.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-Fixed: Hsmtool: Fix segmentation fault when calling `getcodexsecret` without id.
2023-11-28 11:01:26 +01:00
Peter Neuroth
194dd2bb7d CHANGELOG.md: Update for 23.11rc3
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-21 19:14:21 +01:00
Peter Neuroth
eb3b1b8134 clnrest: Import sys in except clause explicitly
This fixes a crash on startup of core-lightning where gevent could not
be imported. This happens before sys is imported and throws us into the
except clause which calls sys.
By importing it explicitly in the except clause we are not dependend of
the order of imports in the try bracket.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>

Changelog-Fixes: Plugin: clnrest crashed on startup when gevent was
missing.
2023-11-21 08:24:09 +01:00
Rusty Russell
0b23133ab2 lightningd: don't print out notification msat fields as strings.
Reported-by: Shahana Farooqui
Changelog-Fixed: JSON-RPC: Plugin notification `msat` fields in `invoice_payment` and `invoice_created` hooks now a number, not a string with "msat" suffix.
Changelog-Fixed: JSON-RPC: Plugin hook `payment` `msat` field is now a number, not a string with "msat" suffix.
2023-11-21 08:23:39 +01:00
ShahanaFarooqui
fdc7fcbf34 tests/clnrest: Test to confirm that msat suffix has been removed from notifications 2023-11-21 08:23:39 +01:00
ShahanaFarooqui
2a7a3d2371 tests/clnrest: Added rpc method name and params to use the same notifications_received_via_websocket function for other rpc requests too 2023-11-21 08:23:39 +01:00
Peter Neuroth
8aff6f788f CHANGELOG.md: Update for 23.11rc2
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-20 08:26:58 +01:00
Dusty Daemon
a6a9e5b1e3 splice: Reestablish when commit or sig sends fail
Adds tests for when the connection fails during
1) splice tx_signature
2) splice commitment_signed

Fleshed out the reestablish flow for these two cases and implemented the fixes to make these reestablish flows work.

Part of this work required changing commit process for splices: Now we send a single commit_part for the splice where previously we sent all commits, and accordingly, we no longer revoke in response.

Changelog-Fixed: Implemented splicing restart logic for tx_signature and commitment_signed. Splice commitments are reworked in a manner incompatible with the last version.
2023-11-20 07:35:22 +01:00
Rusty Russell
eae0d1da1e lightningd: fix up deprecated rest-port, rest-protocol, rest-host and rest-certs option if we would otherwise fail.
Since these worked in v23.08, we can't just rename them.  So if they are
used and unclaimed, we should rename them internally (if they're claimed,
it's probably clightning-rest, and we should *NOT* touch them!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `clnrest` parameters `rest-port`, `rest-protocol`, `rest-host` and `rest-certs`: prefix `cln` to them
2023-11-17 10:51:41 +01:00
Peter Neuroth
8f8202daa7 meta: Update release checklist
Add a version bump for clnrest and the main poetry.lock to the makefile
and update the release checklist to include the new target.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-16 15:20:33 +01:00
ShahanaFarooqui
cc2665da7f clnrest: prefixing all rest config options with cln
This will allow users to use clnrest with c-lightning-REST without conflicts.
It was required for applications to have enough time for migrating from c-lightning-REST to clnrest.

Changelog-Changed:
config option `rest-certs` changed to `clnrest-certs`
config option `rest-protocol` changed to `clnrest-protocol`
config option `rest-host` changed to `clnrest-host`
config option `rest-port` changed to `clnrest-port`
config option `rest-cors-origins` changed to `clnrest-cors-origins`
config option `rest-csp` changed to `clnrest-csp`
2023-11-16 11:21:04 +01:00
ShahanaFarooqui
017581cd79 doc: update decode instruction for emergency.recover
Related to:
https://github.com/ElementsProject/lightning/issues/6711
https://github.com/ElementsProject/lightning/pull/6773

Changelog-None
2023-11-16 09:13:34 +01:00
ShahanaFarooqui
9e9c3dbc00 clnrest: add gevent ssl monkey patch to remove warning log 2023-11-16 09:07:07 +01:00
Rusty Russell
8b84c8945b clnrest: don't convert *_msat fields to strings.
We have a global JSON encoder hack, which means that any field ending in msat gets special treatment (so we can safely talk to lightningd, even if a field expects satoshi amounts, we are explicit).  However, requests uses the JSON parser and DOES NOT want this conversion when sending it out as an HTTP response!

The simplest local fix we could find was Shahana's suggestion to iterate and covert away from Millisatoshi(): the reverse of what our JSON encoder does.

Fixes: https://github.com/ElementsProject/lightning/issues/6848
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-16 09:07:07 +01:00
Rusty Russell
f2f6c73977 clnrest: don't try to handle plugin.rpc.call results.
It already throws an exception or error, or decodes the JSON response:
we can simply pass it through.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-11-16 09:07:07 +01:00
Tony Aldon
b3f122a2c5 pytest: adds test suite for clnrest plugin
- cln-grpc certificate reuse
- new certificate generation
- `GET` and `POST` requests
- websocket server
- config options and HTTP headers

Link to #6436.
2023-11-16 09:07:07 +01:00
Tony Aldon
672477c95c poetry: add requests dependency for clnrest tests in contrib/pyln-testing
`poetry update` did not change the poetry.lock file for contrib/pyln-testing
because it already has `requests`.
2023-11-16 09:07:07 +01:00
Christian Decker
94f4d75006 msggen: Add Wait method to managed API
Fixes #6792

Changelog-Added: Cln-RPC: Add `wait` system to cln-rpc and cln-grpc.
2023-11-16 09:02:33 +01:00
Christian Decker
bdfd2d5fda msggen: Add FetchInvoice method to Rust and grpc bindings
Closes #6844

Changelog-Added: Cln-RPC: Add `fetchinvoice` method to cln-rpc and cln-grpc.
2023-11-16 09:02:33 +01:00
Ken Sedgwick
9e0b244058 splice: fix WIRE_HSMD_SIGN_SPLICE_TX capability check
Fixes #6866

Changelog-Fixed: The WIRE_HSMD_SIGN_SPLICE_TX HSM capability is now correctly checked.
2023-11-15 18:42:08 +10:30
Tony Aldon
cce4f68f7a doc: fix cln-grpc example.
- Add instruction to generate `primitives_pb2.py` file,
- Import `grpc` in the Python script.

Changelog-None
2023-11-14 10:48:41 +01:00
Peter Neuroth
02ca226f88 CHANGELOG.md: Update for 23.11rc1
Update changelog, pyln versions, clnrest version and poetry.lock.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-03 14:49:12 +01:00
Peter Neuroth
b135ba2224 reprobuild: Upgrade rust version
We need to upgrade the rust version of the ubuntu docker images as
time-core v0.1.2 needs a version newer than 1.67.0

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-02 19:52:46 +01:00
Peter Neuroth
1546127b23 docker: Add fix for broken setuptools installation
The fedora image installs a setuptools version that can not be upgraded
via pip. A fix is to remove the files manually and reinstall it via pip.

Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
2023-11-02 19:52:46 +01:00
Ken Sedgwick
88a536f3eb Possible fix for CI_SERVER issue in VLS CI
Code using `CI_SERVER` was added recently to track something.  It
attempts to short-cut if `CI_SERVER` is not set, but on gitlab it
is set to `yes`.  https://docs.gitlab.com/ee/ci/variables/

Instead use `CI_SERVER_URL`, maybe that is what is intended?
2023-11-02 13:53:03 +01:00
niftynei
3190c26bc9 dualfund: error on out of order sigs
We weren't blocking if the tx-sigs arrived before the commitment sigs.

This was causing problems in the openchannel (spender plugin)

spenderp: FATAL SIGNAL 11 (version v23.08.1-404-g62ff475-modded)
0x559836dc98ba send_backtrace
	common/daemon.c:33
0x559836dc9951 crashdump
	common/daemon.c:75
0x7f37f42c351f ???
	./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x7f37f441ac92 ???
	../sysdeps/x86_64/multiarch/memcmp-avx2-movbe.S:83
0x559836db7760 bitcoin_txid_eq
	./bitcoin/tx.h:29
0x559836db7760 collect_sigs
	plugins/spender/openchannel.c:509
0x559836db81de check_sigs_ready
	plugins/spender/openchannel.c:531
0x559836db84dd json_peer_sigs
	plugins/spender/openchannel.c:611
0x559836dbcad7 ld_command_handle
	plugins/libplugin.c:1611
0x559836dbcd9d ld_read_json_one
	plugins/libplugin.c:1721
0x559836dbce29 ld_read_json
	plugins/libplugin.c:1741
0x559836ef3bff next_plan
	ccan/ccan/io/io.c:59
0x559836ef40da do_plan
	ccan/ccan/io/io.c:407
0x559836ef4177 io_ready
	ccan/ccan/io/io.c:417
0x559836ef5b14 io_loop
	ccan/ccan/io/poll.c:453
0x559836dbd48d plugin_main
	plugins/libplugin.c:1948
0x559836db22bf main
	plugins/spender/main.c:35
0x7f37f42aad8f __libc_start_call_main
	../sysdeps/nptl/libc_start_call_main.h:58
0x7f37f42aae3f __libc_start_main_impl
	../csu/libc-start.c:392
0x559836da3774 ???
	???:0
0xffffffffffffffff ???
	???:0
2023-10-31T15:15:57.458Z INFO    plugin-spenderp: Killing plugin: exited during normal operation
2023-10-31T15:15:57.458Z **BROKEN** plugin-spenderp: Plugin marked as important, shutting down lightningd!
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_break: lightningd_exit
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_loop_with_timers: main
2023-10-31T15:15:57.458Z DEBUG   connectd: REPLY WIRE_CONNECTD_START_SHUTDOWN_REPLY with 0 fds
2023-10-31T15:15:57.458Z DEBUG   lightningd: io_break: connectd_start_shutdown_reply
2023-10-31T15:15:57.458Z DEBUG   021ccce7bc396996c8f3b7bfeb1e30c6600269517026a74adfe2217b7187879797-dualopend-chan#1: Status closed, but not exited. Killing
2023-10-31T15:15:57.458Z DEBUG   lightningd: Command returned result after jcon close
2023-10-31T15:15:57.458Z INFO    021ccce7bc396996c8f3b7bfeb1e30c6600269517026a74adfe2217b7187879797-chan#1: Unsaved peer failed. Deleting channel.
2023-10-31T15:15:57.464Z DEBUG   lightningd: io_break: destroy_plugin
2023-10-31T15:15:57.464Z DEBUG   connectd: Shutting down
2023-10-31T15:15:57.464Z DEBUG   gossipd: Shutting down
2023-10-31T15:15:57.464Z DEBUG   hsmd: Shutting down

Reported-By: @t-bast
2023-11-02 19:32:05 +10:30
niftynei
fa8458c00a dualfund: add test to make sure that tx-sigs sent before commitment
results in an error.
2023-11-02 19:32:05 +10:30
niftynei
30ec8cbf8e dualfund, test: add test for dropping to chain during RBF
Here we make sure we can drop the initial tx to chain, and that an
inflight txid that's missing its commitment sigs is properly ignored.
2023-11-02 19:32:05 +10:30
niftynei
f4b4f772f3 dualfund, bump: when bumping a channel make sure it's in ok state
If we disconnect, we lose the open_attempt record. Which is fine, but we
should prevent the user from starting another RBF if the last one isn't
done yet!
2023-11-02 19:32:05 +10:30
niftynei
dbcdfd7d66 dualfund, memleak: don't leak the msg on error
We don't let go of the `msg` on error, which triggers a memleak warning!

lightningd-2 2023-10-31T19:54:06.582Z **BROKEN** lightningd: MEMLEAK: 0x55ae3615b498
lightningd-2 2023-10-31T19:54:06.582Z **BROKEN** lightningd:   label=openingd/dualopend_wiregen.c:919:u8[]
lightningd-2 2023-10-31T19:54:06.582Z **BROKEN** lightningd:   alloc:
lightningd-2 2023-10-31T19:54:06.685Z **BROKEN** lightningd:     ccan/ccan/tal/tal.c:477 (tal_alloc_)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     ccan/ccan/tal/tal.c:506 (tal_alloc_arr_)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     openingd/dualopend_wiregen.c:919 (towire_dualopend_send_tx_sigs)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     lightningd/dual_open_control.c:1122 (openchannel2_sign_hook_cb)
lightningd-2 2023-10-31T19:54:06.686Z **BROKEN** lightningd:     lightningd/plugin_hook.c:194 (plugin_hook_call_next)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin_hook.c:169 (plugin_hook_callback)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin.c:660 (plugin_response_handle)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin.c:772 (plugin_read_json_one)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/plugin.c:823 (plugin_read_json)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/io.c:59 (next_plan)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/io.c:407 (do_plan)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/io.c:417 (io_ready)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     ccan/ccan/io/poll.c:453 (io_loop)
lightningd-2 2023-10-31T19:54:06.687Z **BROKEN** lightningd:     lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:     lightningd/lightningd.c:1333 (main)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:     ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:     ../csu/libc-start.c:392 (__libc_start_main_impl)
lightningd-2 2023-10-31T19:54:06.688Z **BROKEN** lightningd:   parents:
2023-11-02 19:32:05 +10:30
niftynei
da34c369f3 dualfund, tests: break out "peer forgets" test
Now that we save the commitment sigs immediately, we have to drop the
connection elsewhere in the flow to get the state where only one peer
remembers.
2023-11-02 19:32:05 +10:30
niftynei
89f6fd27e3 dual-fund: have accepter send their commitment sigs asap
Originally the accepter waited for the peer to send us their commitment
sigs before we send ours; this changes things so that the accepter
sends their commitment sigs ASAP.

	This test fails: when cln is not the channel initiator, it waits for the other node to send commit_sig before sending its own commit_sig. There is no reason to do that, both nodes should send commit_sig immediately after exchanging tx_complete? Otherwise it's a missed opportunity to finalize the channel creation on reconnection, because in that case cln hasn't saved the channel and fails it on reconnection.

Reported-By: @t-bast
2023-11-02 19:32:05 +10:30
niftynei
48bb2d831b dual-fund: don't re-notify plugin on arrival of sigs (2nd time)
When we got our peer's sigs, if we were the remote, we would re-notify
the plugin, which in turn would re-send the tx-sigs to use.

In the case of CLN, we'd then
- break, because we'd re-forward the sigs to the `openchannel` plugin,
  which was then in the wrong state (MULTIFUNDCHANNEL_SIGNED)

    spenderp: plugins/spender/openchannel.c:598: json_peer_sigs: Assertion `dest->state == MULTIFUNDCHANNEL_SECURED' failed.
    spenderp: FATAL SIGNAL 6 (version 5880d59-modded)

In the case of eclair, they'd just see our 2nd TX_SIGS message and
@t-bast would complain:

	> This test works, with one minor issue: on reconnection, cln sends its tx_signatures twice (duplicate?).

This commit does two things:
	- has the openchannel / spender plugin log a broken instead of
	  crashing when the state is not what we're expecting
	- stops us from calling the `funder` plugin if this is a
	  replay/second receipt of commit-sigs.
2023-11-02 19:32:05 +10:30