Commit graph

14874 commits

Author SHA1 Message Date
Lagrang3
0e2660733f renepay: update unit tests 2024-05-08 22:32:13 -05:00
Lagrang3
13a2828b12 renepay: add a test for concurrent payments 2024-05-08 22:32:13 -05:00
Lagrang3
1ee157f02b renepay: update the Makefile 2024-05-08 22:32:13 -05:00
Lagrang3
cc00d509b1 renepay: add a helper module json.c 2024-05-08 22:32:13 -05:00
Lagrang3
73c6142965 renepay: add error codes
Using enum renepay_errorcode simplifies the low level API of chan_extra and flow.
We can extract information about the nature of a function call failure
from its return value.
2024-05-08 22:32:13 -05:00
Lagrang3
7584629696 renepay: add routefail.c
Routefail consist mainly of an API `routefail_start` that handles the
failure of a forwarding request (encoded in a route). Internally there
is a routefail datastructure that goes through a series of execution
steps, eg. updating the gossmap, updating the uncertainty network, etc.
2024-05-08 22:32:13 -05:00
Lagrang3
480dec58f2 renepay: add routetracker.c
Routetracker is a structure that is used to follow the progress of the
route forwarding requests.
2024-05-08 22:32:13 -05:00
Lagrang3
84651fa4ba renepay: add routebuilder.c
Routebuilder constitute the module that builds the routes we try in the
payment. The public API is simply `get_routes`, it replaces the similar
interface for pay_flows.
2024-05-08 22:32:13 -05:00
Lagrang3
6766acbcdf renepay: payment state machine (mods)
Add a new implementation of the payment state machine.
This is based in the `pay` plugin concept of payment modifiers,
but here we take it to the next level.
The payment goes through a virtual machine that includes calling
functions and evaluating conditions.
2024-05-08 22:32:13 -05:00
Lagrang3
4b9fe23293 renepay: the plugin executable is main.c
- move the plugin executable from pay.c to main.c,
- adapt the plugin main program to handle concurrent pay calls,
2024-05-08 22:32:13 -05:00
Lagrang3
190649585f renepay: THE plugin global structure
The plugin global structure is now defined in payplugin.h
2024-05-08 22:32:13 -05:00
Lagrang3
7ffb722b27 renepay: refactor uncertainty API
The uncertainty structure is updated based on the result of a route,
previously chan_extra and pay_flow were used instead.
2024-05-08 22:32:13 -05:00
Lagrang3
2a4a23a362 renepay: rename uncertainty_network for uncertanty 2024-05-08 22:32:13 -05:00
Lagrang3
4c91bc0ef0 renepay: abandon pay_flow for route
The role of the structure pay_flow is not taken by a new structure
called route.
2024-05-08 22:32:13 -05:00
Lagrang3
ba54d678bc renepay: concurrent pay calls
Refactor the payment structure to handle multiple pay requests for the
same or different invoices.
2024-05-08 22:32:13 -05:00
Lagrang3
3cbbaa7ab5 renepay: separate flow and chan_extra
Flow and chan_extra are two different concepts, we keep their
definitions and APIs separate.
2024-05-08 22:32:13 -05:00
Vincenzo Palazzo
e8372e7481 plugin: notify only the plugins with init state
This is fixing a bug that @chrisguida reported,
when we notify a plugin we also notify a plugin
that it is in the middle of initialization.

So imagine the following use case:

- 1. Plugin A sends get manifest
- 2. cln makes a check and generates a warning
- 3. Plugin A is subscribed to the warning or * notification
- 4. Core Lightning gets upset because it receives a warning message but it was waiting for a init.

I think it is still a bug in how Core Lightning handles the init, but also I think we should communicate with the plugin only if it is initialized.

```
lightningd-1 2024-04-23T23:20:34.154Z DEBUG   plugin-clnrest: Notification: {'warning': {'level': 'warn', 'time': '1713914433.818230531', 'timestamp': '2024-04-23T23:20:33.818Z', 'source': 'plugin-bookkeeper', 'log': \"topic 'utxo_deposit' is not a known notification topic\"}}
lightningd-1 2024-04-23T23:20:34.154Z DEBUG   hsmd: new_client: 0
lightningd-1 2024-04-23T23:20:34.154Z **BROKEN** plugin-test_libplugin: Did not receive 'init' yet, but got 'warning' instead
lightningd-1 2024-04-23T23:20:34.155Z INFO    plugin-test_libplugin: Killing plugin: exited before we sent init
lightningd-1 2024-04-23T23:20:34.155Z **BROKEN** plugin-test_libplugin: Plugin marked as important, shutting down lightningd!
lightningd-1 2024-04-23T23:20:34.155Z DEBUG   lightningd: io_break: lightningd_exit
lightningd-1 2024-04-23T23:20:34.155Z DEBUG   lightningd: io_loop: connectd_init
Time-out: can't find [re.compile('Server started with public key')] in logs
{'github_repository': 'ElementsProject/lightning', 'github_sha': '014c1eb383b0a65394cf166b3aa0174cf2077896', 'github_ref': 'refs/pull/7258/merge', 'github_ref_name': 'HEAD', 'github_run_id': 8808124001, 'github_head_ref': 'cguida/onchain_notif', 'github_run_number': 9395, 'github_base_ref': 'master', 'github_run_attempt': '1', 'testname': 'test_self_disable', 'start_time': 1713914432, 'end_time': 1713914612, 'outcome': 'fail'}
----------------------------- Captured stderr call -----------------------------
Did not receive 'init' yet, but got 'warning' insteadlightningd: lightningd/connect_control.c:767: connectd_init: Assertion `ret == ld->connectd' failed.
lightningd: FATAL SIGNAL 6 (version 014c1eb-modded)
0x555b1dc2d6f3 send_backtrace
	common/daemon.c:33
0x555b1dc2d8de crashdump
	common/daemon.c:75
0x7fda4584251f ???
	???:0
0x7fda458969fc ???
	???:0
0x7fda45842475 ???
	???:0
0x7fda458287f2 ???
	???:0
0x7fda4582871a ???
	???:0
0x7fda45839e95 ???
	???:0
0x555b1db98f8e connectd_init
	lightningd/connect_control.c:767
0x555b1dbb0307 main
	lightningd/lightningd.c:1249
0x7fda45829d8f ???
	???:0
0x7fda45829e3f ???
	???:0
0x555b1db6fd64 ???
	???:0
0xffffffffffffffff ???
	???:0
```

Reported-by: @chrisguida
Co-Developed-by: @chrisguida
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2024-05-08 21:57:37 -05:00
Rusty Russell
1c3312c8ac pytest: clean up tests/test_db.py::test_last_tx_psbt_upgrade
Ignore bad gossip, and use the broken_log option rather than manually filtering logs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-08 21:54:50 -05:00
Rusty Russell
61cc3aa887 pytest: catch more broken messages in test_important_plugins.
This can happen, so allow them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-08 21:54:50 -05:00
Rusty Russell
92c28a5bcd Add missing docs and Changelog-Added line.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: new log level `trace` where we moved the very noisiest `debug` logs.
2024-05-08 21:05:49 -05:00
Ken Sedgwick
5b2c237718 drive-by: fix valgrind complaint about uninitialized memory 2024-05-08 21:05:49 -05:00
Ken Sedgwick
e93ad5d124 logging: move noisy gossip recv message to TRACE
```
lightningd-test   | 2024-04-26T13:07:01.581Z DEBUG   031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569-gossipd: handle_recv_gossip: WIRE_CHANNEL_UPDATE
lightningd-test   | 2024-04-26T13:07:01.581Z DEBUG   031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569-gossipd: handle_recv_gossip: WIRE_CHANNEL_UPDATE
lightningd-test   | 2024-04-26T13:07:01.582Z DEBUG   031c64a68e6d1b9e50711336d92b434c584ce668b2fae59ee688bd73713fee1569-gossipd: handle_recv_gossip: WIRE_CHANNEL_UPDATE
```
2024-05-08 21:05:49 -05:00
Ken Sedgwick
788e102482 logging: move two noisy plugin_hook logging entries to TRACE
These are very noisy at the debug level:
DEBUG   lightningd: Calling rpc_command hook of plugin clboss
DEBUG   lightningd: Plugin clboss returned from rpc_command hook call

As seen in https://github.com/ZmnSCPxj/clboss/issues/194
2024-05-08 21:05:49 -05:00
Ken Sedgwick
e3c600b0c8 logging: add TRACE between DEBUG and IO
By moving super-noisy debugging logging to new TRACE level we can
make long-term logging at DEBUG possible w/o removing any logging.

Addresses ([#7279])

Related Issues:
https://github.com/ElementsProject/lightning/issues/6927
https://github.com/chrisguida/smaug/issues/34
https://github.com/ZmnSCPxj/clboss/issues/194
2024-05-08 21:05:49 -05:00
Alex Myers
c3280676e8 lightningd: avoid crash from hsmd not signing anchor spend
Temporary patch to avoid issue #6973

[ Add much more logging! --RR ]

Changelog-Fixed: lightningd: avoid crash on signing failure when trying to spend anchor outputs.
2024-05-07 16:41:47 -05:00
Rusty Russell
a481dbcddd pytest: attempt to reproduce hsm signing failure.
This didn't trigger the bug, but worth explicitly testing: we spend a
to-remote output from a previous unilateral close, to spend an anchor
on a current unilateral close.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-07 16:41:47 -05:00
ShahanaFarooqui
e38c828bf0 docker: Fix jq installation error for focal
`jq` is not available in the default repositories of the Ubuntu Focal image. To resolve this, adding the official jq package repository and then installing it.

Changelog-None.
2024-05-06 22:13:54 -05:00
Rusty Russell
153567d699 pytest: test that the preapprove hooks get called by check.
We add a dev flag so we can decline them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
68c4b83ab1 pay: move preapproveinvoice command out so it can be called bu check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` `keysend` now checks with HSM that it will approve it.
2024-05-06 20:51:19 -05:00
Rusty Russell
e1034eec43 keysend: move preapprovekeysend outside pay loop, so it can be called by check.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` `keysend` now checks with HSM that it will approve it.
2024-05-06 20:51:19 -05:00
Rusty Russell
c7339ea310 pytest: test various preapprove scenarios.
We wire through --dev options into the hsmd, and test preapprove accept and deby
with both old and new protocols.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
bd6cf99e4d lightningd: extend preapproveinvoice and preapprovekeysend to check with HSM.
If they support it, we can actually ask hsmd when we are called in
check mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
6ea95da342 hsmd: add variant of preapprove commands to have it check only, not do anything.
Apparently VLS actually does something when we preapprove: if caller is just
checking we want to tell it not to do that!

I put in a flag so we can test both old and new APIs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
33cfef84d8 hsmd: add preinit message to set HSM dev flags.
This should make VLS's life easier: they can ignore dev flags they
don't understand, but we will know their capabilites after init and so
know what they didn't understand (if required).

The only flag for now is a flag to force failure for "preapprove" calls.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
e99f061909 setconfig: more thorough check for internal config options.
Now we can check that the setconfig would actually parse.  We do this
by handing NULL to the calback to indicate it's a test, which may not
work in general but works for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `check` `setconfig` now checks that the new config setting would be valid.
2024-05-06 20:51:19 -05:00
Rusty Russell
dcbdb85497 libplugin: allow check setconfig on all dynamic options.
We need to pass through setconfig in check mode, then we need to have libplugin
add (and use!) a `check_only` parameter to all option setting.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` `setconfig` on plugin options can now check the config value would be accepted.
2024-05-06 20:51:19 -05:00
Rusty Russell
03db143216 pytest: test the check command in libplugin.
For extra points, we use an async command which calls out to listdatastore
before returning.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
fc21fc56dc libplugin: handle check for our commands.
We don't thoroughly handle `check setconfig`: it would be good to
allow this to do further checking!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
11ffbc305e lightningd: allow plugins to specify cancheck for us to pass check commands through.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: Can now opt in to handle `check` command on their commands, for more thorough checking.
2024-05-06 20:51:19 -05:00
Rusty Russell
16615f4ed5 lightningd: add optional ->check callback to struct json_command.
If this is set, instead of mangling the request in-place, we will hand
it through raw.  Plugins will use this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
cf72fb418e param: generalize check handling a little.
We want to extend it to plugins, and we want it to be allowed to be async for more power,
so rather than not completing the cmd if we're checking, do it in command_check_done()
and call it.

This is cleaner than the special case we had before, and allows check to us all the
normal jsonrpc mechanisms, especially async requests (which we'll need if we want to
hand check requests to plugins!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
e3c4bdf209 renepay: fix gcc -O3 overzealous warning.
gcc-12 (Ubuntu 12.3.0-9ubuntu2) 12.3.0 with -O3 gives:

```
In file included from plugins/renepay/test/../mcf.c:5,
                 from plugins/renepay/test/run-arc.c:13:
ccan/ccan/tal/str/str.h: In function ‘minflow’:
ccan/ccan/tal/str/str.h:43:9: error: ‘errmsg’ may be used uninitialized [-Werror=maybe-uninitialized]
   43 |         tal_fmt_(ctx, TAL_LABEL(char, "[]"), __VA_ARGS__)
      |         ^~~~~~~~
plugins/renepay/test/../mcf.c:1565:15: note: ‘errmsg’ was declared here
 1565 |         char *errmsg;
      |               ^~~~~~
cc1: all warnings being treated as errors
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
b22095b275 ccan: update tal so we can annote allcators as never returning NULL.
This helps with -O3 warnings on these commits.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-06 20:51:19 -05:00
Rusty Russell
168ecb8979 pyln-client: pass through level parameter on command notifications.
Without this, everything came out as level INFO.

Changelog-Fixed: pyln-client: Fix Plugin.notify_message() not to ignore `level` parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-04 11:36:23 -05:00
Erik De Smedt
f1dc64b802 cln_plugin: Support wildcard subscriptions
Adapts `cln_plugin` to make it support wildcard `*`-subscriptions.
2024-04-30 15:24:00 -05:00
Erik De Smedt
27eaa1ef44 Repro: cln-plugin cannot subscribe to wildcard "*"
This test reproduces a bug in the `cln-plugin`-crate.
Core Lightning supports a wildcard `*` that plugins can use to
subscribe to all notifications.

However, `cln-plugin` does not support this case.
It allows the developer to subscribe `*`.
But the plug-in crashes when the first notification is received
2024-04-30 15:24:00 -05:00
Erik De Smedt
95b98cf0ee cln_plugin: Example package subscribing to "*"
Creates an example package that subscribes to all notifications and logs
them. This is useful for testing the behavior of subscribing to "*".

I've also edited the Makefile to ensure that `make` builds the example
and that `make clean` removes the example
2024-04-30 15:24:00 -05:00
Alex Myers
ff7efec723 pay: ignore uncommited channels in listpeerchannels output
Uncommited channels are missing several fields which would normally be
populated by an active channel.  This simply skips them for the purposes
of finding a route.  The particular culprit was:
{
  "peer_id": "038cd9f3679d5b39bb2105978467918d549572de472f07dd729e37c7a6377d41d5",
  "peer_connected": true,
  "state": "OPENINGD",
  "owner": "lightning_openingd",
  "opener": "local",
  "to_us_msat": 8317559000,
  "total_msat": 8317559000,
  "features": [
    "option_static_remotekey",
    "option_anchors_zero_fee_htlc_tx"
  ]
}

Fixes #7197 - SEGV in direct_pay_listpeerchannels when field private missing

Changelog-Fixed: Fixed crash in pay plugin caused by parsing uncommitted dual open channels
2024-04-30 14:52:22 -05:00
Alex Myers
56d5d45448 pytest: test pay during channel open
Notably this parses the listpeerchannels output while the state is
still uncommitted, i.e., state = OPENINGD.
2024-04-30 14:52:22 -05:00
daywalker90
dc099a046c msggen: add disableoffer method 2024-04-25 14:23:30 +02:00