This is fixing a bug that @chrisguida reported,
when we notify a plugin we also notify a plugin
that it is in the middle of initialization.
So imagine the following use case:
- 1. Plugin A sends get manifest
- 2. cln makes a check and generates a warning
- 3. Plugin A is subscribed to the warning or * notification
- 4. Core Lightning gets upset because it receives a warning message but it was waiting for a init.
I think it is still a bug in how Core Lightning handles the init, but also I think we should communicate with the plugin only if it is initialized.
```
lightningd-1 2024-04-23T23:20:34.154Z DEBUG plugin-clnrest: Notification: {'warning': {'level': 'warn', 'time': '1713914433.818230531', 'timestamp': '2024-04-23T23:20:33.818Z', 'source': 'plugin-bookkeeper', 'log': \"topic 'utxo_deposit' is not a known notification topic\"}}
lightningd-1 2024-04-23T23:20:34.154Z DEBUG hsmd: new_client: 0
lightningd-1 2024-04-23T23:20:34.154Z **BROKEN** plugin-test_libplugin: Did not receive 'init' yet, but got 'warning' instead
lightningd-1 2024-04-23T23:20:34.155Z INFO plugin-test_libplugin: Killing plugin: exited before we sent init
lightningd-1 2024-04-23T23:20:34.155Z **BROKEN** plugin-test_libplugin: Plugin marked as important, shutting down lightningd!
lightningd-1 2024-04-23T23:20:34.155Z DEBUG lightningd: io_break: lightningd_exit
lightningd-1 2024-04-23T23:20:34.155Z DEBUG lightningd: io_loop: connectd_init
Time-out: can't find [re.compile('Server started with public key')] in logs
{'github_repository': 'ElementsProject/lightning', 'github_sha': '014c1eb383b0a65394cf166b3aa0174cf2077896', 'github_ref': 'refs/pull/7258/merge', 'github_ref_name': 'HEAD', 'github_run_id': 8808124001, 'github_head_ref': 'cguida/onchain_notif', 'github_run_number': 9395, 'github_base_ref': 'master', 'github_run_attempt': '1', 'testname': 'test_self_disable', 'start_time': 1713914432, 'end_time': 1713914612, 'outcome': 'fail'}
----------------------------- Captured stderr call -----------------------------
Did not receive 'init' yet, but got 'warning' insteadlightningd: lightningd/connect_control.c:767: connectd_init: Assertion `ret == ld->connectd' failed.
lightningd: FATAL SIGNAL 6 (version 014c1eb-modded)
0x555b1dc2d6f3 send_backtrace
common/daemon.c:33
0x555b1dc2d8de crashdump
common/daemon.c:75
0x7fda4584251f ???
???:0
0x7fda458969fc ???
???:0
0x7fda45842475 ???
???:0
0x7fda458287f2 ???
???:0
0x7fda4582871a ???
???:0
0x7fda45839e95 ???
???:0
0x555b1db98f8e connectd_init
lightningd/connect_control.c:767
0x555b1dbb0307 main
lightningd/lightningd.c:1249
0x7fda45829d8f ???
???:0
0x7fda45829e3f ???
???:0
0x555b1db6fd64 ???
???:0
0xffffffffffffffff ???
???:0
```
Reported-by: @chrisguida
Co-Developed-by: @chrisguida
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
We need to pass through setconfig in check mode, then we need to have libplugin
add (and use!) a `check_only` parameter to all option setting.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `check` `setconfig` on plugin options can now check the config value would be accepted.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: Can now opt in to handle `check` command on their commands, for more thorough checking.
We want to extend it to plugins, and we want it to be allowed to be async for more power,
so rather than not completing the cmd if we're checking, do it in command_check_done()
and call it.
This is cleaner than the special case we had before, and allows check to us all the
normal jsonrpc mechanisms, especially async requests (which we'll need if we want to
hand check requests to plugins!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We do this for DEBUG_SUBD already, but I wanted to debug the main lightningd.
(We rename --debugger to the more accurate --dev-debug-self)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unfortunately, this is awkward: we just copy through most requests,
so we can't easily add a "deprecation" field to each one. So we do
a notification if the next command has a different deprecation status
than the global one, but that requires opt-in from the plugin.
We didn't previously document the subscriptions array, so do that now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `deprecated_oneshot` notifiction subscription to change deprecated status for a single command.
I did some CHANGELOG and git digging to see when these were deprecated, and
some were very old (v0.8.2!). But since they didn't warn users loudly, I
chose to do so this release only.
I renamed ld's `deprecated_apis` to `deprecated_ok` to make sure I
caught them all.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We still accept boolean: the plugin may not want to commit to a deprecation schedule.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: rpcmethods and options can set `deprecated` to a pair of version strings, not just a boolean.
```
Already in transaction from lightningd/plugin.c:727
```
There are two callers, and one didn't disable transactions, so do it in plugin_exclusive_loop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We remove it from the pending_requests strmap before calling it,
so it doesn't get called again by destroy_plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We should really unify the cases of a local request, vs a forwarded
request, but for now, don't steal the request onto the plugin, and
if we return from the plugin and the request is gone, don't get upset.
This uncovered a case where we weren't inside a transaction, in
test_hook_crash, so fix that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now the internal code will generate a "PLUGIN_TERMINATED" response
when the plugin dies, we can handle it in plugin_hook.
But we can also simplify it by turning the snapshot of hooks into
a simple array: this means we are robust against any combination of plugins
exiting at any time.
Note: this reveals an issue with test_rpc_command_hook where we run
the request hook again (unexpectedly), so we disable that for the next
patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had special code to fail a forwarded request, but not for an
internally-generated request. Instead, we should pretend the (dead)
plugin responded with a PLUGIN_TERMINATED error, and handle the
request through the normal paths.
This breaks the case where a plugin crashes (or stops itself) in a
hook, so we handle that next.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was always a bit weird they weren't, and it seems a premature
optimization to make the callbacks to this themselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's per-plugin, so why is there a single map for all plugins? It
works because we always make unique ids, but it's weird.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: plugins can now specify (unknown) even messages we should accept from peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And require --developer to use them.
Also refuse redirection to deprecated APIs if deprecated APIs are disabled!
Changelog-Removed: `dev-sendcustommsg` (use `sendcustommsg`, which was added in v0.10.1)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also requires us to expose memleak when !DEVELOPER, however we only
ever used the memleak tracking when the LIGHTNINGD_DEV_MEMLEAK
environment variable was set, so keep that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Reported-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: we clean up properly if a plugin fails to start, and we don't kill all processes if it's from `plugin startdir`.
`struct log` becomes `struct logger`, and the member which points to the
`struct log_book` becomes `->log_book` not `->lr`.
Also, we don't need to keep the log_book in struct plugin, since it has
access to ld's log_book.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Requested-by: Shahana Farooqui @Shahana
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: plugins can subscribe to all notifications using "*".
We usually have access to `ld`, so avoid the global.
The only place generic code needs it is for the json command struct,
and that already has accessors: add one for libplugin and lightningd
to tell it if deprecated apis are OK.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If it's a plugin opt, we'll need a callback, so reshuffle logic. Also
add infra to map option name to plugin and option.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previously, if these failed we always exited; once we have dymamic
configs this would be a (tiny) memory leak, so use tmpctx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
defaults were deprecated in 0df97547dd, but that was a bit
harsh as several plugins do that (summary, for example). So allow false, but warn
that we ignore anything else.
Reported-by: @microsatoshi on Discord.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: ...actually, `default` `false` still accepted on `flag` type parameters.
Memory leak detected by ASan:
==880002==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x5039e7 in malloc (lightningd/lightningd+0x5039e7)
#1 0x7f2e8c203884 in __alloc_dir (/lib64/libc.so.6+0xd2884)
This integrates them with configvars properly: they almost "just work"
in listconfigs now, and we don't put them in a special sub-object
under their plugin.
Unfortunately, this means `listconfigs` now has a loose schema: any
plugin can add something to it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: reloaded plugins get passed any vars from configuration files.
Changelog-Deprecated: Config: boolean plugin options set to `1` or `0` (use `true` and `false` like non-plugin options).
Changelog-Deprecated: Plugins: `default` no longer accepted on `flag` type parameters (it was silently ignored, so just don't set it).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We leave the code in contrib/pyln-client/pyln/client/lightning.py to handle
msat null fields for now, though, for a bit more compatibility.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids any confusion between primitive and string ids, and in
particular stops an issue with commando once it starts chaining ids,
that weird ids can be double-escaped and commando will not recognize
the response, leaving the client hanging. It's the client's fault for
using a weird id, but it's still rude (and triggered by our tests!).
It also makes substituting the id in passthrough simpler, FTW.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was found by tests/test_plugin.py::test_important_plugin and
was NOT a flake!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-None: only just committed
This is a minimal fix: we wait until all plugins reply from init before
continuing. Really large or busy nodes can have other things monopolize
lightningd, then the timer goes off and we blame the plugin (which has
responded, we just haven't read it yet!).
The real answer is to have some timeouts only advance when we're idle,
or have them low-priority so we only activate them when we're idle (this
doesn't apply to all timers: some are probably important!). But
this is a minimal fix for -rc3.
Fixes: https://github.com/ElementsProject/lightning/issues/5736
Changelog-Fixed: plugins: on large/slow nodes we could blame plugins for failing to answer init in time, when we were just slow.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And document support for it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `getmanfest` response can contain `nonnumericids` to indicate support for modern string-based JSON request ids.
Changelog-Deprecated: Plugins: numeric JSON request ids: modern ones will be strings (see doc/lightningd-rpc.7.md!)
Usually the calls are spontanous, so it's just "cln:<method>#NNN", but
json_invoice() calls listincoming, and json_checkmessage calls
listnodes, so those become "cli:invoice-<pid>/cln:listincoming#NNN".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
