Commit graph

37 commits

Author SHA1 Message Date
Rusty Russell
c797b6fb20 libplugin: add method string to jsonrpc callbacks, implement generic helpers.
Without knowing what method was called, we can't have useful general logging
methods, so go through the pain of adding "const char *method" everywhere,
and add:

1. ignore_and_complete - we're done when jsonrpc returned
2. log_broken_and_complete - we're done, but emit BROKEN log.
3. plugin_broken_cb - if this happens, fail the plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-11-07 17:04:35 +10:30
Rusty Russell
c5099b1647 libplugin: clean up API.
When we used to allow cmd to be NULL, we had to hand the plugin
everywhere.  We no longer do.

1. Various jsonrpc_ functions no longer need the plugin arg.
2. send_outreq no longer needs a plugin arg.
3. The init function takes a command, not a plugin.
4. Remove command_deprecated_in_nocmd_ok.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-11-07 17:04:35 +10:30
Rusty Russell
9a6325c4f3 doc: properly document sub-subobject fields.
Inside listpeerchannels, there's an object called `updates`, and inside that objects `local` and `remote`.  We flatten this, but the documentation doesn't mention the `updates` field at all.

Ideally we would rename these fields to include `updates_` but that would be a breaking change.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

```
     - `row` (reference to `peerchannels_channel_type.rowid`, sqltype `INTEGER`)
     - `arrindex` (index within array, sqltype `INTEGER`)
     - `names` (type `string`, sqltype `TEXT`)
-  - `local_htlc_minimum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `local`)
-  - `local_htlc_maximum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `local`)
-  - `local_cltv_expiry_delta` (type `u32`, sqltype `INTEGER`, from JSON object `local`)
-  - `local_fee_base_msat` (type `msat`, sqltype `INTEGER`, from JSON object `local`)
-  - `local_fee_proportional_millionths` (type `u32`, sqltype `INTEGER`, from JSON object `local`)
-  - `remote_htlc_minimum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `remote`)
-  - `remote_htlc_maximum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `remote`)
-  - `remote_cltv_expiry_delta` (type `u32`, sqltype `INTEGER`, from JSON object `remote`)
-  - `remote_fee_base_msat` (type `msat`, sqltype `INTEGER`, from JSON object `remote`)
-  - `remote_fee_proportional_millionths` (type `u32`, sqltype `INTEGER`, from JSON object `remote`)
+  - `local_htlc_minimum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `updates.local`)
+  - `local_htlc_maximum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `updates.local`)
+  - `local_cltv_expiry_delta` (type `u32`, sqltype `INTEGER`, from JSON object `updates.local`)
+  - `local_fee_base_msat` (type `msat`, sqltype `INTEGER`, from JSON object `updates.local`)
+  - `local_fee_proportional_millionths` (type `u32`, sqltype `INTEGER`, from JSON object `updates.local`)
+  - `remote_htlc_minimum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `updates.remote`)
+  - `remote_htlc_maximum_msat` (type `msat`, sqltype `INTEGER`, from JSON object `updates.remote`)
+  - `remote_cltv_expiry_delta` (type `u32`, sqltype `INTEGER`, from JSON object `updates.remote`)
+  - `remote_fee_base_msat` (type `msat`, sqltype `INTEGER`, from JSON object `updates.remote`)
+  - `remote_fee_proportional_millionths` (type `u32`, sqltype `INTEGER`, from JSON object `updates.remote`)
   - `ignore_fee_limits` (type `boolean`, sqltype `INTEGER`)
   - `lost_state` (type `boolean`, sqltype `INTEGER`)
   - `feerate_perkw` (type `u32`, sqltype `INTEGER`, from JSON object `feerate`)
```
2024-08-16 07:19:52 +09:30
Rusty Russell
054eeb8aa2 plugins/sql: fix crash when subobject is missing which has subobject inside it.
There's only one place (added in v24.02) where we have nested subobjects, and
if the outer sub-object is not there it crashes.

```
sql: FATAL SIGNAL 11 (version v24.08rc1-2-ge134f2f-modded)
0x5781d447239b send_backtrace
        common/daemon.c:33
0x5781d4472432 crashdump
        common/daemon.c:75
0x787bcd64531f ???
        ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x5781d447782a json_get_membern
        common/json_parse_simple.c:210
0x5781d44778cc json_get_member
        common/json_parse_simple.c:223
0x5781d445d0e2 process_json_obj
        plugins/sql.c:537
0x5781d445d10d process_json_obj
        plugins/sql.c:538
0x5781d445d598 process_json_list
        plugins/sql.c:684
0x5781d445d65a process_json_result
        plugins/sql.c:699
0x5781d445d7cb default_list_done
        plugins/sql.c:722
0x5781d446349d handle_rpc_reply
        plugins/libplugin.c:1016
0x5781d4463640 rpc_read_response_one
        plugins/libplugin.c:1202
0x5781d44636f1 rpc_conn_read_response
        plugins/libplugin.c:1226
0x5781d459e56c next_plan
        ccan/ccan/io/io.c:60
0x5781d459ea3d do_plan
        ccan/ccan/io/io.c:422
0x5781d459eafa io_ready
        ccan/ccan/io/io.c:439
0x5781d45a0469 io_loop
        ccan/ccan/io/poll.c:455
0x5781d4463dcb plugin_main
        plugins/libplugin.c:2230
```

Changelog-Fixed: Plugins: `sql` crash on querying `listpeerchannels` during channel establishment.
Fixes: https://github.com/ElementsProject/lightning/issues/7505
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-16 07:19:52 +09:30
Rusty Russell
11bfbf5deb libplugin: add data pointer for plugin convenience.
This avoids globals (and means memleak traverses the variables!): we
only change over the test plugin though, to avoid unnecessary churn.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-08-07 20:35:30 +09:30
ShahanaFarooqui
2ff3e55f08 plugin: Removing category, description and long_description from plugin_command struct
Changelog-None.
2024-07-31 14:42:58 +09:30
Joseph Goulden
89ede8a208 plugins/sql: allow datetime functions
Changelog-Changed: Plugins:  now allows date and time sqlite functions.
2024-07-19 15:40:06 +09:30
Rusty Russell
f7afe1a35f plugins: two minor things I noticed.
1. sql's dev-sqlfilename should be registered as a dev option.
2. bcli's timeout is an integer, not a string.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-05-15 15:50:54 -05:00
Rusty Russell
9450d46db1 bitcoin/short_channel_id: pass by copy everywhere.
It's a u64, we should pass by copy.  This is a big sweeping change,
but mainly mechanical (change one, compile, fix breakage, repeat).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-03-20 13:51:48 +10:30
Rusty Russell
e0e879c003 common: remove type_to_string files altogther.
This means including <common/utils.h> where it was indirectly included.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-03-20 13:51:48 +10:30
Rusty Russell
37d22f9141 global: change all type_to_string to fmt_X.
This has the benefit of being shorter, as well as more reliable (you
will get a link error if we can't print it, not a runtime one!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-03-20 13:51:48 +10:30
Rusty Russell
d8c06dccac global: expose all fmt_X functions for direct use, make uniform.
We have various functions to convert to a string, rename them all so we can
count on fmt_X being the formatter for struct X, and make them all return
`char *`.

Sometimes they existed but were private, sometimes they had a
different name.  Most take a pointer, but simple types pass by copy:
short_channel_id, amount_msat and amount_sat.

The following public functions changed:
1. psbt_to_b64 -> fmt_wally_psbt.
2. pubkey_to_hexstr -> fmt_pubkey.
3. short_channel_id_to_str -> fmt_short_channel_id (scid by copy now!)
4. fmt_signature -> fmt_secp256k1_ecdsa_signature
5. fmt_amount_sat/fmt_amount_msat pass copy not pointer, return non-const char *.
6. node_id_to_hexstr -> fmt_node_id

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-03-20 13:51:48 +10:30
Rusty Russell
f2f43eeffa gossipd: strip private updates from gossip_store on startup.
We rename them to _obs, too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-01-31 14:47:33 +10:30
Rusty Russell
a8bdde0667 plugins/sql: use per-command deprecations.
In this case the cmd is `sql` but the field we're talking about is from
a different command, so we need a new libplugin API.

Note: there are still no deprecations in any tables used by `sql`, so this
is a bit moot for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-01-26 10:30:22 +10:30
Rusty Russell
1131e40ddf plugins/sql: do deprecations at request time, not schema loading.
When we allow deprecation to be set per-connection, we need to
generalize this approach.  Instead of filtering out deprecated
fields at schema loading, we need to load them, then refuse
to serve deprecated fields on a per-request basis.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-01-26 10:30:22 +10:30
Rusty Russell
46ba105518 plugins/sql: make columns an array of pointers.
`struct column` has a dynamically allocated member, which is neater if
it is a tal object itself (we allocated the member off the array, instead).
We're about to add two new members, so clean this up first.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-01-26 10:30:22 +10:30
Rusty Russell
dadbdf488c schemas: deprecated is now a range.
Don't assume removal is +6 months, but have a start deprecation/end support range.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-01-26 10:30:22 +10:30
Rusty Russell
f725edad62 plugins: remove #if DEVELOPER.
And rename dev-only-option `use_shadow` to `dev_use_shadow`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-09-21 20:08:24 +09:30
Rusty Russell
b5d1708900 plugins/sql: make testing-only sqlfilename option dev-sqlfilename.
dev options don't need to be documented, and this is only for debugging
the plugin itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-08-18 20:29:34 +09:30
Rusty Russell
aa5c7e763f libplugin: make set callback for options take plugin ptr, check correct type.
I added a plugin arg and was surprised that compile didn't break.
This is because typesafe_cb et al are conditional casts: if the type
isn't as expected it has no effect, but we're passing plugin_option() through
varargs, so everything is accepted!

Add a noop inline to check type, and fix up the two cases where we
used `const char *` instead of `char *`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-20 20:08:25 +09:30
Rusty Russell
d818614aa9 plugins/sql: recurse correctly into complex objects during processing.
We didn't handle the case of an array inside a subobject.  But that
happens when we add the next commit!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell
c9ddf9d1c3 plugins/sql: handle case of subobject with sub-arrays.
i.e. recurse properly in SQL generation.  This is about to happen.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-25 15:28:02 +10:30
Rusty Russell
1e2bc665ae sql: fix nodes table update.
Without this patch, we only ever loaded the "nodes" table once, then
didn't see updates.

How this ever got past CI is a mystery; perhaps valgrind was so slow that
the updated node_announcement hit the gossmap before we even asked sql
on l3 about the nodes table?

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `sql` nodes table now gets refreshed when gossip changes.
2023-03-06 16:14:28 -06:00
Rusty Russell
df0661ce22 sql: fix bug where nodes table would get duplicate entries.
As soon as we apply the next commit, we get a new problem: the
delete code didn't work.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-06 16:14:28 -06:00
Rusty Russell
f3baa3e510 sql: fix crash on fresh node_announcment.
Missing quotes when we delete the old one!

Reported-by: Alex Myers
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-28 14:11:12 -06:00
Rusty Russell
f87c7ed439 plugins/sql: fix foreign keys.
I noticed that our subtables were not being cleaned, despite being "ON
DELETE CASCADE".  This is because foreign keys were not enabled, but
then I got foreign key errors: rowid cannot be a foreign key anyway!

So create a real "rowid" column.  We want "ON DELETE CASCADE" for
nodes and channels (and other tables in future) where we update
partially.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-02-02 14:56:04 -06:00
Rusty Russell
0240c24936 plugins/sql: listsqlschemas command to retrieve schemas.
Good for detection of what fields are present.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
9a591277f5 plugins/sql: allow some simple functions.
And document them!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
14aac0769c doc: document the sql command.
In particular, we generate the schema part from the plugin itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
2023-01-30 15:15:41 -06:00
Rusty Russell
adb8de3e07 plugins/sql: print out part of man page referring to schemas.
We now add tables to the strmap as we allocate them, since we don't
want to call "finish_td" when we're merely invoked for the
documentation, and don't need a database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
20654ebd49 plugins/sql: pay attention to deprecated in schema.
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
9b08c4f25a plugins/sql: refresh listnodes and listchannels by monitoring the gossip_store.
It's quite a lot of code, but these are the most expensive commands we
do so it's worth it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
aa3a1131aa plugins/sql: include the obvious indexes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
51ae7118f1 plugins/sql: make tables for non-object arrays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
c230291141 plugins/sql: rework to parse schemas.
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
260643157d plugins/sql: create struct column to encode column details.
Rather than two arrays "columns" (for names) and "fieldtypes" (for
types), use a struct.  This makes additions easier for successive
patches.

Also pull process_json_obj() out of the loop in list_done().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
24d86a85c3 plugins/sql: initial commit of new plugin.
This is designed to allow you to perform complex server-side
queries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00