Commit Graph

12548 Commits

Author SHA1 Message Date
Rusty Russell
9a591277f5 plugins/sql: allow some simple functions.
And document them!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
14aac0769c doc: document the sql command.
In particular, we generate the schema part from the plugin itself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `sql` plugin command to perform server-side complex queries.
2023-01-30 15:15:41 -06:00
Rusty Russell
adb8de3e07 plugins/sql: print out part of man page referring to schemas.
We now add tables to the strmap as we allocate them, since we don't
want to call "finish_td" when we're merely invoked for the
documentation, and don't need a database.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
20654ebd49 plugins/sql: pay attention to deprecated in schema.
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
40fe893172 doc/schemas: fix old deprecations.
`"deprecated": true` is obsolete; we don't document them anyway.

Where it would have otherwise changed the GRPC wrappers, I actually put the
version number in.

We allow "listchannels" to have "satoshis" since we have some tests
that run in deprecated-api mode.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
9b08c4f25a plugins/sql: refresh listnodes and listchannels by monitoring the gossip_store.
It's quite a lot of code, but these are the most expensive commands we
do so it's worth it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
aa3a1131aa plugins/sql: include the obvious indexes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
68370a203e pytest: perform more thorough testing.
Painfully created by hand from the source.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
8a0ee5f56e plugins/sql: add listpeerchannels support.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
51ae7118f1 plugins/sql: make tables for non-object arrays.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
c230291141 plugins/sql: rework to parse schemas.
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
260643157d plugins/sql: create struct column to encode column details.
Rather than two arrays "columns" (for names) and "fieldtypes" (for
types), use a struct.  This makes additions easier for successive
patches.

Also pull process_json_obj() out of the loop in list_done().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
24d86a85c3 plugins/sql: initial commit of new plugin.
This is designed to allow you to perform complex server-side
queries.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
2c41c5d52d doc: use specific types in schema rather than "hex".
We have "secret" and "hash" types which are often more appropriate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
fa127a4071 doc/schemas: remove unnecessary length restrictions.
If we have a specific type (not just "hex") the length is implied.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
9589ea0240 common: add routine to get double from JSON.
I don't like it, but we do expose some times like this :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
eb6b8551d4 tools/fromschema.py: don't try to handle more complex cases.
We only handle top-level objects with an array of objects:
make sure it is one before we call the routines.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
153b7bf192 common/gossip_store: move subdaemon-only routines to connectd.
connectd is the only one who uses these routines now.  The
rest can be linked into a plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
7e8b93daa1 common/gossip_store: expose routine to read one header.
This is useful when you're writing routines to scan it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
0274d88bad common/gossip_store: clean up header.
It's actually two separate u16 fields, so actually treat it as
such!

Cleans up zombie handling code a bit too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
83c690fe5f doc: fix listsendpays man page.
We actually had a partid allowed (in the oneOf clauses), but didn''t
document it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
d6f46e2373 lightningd: fix type of listhtlcs payment_hash.
`hash` is a tighter requirement than simply `hex`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
9ab488fc41 plugins/topology: add direction field to listchannels.
It's a core concept in the spec which isn't directly exposed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
2023-01-30 15:15:41 -06:00
Rusty Russell
611795beee listtransactions: get rid of per-tx type annotations.
We didn't actually populate them properly, and the real annotations
are on inputs and outputs.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `listtransactions` `channel` and `type` field removed at top level.
2023-01-30 15:15:41 -06:00
Rusty Russell
578f075407 wallet: remove unused TX_ANNOTATION type in transaction_annotations table.
We only ever use this table for output and input transactions: indeed, my node
doesn't have any annotation types 0.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Rusty Russell
4b9cb7eb76 doc: remove unused offerout schema.
We removed the command for v22.11.

Also, we removed the `refund_for` offer parameter, so remove its description
from the manpage.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 15:15:41 -06:00
Alex Myers
98bfd23fff pytest: test_channel_resurrection now succeeds 2023-01-30 16:33:03 +10:30
Alex Myers
1bae8cd28a gossipd: zombify inactive channels instead of pruning
Though BOLT 7 says a channel may be pruned when one side becomes inactive
and fails to refresh their channel_update, in practice, the
channel_announcement can be difficult to recover if deleted entirely.
Here the channel_announcement is tagged as zombie such that gossip_store
consumers may safely ignore it, but it may be retained should the channel
come back online in the future. Node_announcements and channel_updates may
also be retained in such a fashion until the channel is ready to be
resurrected.

Changelog-Fixed: Pruned channels are more reliably restored.
2023-01-30 16:33:03 +10:30
Alex Myers
6bff10cd40 gossip_store: add a flag for zombie entries
This will allow gossipd to store and persist gossip for channels rather
than deleting them entirely when the channels are pruned from the
network.
2023-01-30 16:33:03 +10:30
Alex Myers
ed4815527a gossipd: avoid gossipd crash due to double freeing timer 2023-01-30 16:33:03 +10:30
Alex Myers
f2f05117aa pytest: gossipd: test resumption of pruned channels 2023-01-30 16:33:03 +10:30
Rusty Russell
a3ca3fb047 common/bolt11: fix 32-bit compilation.
Fixes d9fed06b90:

```
common/bolt11.c:868:31: error: format specifies type 'size_t' (aka 'unsigned long') but the argument has type 'u64' (aka 'unsigned long long') [-Werror,-Wformat]
                                           bech32_charset[type], field_len);
                                                                 ^~~~~~~~~
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 16:30:46 +10:30
Rusty Russell
2dec805465 decode: fix handling of blinded_payinfo.
Our pay code handles this correctly, but decode was still using an old model
where there was a payinfo per hop, not per path.

Reported-by: @t-bast
See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell
6eb7a4cbf2 plugins: update to match latest offers text.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-offers" BOLTVERSION=guilt/offers
```

In this case, only trivial mods.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell
9aefe3d40a common: update to latest onion-message spec.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-onion-message" BOLTVERSION=guilt/offers
```

Mainly textual, though I neatened the extra fields check for TLVs with
blinding, and implemented the "no other fields" requirement for
non-final onion message hops.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell
8e630e7c53 common/test: fix up name of test file to match latest version.
Otherwise it's skipped!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell
e9eb5f493b common: update to latest route-blinding spec.
```
make check-source-bolt CHECK_BOLT_PREFIX="--prefix=BOLT-route-blinding" BOLTVERSION=guilt/offers
```

Other than textual changes, this does:

1. Ensures we put total_amount_msat in onion final hop (reported by @t-bast).
2. Require that they put total_amount_msat in onion final hop.
3. Return `invalid_onion_blinding` exactly as defined by the spec (i.e. less
   aggressive when we're the final hop) (also reported by @t-bast, but I already
   knew).

See: #5823
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (total_amount_sat required, Eclair compat)
2023-01-30 13:24:29 +10:30
Rusty Russell
d5c19b23d8 common/onion_decode: put final flag in onion_payload.
You can use rs->nextcase, but we don't always keep that around, so
keep a flag in onion_payload.

We'll use this in the "do we need to return a blinded error code"
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell
885506765e tools/check-bolt.c: don't leak open directory.
Thanks valgrind!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Rusty Russell
da3506e6a0 wire: use correct number of update_add_tlvs blinding field.
See: #5823
Reported-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `offers` breaking blinded payments change (update_add_tlvs fix, Eclair compat)
2023-01-30 13:24:29 +10:30
Rusty Russell
5958c9c3d6 common/test: remove unused padding in bolt04/blinded-onion-message-onion-test.json
This was reported by @valentinewallace: Dave would only use padding to
make all his own encrypted_recipient_data equal-length.  We did it
across the entire path, which includes the hop added by Alice, which
Dave wouldn't know about.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-30 13:24:29 +10:30
Christian Decker
13fe27c65f gossipd: Do not send warning when node_announcement parsing fails
Changelog-Fixed: gossip: We removed a warning for old `node_announcement` that was causing LND peers to disconnect
2023-01-27 12:58:49 -06:00
Ken Sedgwick
7b2c5617c1 hsmd: increase HSM_MAX_VERSION to 3 2023-01-26 21:10:15 -06:00
Ken Sedgwick
a4dc714cdc hsmd: add hsmd_preapprove_keysend and check_preapprovekeysend pay modifier
Changelog-added: hsmd: A new message `hsmd_preapprove_keysend` is added.
Changelog-added: JSON-RPC: A new command `preapprovekeysend` is added.
2023-01-26 21:10:15 -06:00
Ken Sedgwick
f29343d740 hsmd: add hsmd_preapprove_invoice and check_preapproveinvoice pay modifier
Changelog-added: hsmd: A new message `hsmd_preapprove_invoice` is added.
Changelog-added: JSON-RPC: A new command `preapproveinvoice` is added.
2023-01-26 21:10:15 -06:00
Braydon
9482e0619c docker: Install protobuf-compiler for builder 2023-01-26 15:22:45 +00:00
Riccardo Casatta
182a9cdcb6 cln-rpc: use serde rename instead of alias
rename is necessary to roundtrip, otherwise the rust name is used.

This also remove the rename if they are not necessary.

Note that:
```
 #[serde(rename="foo", skip_serializing_if=="bar")]
 pub field: bool,
```

is equivalent to:
```
 #[serde(rename="foo")]
 #[serde(skip_serializing_if=="bar")]
 pub field: bool,
```

and for simplicity of construction the latter is used
2023-01-26 15:03:54 +01:00
Rusty Russell
fa4b61d13d common/bolt11: convert to table-driven.
Decode functions are now almost entirely uniform, so just use a table.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-25 13:59:34 -06:00
Rusty Russell
cbd0ef4192 common/bolt11: add pull_all helper for common case of entire field.
And make pull_bits return a uniform error message, since that's what
callers want, rather than asserting success.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: pay: don't assert() on malformed BOLT11 strings.
2023-01-25 13:59:34 -06:00
Rusty Russell
d9fed06b90 common/bolt11: const cleanup, fix parsing errors.
Also, we don't need to pass the total length to the field parsers,
just the length for this field (confusingly, this was called
"data_length").

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-25 13:59:34 -06:00