Commit Graph

11966 Commits

Author SHA1 Message Date
Rusty Russell
2752e04f8f db: add scid field to channels table.
Normally, we'd use the delete_columns function to remove the old
`short_channel_id` string field, *but* we can't do that for sqlite, as
there are other tables with references to it.  So add a FIXME to do
it once everyone has upgraded to an sqlite3 which has native support
for column deletion.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
e286c38c6f wallet: use db_col_scid / db_bind_scid where possible.
Without helpers, we were using the u64 functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
33a6b18891 db/bindings: rename db_bind_short_channel_id to db_bind_short_channel_id_str, add db_bind_scid.
Although it's deprecated already (because it stores as string), it's
better to make the name explicit.  And create a new helper which stores as BIGINT.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
63457229cb wallet: replace forwarded_payments table with forwards table.
This one directly contains the scids of the channels involved, not
references, so can outlive the channels.  As a side-effect, however,
it now never lists `payment_hash`.  Having it listed (via join) is not
possible as it is a *string* in the channels table, and difficult
anyway because of channel aliases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
2022e4a7a9 wallet: simplify payments lookup so sqlite3 uses index.
Filtering by status is rare, so we can do it in the caller; just let sqlite3
filter by payment_hash.

With ~650,000 payments in db:

Before:
```
129/300000 complete 5.60/sec (33078 invs, 169 pays, 0 retries) in 30 seconds. 19 hours-14 hours remaining.
201/300000 complete 7.20/sec (43519 invs, 241 pays, 0 retries) in 40 seconds. 16 hours-11 hours remaining.
257/300000 complete 5.60/sec (54568 invs, 289 pays, 0 retries) in 50 seconds. 16 hours-14 hours remaining.
305/300000 complete 4.80/sec (65772 invs, 337 pays, 0 retries) in 60 seconds. 16 hours-17 hours remaining.
361/300000 complete 5.60/sec (75875 invs, 401 pays, 0 retries) in 70 seconds. 16 hours-14 hours remaining.
```

After:
```
760/300000 complete 40.00/sec (19955 invs, 824 pays, 0 retries) in 20 seconds. 2 hours-2 hours remaining.
1176/300000 complete 41.60/sec (30082 invs, 1224 pays, 0 retries) in 30 seconds. 2 hours-119 minutes remaining.
1584/300000 complete 40.80/sec (40224 invs, 1640 pays, 0 retries) in 40 seconds. 2 hours-2 hours remaining.
1984/300000 complete 40.00/sec (49938 invs, 2048 pays, 0 retries) in 50 seconds. 2 hours-2 hours remaining.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
2a5660b3bc lightningd: index to speed up sendpay / listsendpays
contrib/giantnode.py shows we're spending a lot of time looking up
payments by payment_hash: sendpays does it to see if we've already
paid, and bookkeeper does it in listsendpays:

```
-   94.52%     0.00%  lightningd  lightningd            [.] read_json
   - 94.52% read_json                                                
      - 94.48% parse_request                                         
         - 94.46% plugin_hook_call_rpc_command                       
            - plugin_hook_call_                                      
               - rpc_command_hook_final                              
                  - 94.46% command_exec                              
                     - 49.08% json_sendpay                           
                        - 49.01% send_payment                        
                           - 48.86% send_payment_core                
                              - 48.84% wallet_payment_list           
                                 - 48.80% db_step                    
                                    + db_sqlite3_step                
                     - 45.38% json_listsendpays                      
                        - 45.36% wallet_payment_list                 
                           - 45.30% db_step                          
                              + 45.30% db_sqlite3_step               
```

This doesn't actually make much of a difference, so see next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
4cab396cc8 autoclean: handle cleaning of old payments (not just invoices).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
660c9af1d9 autoclean: allow cleaning of paid invoices too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
7da51892e8 autoclean: save stats on how much we cleaned.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
17858c9490 lightningd: deprecated "delexpiredinvoice", put functionality in autoclean plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `delexpiredinvoice`: use `autoclean-once`.
2022-09-22 15:19:46 +02:00
Rusty Russell
bd76a196f5 autoclean: new interface
In preparation for more things being autocleaned.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
41a52929f7 libplugin: handle JSON reply after command freed.
This can happen, and in fact does below in our test_autoclean_once
test where we update the datastore, and return from the cmd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Christian Decker
4693803c35 ci: Use the new make upgrade-version target to manage versions 2022-09-22 11:41:11 +02:00
Christian Decker
8ae0af7962 pyln: Bump pyln-client dependency in pyln-testing
They were contradicting each other.
2022-09-22 11:41:11 +02:00
Christian Decker
aace5b51ef pyln: Adjust the auto-publish task to trigger on tags
I was wondering why the workflow never published, turns out we were triggering on
branch pushes only, not tags. So the branch would get pushed, but the tag is pushed
afterwards, and thus not triggering.
2022-09-22 11:41:11 +02:00
Rusty Russell
7c2c100145 pyln: results of make update-pyln-versions NEW_VERSION=0.12.0
But I had to make the pyln-proto requirement of pyln-client less
strict.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 11:41:11 +02:00
Rusty Russell
0514269f48 Makefile: add targets to upgrade pyln versions, push releases.
And use those in MAKING-RELEASES.md

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 11:41:11 +02:00
Rusty Russell
3c80b22d6f pyln: use poetry version, add target to check version, use poetry publish.
Remove non-working test-release target.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 11:41:11 +02:00
niftynei
910116c9e6 build-release: configure before submodcheck
We added a submod dep (lowdown) that requires config to run first.
2022-09-22 11:36:10 +02:00
Christian Decker
b9a7f36ab3 msggen: Add conversion from cln-rpc to cln-grpc for Option<Outpoint>
We didn't have optional Outpoints as arguments so far, so let's
backfill that.

Changelog-Changed: cln-rpc: The `wrong_funding` argument for `close` was changed from `bytes` to `outpoint`
2022-09-22 10:41:05 +02:00
Kristaps Kaupe
775d6ba193 doc: Fix wrong_funding description in manpage and type in schema 2022-09-22 10:41:05 +02:00
Rusty Russell
0868fa9f1e lightningd: allow extra tlv types in non-experimental mode.
The old `experimental-accept-extra-tlv-types` is now `accept-htlc-tlv-types`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `accept-htlc-tlv-types` lets us accept unknown even HTLC TLV fields we would normally reject on parsing (was EXPERIMENTAL-only `experimental-accept-extra-tlv-types`).
2022-09-22 17:19:11 +09:30
Rusty Russell
df4b477e88 keysend: allow extratlvs parameter, even in non-experimental mode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `keysend` now has `extratlvs` option in non-EXPERIMENTAL builds.
2022-09-22 17:19:11 +09:30
Rusty Russell
ce0f544073 keysend: try to find description in TLV.
"Who needs specs?"  FFS...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `keysend` will now attach the longest valid text field in the onion to the invoice (so you can have Sphinx.chat users spam you!)
2022-09-22 17:19:11 +09:30
antongalitch
41502be60a Fix a small typo 2022-09-22 10:00:35 +09:30
Christian Decker
4355612646 gh: Mark some derived files as such
These should never be merged manually, rather have the local copy
remain untouched and ask them to regenerate altogether. Also not
showing in Github so reviewers don't get confused.
2022-09-21 14:33:13 +02:00
Rusty Russell
0db01c882f pytest: fix flake in test_sendcustommsg
We assume that because we've told l3 to shut down, l2 already sees it
as disconnected.  But CI is ...slow... today!

```
        # `l3` is disconnected and we can't send messages to it
>       assert(not l2.rpc.listpeers(l3.info['id'])['peers'][0]['connected'])
E       assert not True

tests/test_misc.py:2218: AssertionError
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-21 12:15:48 +02:00
Rusty Russell
5a1c2447cb pyln-client: use f strings to concatenate JSON ids, handle older integer ids.
We don't really support using pyln-client with older Core Lightning versions,
but this is neater anyway.  I checked: f-strings go back to python 3.6, so
we can use them (I think this may be the first!).

Suggested-by: @MiWCryptAnalytics
Fixes: #5609
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-21 12:15:48 +02:00
Christian Decker
774d16a72e openingd: Fail if dust and max_htlcs result in 0output commitment tx
Prior to this we might end up with a commitment transaction without
any outputs, if combined with `--dev-allowdustreserve`. Otherwise the
reserve being larger than dust means the funder could not drop its
direct output to be below dust.

Reported-by: Rusty Russell <@rustyrussell>
2022-09-21 11:25:47 +02:00
Christian Decker
75fe1c1a66 common: Add multiplication primitives for amount_msat and amount_sat
The scale variants weren't usable since they use `double` as a scale
factor, which in turn causes imprecisions.
2022-09-21 11:25:47 +02:00
Christian Decker
493a0dfcd4 pytest: Exercise all dust zeroreserve case
This just shows we need to add that constraint.

Reported-by: Rusty Russell <@rustyrussell>
2022-09-21 11:25:47 +02:00
Christian Decker
759fcb64a8 pay: If the channel_hint matches our allocation allow it
It means we consume the channel completely to the best of our
knowledge, so let that through.

Changelog-Fixed: pay: Squeezed out the last `msat` from our local view of the network
2022-09-21 11:25:47 +02:00
Christian Decker
bdda62e1a4 pytest: Add test for mixed zeroreserve funding
Tests that we can have zeroreserve nodes accept and open new channels
with non-zeroreserve nodes, without throwing errors
2022-09-21 11:25:47 +02:00
Christian Decker
54b4baabb0 opening: Add dev-allowdustreserve option to opt into dust reserves
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
2022-09-21 11:25:47 +02:00
Christian Decker
1bd3d8d9f9 openingd: Remove dust check for reserve imposed on us
This check, while in line with the specification, would cause issues
in mixed setups when the funder or fundee allows dust reserves, but
the counterparty does not. It is not an issue for the non-dust reserve
node since in this case it's the peer giving us more flexibility not
the other way around.
2022-09-21 11:25:47 +02:00
Christian Decker
67467213cb opening: Add dev-allowdustreserve option to opt into dust reserves
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
2022-09-21 11:25:47 +02:00
Christian Decker
c5b2aee5c6 pytest: Add a zeroreserve test 2022-09-21 11:25:47 +02:00
Christian Decker
c3e9cb7a47 openingd: Add zeroconf-no-really-zero mode
This is incompatible with the spec as it removes the enforcement for
reserves being above dust, but from what I can see from other
implementations it seems that others have allowed this as well.

This commit just guards the necessary changes with compilation guards, so
we can decide either way quickly. This part of the PR is not intended
to be final, just as a discussion basis.
2022-09-21 11:25:47 +02:00
Christian Decker
2def843dce pay: Allow using a channel on equality of estimated capacity
In the case of the local channel we set the estimation to the exact
value spendable, which is important when we want to drain a channel,
because there we actually want to get the last msat.
2022-09-21 11:25:47 +02:00
Christian Decker
5a54f450bd openingd: Pass reserve down to openingd when funding 2022-09-21 11:25:47 +02:00
Christian Decker
8d6423389a openingd: Wire reserve value through to openingd 2022-09-21 11:25:47 +02:00
Christian Decker
9a97f8c154 plugin: Add reserve to openchannel hook result
Changelog-Added: plugin: The `openchannel` hook may return a custom absolute `reserve` value that the peer must not dip below.
2022-09-21 11:25:47 +02:00
Christian Decker
5c1de8029a openingd: Add reserve to fundchannel and multifundchannel
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel` and `fundchannel_start` now accept a `reserve` parameter to indicate the absolute reserve to impose on the peer.
2022-09-21 11:25:47 +02:00
Christian Decker
7159a25e73 openingd: Add method to set absolute reserve 2022-09-21 11:25:47 +02:00
Christian Decker
fcd2320de7 gha: Make the setup and build scripts exit if anything fails
Caused quite a few headaches and red herrings in the past, so let's be
stricter.

Changelog-None
2022-09-20 10:33:40 +02:00
Christian Decker
ab95d2718f pyln: Reduce dependency strictness for pyln-testing 2022-09-20 10:33:40 +02:00
Greg Sanders
248d60d7bd Don't report redundant feerates to subdaemons 2022-09-19 19:10:35 +02:00
Rusty Russell
5b58eda748 libplugin: mark the cmd notleak() whenever command_still_pending() called.
This is what we do in lightningd, which makes memleak much more forgiving:
you can hang temporaries off cmd without getting reports of leaks (also
when send_outreq called).

We remove all the notleak() calls in plugins which worked around this!
And avoid multiple notleak labels, since both send_outreq() and
command_still_pending() can be called multiple times.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell
701dd3dcef memleak: remove exclusions from memleak_start()
Add memleak_ignore_children() so callers can do exclusions themselves.

Having two exclusions was always such a hack!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell
3380f559f9 memleak: simplify API.
Mainly renaming.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30