Previous commit was a hack which *always* batched where possible, this
is a more sophisticated opt-in varaint, with a timeout sanity check.
Final performance for cleaning up 1M pays/forwards/invoices:
```
$ time l1-cli autoclean-once succeededpays 1
{
"autoclean": {
"succeededpays": {
"cleaned": 1000000,
"uncleaned": 26895
}
}
}
real 6m9.828s
user 0m0.003s
sys 0m0.001s
$ time l2-cli autoclean-once succeededforwards 1
{
"autoclean": {
"succeededforwards": {
"cleaned": 1000000,
"uncleaned": 40
}
}
}
real 3m20.789s
user 0m0.004s
sys 0m0.001s
$ time l3-cli autoclean-once paidinvoices 1
{
"autoclean": {
"paidinvoices": {
"cleaned": 1000000,
"uncleaned": 0
}
}
}
real 6m47.941s
user 0m0.001s
sys 0m0.000s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `batching` command to allow database transactions to cross multiple back-to-back JSON commands.
This shows the benefits of batching when being slammed by autoclean:
Before:
```
plugin-autoclean: last 1000 deletes took 6699022 nsec each
plugin-autoclean: last 1000 deletes took 6734025 nsec each
plugin-autoclean: last 1000 deletes took 6681189 nsec each
plugin-autoclean: last 1000 deletes took 6597148 nsec each
plugin-autoclean: last 1000 deletes took 6637085 nsec each
plugin-autoclean: last 1000 deletes took 6801425 nsec each
plugin-autoclean: last 1000 deletes took 6788572 nsec each
plugin-autoclean: last 1000 deletes took 6603641 nsec each
plugin-autoclean: last 1000 deletes took 6642947 nsec each
plugin-autoclean: last 1000 deletes took 6590495 nsec each
plugin-autoclean: last 1000 deletes took 6695076 nsec each
plugin-autoclean: last 1000 deletes took 6477981 nsec each
```
After:
```
plugin-autoclean: last 1000 deletes took 342764 nsec each
plugin-autoclean: last 1000 deletes took 375031 nsec each
plugin-autoclean: last 1000 deletes took 357564 nsec each
plugin-autoclean: last 1000 deletes took 381581 nsec each
plugin-autoclean: last 1000 deletes took 337989 nsec each
plugin-autoclean: last 1000 deletes took 329391 nsec each
plugin-autoclean: last 1000 deletes took 328322 nsec each
plugin-autoclean: last 1000 deletes took 372810 nsec each
plugin-autoclean: last 1000 deletes took 351228 nsec each
plugin-autoclean: last 1000 deletes took 413885 nsec each
plugin-autoclean: last 1000 deletes took 348317 nsec each
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
autoclean was using 98% of its time in memmove; we should simply keep
an offset, and memmove when it's empty. And also, only memmove the
used region, not the entire buffer!
Running on product of giantnodes.py:
$ time l1-cli autoclean-once failedpays 1
{
"autoclean": {
"failedpays": {
"cleaned": 26895,
"uncleaned": 1000000
}
}
}
Before:
real 20m46.579s
user 0m0.000s
sys 0m0.001s
After:
real 2m10.568s
user 0m0.001s
sys 0m0.000s
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And remove deprecated autocleaninvoice docs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `autoclean-status` command to see what autoclean is doing.
Changelog-Deprecated: JSON-RPC: `autocleaninvoice` (use option `autoclean-expiredinvoices-age`)
It's more natural: we will eventually support dynamic config variables,
so this will be quite nice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `autoclean` can now delete old forwards, payments, and invoices automatically.
And take the opportunity to rename l0 and l1 in the tests to the
more natural l1 l2 (since we add l3).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using `listfowards` for this wrong; expose this directly if people
care (and unlike listforwards, which could be deleted, we have to
remember these while the channel is still open!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listhtlcs` new command to list all known HTLCS.
And document that we never know payment_hash.
Changelog-Added: JSON-RPC: `listforwards` now shows `in_htlc_id` and `out_htlc_id`
Changelog-Changed: JSON-RPC: `listforwards` now never shows `payment_hash`; use `listhtlcs`.
Normally, we'd use the delete_columns function to remove the old
`short_channel_id` string field, *but* we can't do that for sqlite, as
there are other tables with references to it. So add a FIXME to do
it once everyone has upgraded to an sqlite3 which has native support
for column deletion.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Although it's deprecated already (because it stores as string), it's
better to make the name explicit. And create a new helper which stores as BIGINT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This one directly contains the scids of the channels involved, not
references, so can outlive the channels. As a side-effect, however,
it now never lists `payment_hash`. Having it listed (via join) is not
possible as it is a *string* in the channels table, and difficult
anyway because of channel aliases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
contrib/giantnode.py shows we're spending a lot of time looking up
payments by payment_hash: sendpays does it to see if we've already
paid, and bookkeeper does it in listsendpays:
```
- 94.52% 0.00% lightningd lightningd [.] read_json
- 94.52% read_json
- 94.48% parse_request
- 94.46% plugin_hook_call_rpc_command
- plugin_hook_call_
- rpc_command_hook_final
- 94.46% command_exec
- 49.08% json_sendpay
- 49.01% send_payment
- 48.86% send_payment_core
- 48.84% wallet_payment_list
- 48.80% db_step
+ db_sqlite3_step
- 45.38% json_listsendpays
- 45.36% wallet_payment_list
- 45.30% db_step
+ 45.30% db_sqlite3_step
```
This doesn't actually make much of a difference, so see next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This can happen, and in fact does below in our test_autoclean_once
test where we update the datastore, and return from the cmd.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I was wondering why the workflow never published, turns out we were triggering on
branch pushes only, not tags. So the branch would get pushed, but the tag is pushed
afterwards, and thus not triggering.
We didn't have optional Outpoints as arguments so far, so let's
backfill that.
Changelog-Changed: cln-rpc: The `wrong_funding` argument for `close` was changed from `bytes` to `outpoint`
The old `experimental-accept-extra-tlv-types` is now `accept-htlc-tlv-types`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `accept-htlc-tlv-types` lets us accept unknown even HTLC TLV fields we would normally reject on parsing (was EXPERIMENTAL-only `experimental-accept-extra-tlv-types`).
"Who needs specs?" FFS...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `keysend` will now attach the longest valid text field in the onion to the invoice (so you can have Sphinx.chat users spam you!)
These should never be merged manually, rather have the local copy
remain untouched and ask them to regenerate altogether. Also not
showing in Github so reviewers don't get confused.
We assume that because we've told l3 to shut down, l2 already sees it
as disconnected. But CI is ...slow... today!
```
# `l3` is disconnected and we can't send messages to it
> assert(not l2.rpc.listpeers(l3.info['id'])['peers'][0]['connected'])
E assert not True
tests/test_misc.py:2218: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't really support using pyln-client with older Core Lightning versions,
but this is neater anyway. I checked: f-strings go back to python 3.6, so
we can use them (I think this may be the first!).
Suggested-by: @MiWCryptAnalytics
Fixes: #5609
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Prior to this we might end up with a commitment transaction without
any outputs, if combined with `--dev-allowdustreserve`. Otherwise the
reserve being larger than dust means the funder could not drop its
direct output to be below dust.
Reported-by: Rusty Russell <@rustyrussell>
It means we consume the channel completely to the best of our
knowledge, so let that through.
Changelog-Fixed: pay: Squeezed out the last `msat` from our local view of the network
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
This check, while in line with the specification, would cause issues
in mixed setups when the funder or fundee allows dust reserves, but
the counterparty does not. It is not an issue for the non-dust reserve
node since in this case it's the peer giving us more flexibility not
the other way around.
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
