Commit Graph

957 Commits

Author SHA1 Message Date
Rusty Russell
662c6931f3 Remove point32.
The x-only dream is dead.  Remove all trace.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell
1cdf21678e offers: print out more details, fix up schema for decode of blinded paths.
We need to print out first_node_id, and "node_id" is now called
"blinded_node_id" in the spec.

And the schema didn't include the payment fields in the blinded path
for invoices (which broke as soon as we actually tested one!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell
e30ea91908 BOLTs: update to more recent bolt12 spec.
It's 2b7ad577d7a790b302bd1aa044b22c809c76e49d, which reverts the
point32 changes.

It also restores send_invoice in `invoice`, which we had removed
from spec and put into the recurrence patch.

I originally had implemented compatibility, but other changes
which followed this are far too widespread.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: complete rework of spec from other teams (yay!) breaks previous compatibility (boo!)
2022-10-26 11:29:06 +10:30
Rusty Russell
eac8401f84 Makefile: separate bolt12 wireobjects
Most things don't want them, so don't link it into everything by default.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell
82d98e4b96 gossmap: move gossmap_guess_node_id to pay plugin.
This removes a point32 dependency.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
Rusty Russell
bed905a394 lightningd: use 33 byte pubkeys internally.
We still use 32 bytes on the wire, but internally don't use x-only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-10-26 11:29:06 +10:30
niftynei
e00857827f funder: cleanup datastore on state-change/channel failure
Let's not leave old state hanging around! Note that this fires
for pretty much every/any channel (even if we're not the opener).
2022-10-20 13:42:41 +02:00
niftynei
efd096dc96 funder: filter prev-outs such that we only use still unspent ones
If for some reason a utxo we used previously is no longer 'unspent',
we shouldn't use it for the next transaction.
2022-10-20 13:42:41 +02:00
niftynei
38e2428f12 funder: use utxopsbt to build psbt for RBFs
We use the saved previous outputs (plus maybe some new ones?) to build a
psbt for an RBF request.

RBFs utxo reuse is now working so we can unfail the test (and update
it to reflect that the lease sticks around through an RBF cycle).

Changelog-Fixed: Plugins: `funder` now honors lease requests across RBFs
2022-10-20 13:42:41 +02:00
niftynei
45acc20a8d funder: use previous outputs in count towards available funding
Still need to use them to build the PSBT for the rbf however.
2022-10-20 13:42:41 +02:00
niftynei
7733c2b0f4 funder: pull out previous input list from datastore on RBF
It'd be nice to know which utxos we used previously, so we can rebuild a
transaction using them!
2022-10-20 13:42:41 +02:00
niftynei
a5e9035e2e funder: save utxos of signed txs to datastore
We're going to need to know what utxos we used if we RBF this channel;
so we serialize our inputs and save them to the datastore under the channel_id.
2022-10-20 13:42:41 +02:00
niftynei
dc9e79c445 funder: rm quote that makes nifty cringe every time she sees it 2022-10-20 13:42:41 +02:00
niftynei
85039d4f4e df: pass lease data back to funder for rbfs
let's let RBFs know about our lease info!
2022-10-20 13:42:41 +02:00
niftynei
8d864d2299 funder: we always pass in channel_max, no need to special case it
Parse the channel_max along with everything else.
2022-10-20 13:42:41 +02:00
Rusty Russell
e855ac2f9e keysend: just strip even unknown fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `keysend` now removes unknown even (technically illegal!) fields, to try to accept more payments.
2022-10-04 17:48:08 +02:00
Rusty Russell
2fbe0f59f1 plugins/fetchinvoice: remove obsolete string-based API.
Generate the payload in the callers.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-29 16:10:57 +09:30
Rusty Russell
41ef85318d onionmessages: remove obsolete onion message parsing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-29 16:10:57 +09:30
Rusty Russell
8771c86379 common/onion: expunge all trace of different onion styles.
In particular, remove special routines to pull length: it's there,
take it and check it yourself.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-28 13:40:57 +02:00
Rusty Russell
68f15f17bb delforward: allow deletion of "unknown in_htlc_id" and fix autoclean to use it.
Note the caveats: we will delete *all* of them at once!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-27 14:42:03 +09:30
Christian Decker
e1fc88ff70 cln-plugin: Prep the logging payload in a let
I wanted to debug why we weren't getting some log messages, and this
makes that a bit easier in future.
2022-09-25 15:13:12 +02:00
Christian Decker
064a5a6940 cln-plugin: Add log filtering support
We filter based on the environment variable `CLN_PLUGIN_LOG`,
defaulting to `info` as that is not as noisy as `debug` or `trace`, at
least libraries will not spam us too heavily.

Changelog-Added cln-plugin: The logs level from cln-plugins can be configured by the `CLN_PLUGIN_LOG` environment variable.
2022-09-25 15:13:12 +02:00
Christian Decker
8898511cf6 cln-plugin: Defer binding the plugin state until after configuring
We had a bit of a chicken-and-egg problem, where we instantiated the
`state` to be managed by the `Plugin` during the very first step when
creating the `Builder`, but then the state might depend on the
configuration we only get later. This would force developers to add
placeholders in the form of `Option` into the state, when really
they'd never be none after configuring.

This defers the binding until after we get the configuration and
cleans up the semantics:

 - `Builder`: declare options, hooks, etc
 - `ConfiguredPlugin`: we have exchanged the handshake with
   `lightningd`, now we can construct the `state` accordingly
 - `Plugin`: Running instance of the plugin

Changelog-Changed: cln-plugin: Moved the state binding to the plugin until after the configuration step
2022-09-25 15:13:12 +02:00
Rusty Russell
bb49e1bea5 common: assume htlc_maximum_msat, don't check bit any more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell
253b25522b BOLT: update to version which requires option_channel_htlc_max.
We will now simply reject old-style ones as invalid.  Turns out the
only trace we could find is a channel between two nodes unconnected to
the rest of the network.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now require all channel_update messages include htlc_maximum_msat (as per latest BOLTs)
2022-09-24 15:22:27 +09:30
Rusty Russell
6338758018 gossmap: make API more robust against future changes.
Many changes to gossmap (including the pending ones!) don't actually
concern readers, as long as they obey certain rules:

1. Ignore unknown messages.
2. Treat all 16 upper bits of length as flags, ignore unknown ones.

So now we split the version byte into MAJOR and MINOR, and you can
ignore MINOR changes.

We don't expose the internal version (for creating the map)
programmatically: you should really hardcode what major version you
understand!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell
f52ff07558 lightningd: allow delpay to delete a specific payment.
This is actually what the autoclean plugin wants, especially since
you can't otherwise delete a payment which has failed then succeeded.

But insist on neither or both being specified, at least for now.

Changelog-Added: JSON-RPC: `delpay` takes optional `groupid` and `partid` parameters to specify exactly what payment to delete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
fa7d732ba6 lightningd: allow a connection to specify db batching.
Previous commit was a hack which *always* batched where possible, this
is a more sophisticated opt-in varaint, with a timeout sanity check.

Final performance for cleaning up 1M pays/forwards/invoices:

```
$ time l1-cli autoclean-once succeededpays 1
{
   "autoclean": {
      "succeededpays": {
         "cleaned": 1000000,
         "uncleaned": 26895
      }
   }
}

real	6m9.828s
user	0m0.003s
sys	0m0.001s
$ time l2-cli autoclean-once succeededforwards 1
{
   "autoclean": {
      "succeededforwards": {
         "cleaned": 1000000,
         "uncleaned": 40
      }
   }
}

real	3m20.789s
user	0m0.004s
sys	0m0.001s
$ time l3-cli autoclean-once paidinvoices 1
{
   "autoclean": {
      "paidinvoices": {
         "cleaned": 1000000,
         "uncleaned": 0
      }
   }
}

real	6m47.941s
user	0m0.001s
sys	0m0.000s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `batching` command to allow database transactions to cross multiple back-to-back JSON commands.
2022-09-22 15:19:46 +02:00
Rusty Russell
8b7a8265e7 libplugin: avoid memmove if we have many outputs to lightningd.
Use linked list, not an array.

```
+   97.89%     0.01%  autoclean  autoclean             [.] next_plan
-   97.08%     0.01%  autoclean  autoclean             [.] json_stream_output_write
   - 97.06% json_stream_output_write
      - 84.29% ld_stream_complete
         - 83.87% tal_arr_remove_
            + 83.71% __memcpy_avx_unaligned_erms (inlined)
      + 12.76% rpc_stream_complete
+   96.59%     0.03%  autoclean  autoclean             [.] tal_arr_remove_
+   96.48%     0.00%  autoclean  libc.so.6             [.] __memcpy_avx_unaligned_erms (inlined)
+   94.98%    94.98%  autoclean  libc.so.6             [.] __memmove_avx_unaligned_erms
+   84.29%     0.01%  autoclean  autoclean             [.] ld_stream_complete
+   12.76%     0.00%  autoclean  autoclean             [.] rpc_stream_complete
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
4d8c321517 libplugin: optimize parsing lightningd rpc responses.
autoclean was using 98% of its time in memmove; we should simply keep
an offset, and memmove when it's empty.  And also, only memmove the
used region, not the entire buffer!

Running on product of giantnodes.py:

	$ time l1-cli autoclean-once failedpays 1
	{
	   "autoclean": {
	      "failedpays": {
	         "cleaned": 26895,
	         "uncleaned": 1000000
	      }
	   }
	}

Before:
	real	20m46.579s
	user	0m0.000s
	sys	0m0.001s
	
After:
	real	2m10.568s
	user	0m0.001s
	sys	0m0.000s

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
540a6e4b99 autoclean: remove per-delete debugging messages.
They slow down benchmarking, which is kind of unfair!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
13e10877de autoclean: add autoclean-once command.
Changelog-Added: Plugins: `autoclean-once` command for a single cleanup.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
399288db3f autoclean: use config variables, not commands.
It's more natural: we will eventually support dynamic config variables,
so this will be quite nice.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `autoclean` can now delete old forwards, payments, and invoices automatically.
2022-09-22 15:19:46 +02:00
Rusty Russell
a15f1be5f8 autoclean: clean up listforwards as well.
And take the opportunity to rename l0 and l1 in the tests to the
more natural l1 l2 (since we add l3).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
4cab396cc8 autoclean: handle cleaning of old payments (not just invoices).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
660c9af1d9 autoclean: allow cleaning of paid invoices too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
7da51892e8 autoclean: save stats on how much we cleaned.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
17858c9490 lightningd: deprecated "delexpiredinvoice", put functionality in autoclean plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `delexpiredinvoice`: use `autoclean-once`.
2022-09-22 15:19:46 +02:00
Rusty Russell
bd76a196f5 autoclean: new interface
In preparation for more things being autocleaned.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
41a52929f7 libplugin: handle JSON reply after command freed.
This can happen, and in fact does below in our test_autoclean_once
test where we update the datastore, and return from the cmd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-22 15:19:46 +02:00
Rusty Russell
df4b477e88 keysend: allow extratlvs parameter, even in non-experimental mode.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `keysend` now has `extratlvs` option in non-EXPERIMENTAL builds.
2022-09-22 17:19:11 +09:30
Rusty Russell
ce0f544073 keysend: try to find description in TLV.
"Who needs specs?"  FFS...

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `keysend` will now attach the longest valid text field in the onion to the invoice (so you can have Sphinx.chat users spam you!)
2022-09-22 17:19:11 +09:30
antongalitch
41502be60a Fix a small typo 2022-09-22 10:00:35 +09:30
Christian Decker
759fcb64a8 pay: If the channel_hint matches our allocation allow it
It means we consume the channel completely to the best of our
knowledge, so let that through.

Changelog-Fixed: pay: Squeezed out the last `msat` from our local view of the network
2022-09-21 11:25:47 +02:00
Christian Decker
2def843dce pay: Allow using a channel on equality of estimated capacity
In the case of the local channel we set the estimation to the exact
value spendable, which is important when we want to drain a channel,
because there we actually want to get the last msat.
2022-09-21 11:25:47 +02:00
Christian Decker
5c1de8029a openingd: Add reserve to fundchannel and multifundchannel
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel` and `fundchannel_start` now accept a `reserve` parameter to indicate the absolute reserve to impose on the peer.
2022-09-21 11:25:47 +02:00
Rusty Russell
5b58eda748 libplugin: mark the cmd notleak() whenever command_still_pending() called.
This is what we do in lightningd, which makes memleak much more forgiving:
you can hang temporaries off cmd without getting reports of leaks (also
when send_outreq called).

We remove all the notleak() calls in plugins which worked around this!
And avoid multiple notleak labels, since both send_outreq() and
command_still_pending() can be called multiple times.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell
701dd3dcef memleak: remove exclusions from memleak_start()
Add memleak_ignore_children() so callers can do exclusions themselves.

Having two exclusions was always such a hack!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell
3380f559f9 memleak: simplify API.
Mainly renaming.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-19 11:34:42 +09:30
Rusty Russell
2da5244e83 jsonrpc: make error codes an enum.
This allows GDB to print values, but also allows us to use them in
'case' statements.  This wasn't allowed before because they're not
constant terms.

This also made it clear there's a clash between two error codes,
so move one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: Error code from bcli plugin changed from 400 to 500.
2022-09-19 10:18:55 +09:30