"sphinx_add_hop" takes a literal hop to include,
"sphinx_add_modern_hop" prepends the length. Now we always prepend a
length, make it clear that the literal version is a shortcut:
* sphinx_add_hop -> sphinx_add_hop_has_length
* sphinx_add_modern_hop -> sphinx_add_hop
In addition, we check that length is actually correct! This means
`createonion` can no longer create legacy or otherwise-invalid onions:
fix tests and update man page to remove legacy usage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `createonion` no longer allows non-TLV-style payloads.
We still have an "enum forward_style" for the database, where old-style
forwards can still exist.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer forward HTLCs with legacy onions.
We have to allow them (as otherwise `fees_collected_msat` in getinfo breaks),
but it means that actually, in_htlc_id might be missing in listforwards
(also, out_htlc_id might be missing, which we didn't catch before).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5628
Otherwise what the hook sees is actually a lie, and if it sets it
we might override it.
The side effect is that we add an explicit "forward_to" field, and
allow hooks to override it. This lets a *hook* control channel
choice explicitly.
Changelod-Added: Plugins: `htlc_accepted_hook` return can specify what channel to forward htlc to.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a bit more optimal, and tells gossipd exactly what height the
spend occurred at (with multiple blocks, it's not always the current
height!). It will need that next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: We now set the `dont_forward` bit on private channel_update's message_flags (as per latest BOLTs).
This is actually what the autoclean plugin wants, especially since
you can't otherwise delete a payment which has failed then succeeded.
But insist on neither or both being specified, at least for now.
Changelog-Added: JSON-RPC: `delpay` takes optional `groupid` and `partid` parameters to specify exactly what payment to delete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Previous commit was a hack which *always* batched where possible, this
is a more sophisticated opt-in varaint, with a timeout sanity check.
Final performance for cleaning up 1M pays/forwards/invoices:
```
$ time l1-cli autoclean-once succeededpays 1
{
"autoclean": {
"succeededpays": {
"cleaned": 1000000,
"uncleaned": 26895
}
}
}
real 6m9.828s
user 0m0.003s
sys 0m0.001s
$ time l2-cli autoclean-once succeededforwards 1
{
"autoclean": {
"succeededforwards": {
"cleaned": 1000000,
"uncleaned": 40
}
}
}
real 3m20.789s
user 0m0.004s
sys 0m0.001s
$ time l3-cli autoclean-once paidinvoices 1
{
"autoclean": {
"paidinvoices": {
"cleaned": 1000000,
"uncleaned": 0
}
}
}
real 6m47.941s
user 0m0.001s
sys 0m0.000s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `batching` command to allow database transactions to cross multiple back-to-back JSON commands.
This shows the benefits of batching when being slammed by autoclean:
Before:
```
plugin-autoclean: last 1000 deletes took 6699022 nsec each
plugin-autoclean: last 1000 deletes took 6734025 nsec each
plugin-autoclean: last 1000 deletes took 6681189 nsec each
plugin-autoclean: last 1000 deletes took 6597148 nsec each
plugin-autoclean: last 1000 deletes took 6637085 nsec each
plugin-autoclean: last 1000 deletes took 6801425 nsec each
plugin-autoclean: last 1000 deletes took 6788572 nsec each
plugin-autoclean: last 1000 deletes took 6603641 nsec each
plugin-autoclean: last 1000 deletes took 6642947 nsec each
plugin-autoclean: last 1000 deletes took 6590495 nsec each
plugin-autoclean: last 1000 deletes took 6695076 nsec each
plugin-autoclean: last 1000 deletes took 6477981 nsec each
```
After:
```
plugin-autoclean: last 1000 deletes took 342764 nsec each
plugin-autoclean: last 1000 deletes took 375031 nsec each
plugin-autoclean: last 1000 deletes took 357564 nsec each
plugin-autoclean: last 1000 deletes took 381581 nsec each
plugin-autoclean: last 1000 deletes took 337989 nsec each
plugin-autoclean: last 1000 deletes took 329391 nsec each
plugin-autoclean: last 1000 deletes took 328322 nsec each
plugin-autoclean: last 1000 deletes took 372810 nsec each
plugin-autoclean: last 1000 deletes took 351228 nsec each
plugin-autoclean: last 1000 deletes took 413885 nsec each
plugin-autoclean: last 1000 deletes took 348317 nsec each
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using `listfowards` for this wrong; expose this directly if people
care (and unlike listforwards, which could be deleted, we have to
remember these while the channel is still open!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listhtlcs` new command to list all known HTLCS.
And document that we never know payment_hash.
Changelog-Added: JSON-RPC: `listforwards` now shows `in_htlc_id` and `out_htlc_id`
Changelog-Changed: JSON-RPC: `listforwards` now never shows `payment_hash`; use `listhtlcs`.
The old `experimental-accept-extra-tlv-types` is now `accept-htlc-tlv-types`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `accept-htlc-tlv-types` lets us accept unknown even HTLC TLV fields we would normally reject on parsing (was EXPERIMENTAL-only `experimental-accept-extra-tlv-types`).
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel` and `fundchannel_start` now accept a `reserve` parameter to indicate the absolute reserve to impose on the peer.
Add memleak_ignore_children() so callers can do exclusions themselves.
Having two exclusions was always such a hack!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows GDB to print values, but also allows us to use them in
'case' statements. This wasn't allowed before because they're not
constant terms.
This also made it clear there's a clash between two error codes,
so move one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: Error code from bcli plugin changed from 400 to 500.
First, merge the _ahf_ and non-ahf interfaces.
Second, remove the always-NULL txs->cmd field.
Then, add optional id_prefix for bitcoind_sendrawx, so if it's
triggered by a command (e.g. "withdraw") it's shown correctly in logs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Usually the calls are spontanous, so it's just "cln:<method>#NNN", but
json_invoice() calls listincoming, and json_checkmessage calls
listnodes, so those become "cli:invoice-<pid>/cln:listincoming#NNN".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is cleaner because, the `remote_addr` and `discovered_ip` are
related but two different things.
Within connectd and lightningd we use the peers `remote_addr` feature
to validate (and guess a port) to be used for IP discovery.
Also when a peer reports us a `remote_addr`, this is given to the plugin API
via the `peer_connected` hook. The network port here is not modified for
godd reason! This can be used i.e. to detect if we are behind a NAT.
But once lightningd figures enough peers report the same `remote_addr`,
it sets the port to the selected network and tells gossipd to use that for
`node_announcement` updates.
Hence, within gossipd, there is no (should not be) `remote_addr`.
Changelog-None
This is a good sanity check that users understand that if they upgrade
to master mid-cycle they can't go back!
Suggested-by: @wtogami
Changelog-Added: Config: `--database-upgrade=true` required if a non-release version wants to (irrevocably!) upgrade the db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can (and probably should!) allocate this off tmpctx rather than
off the response.
Also, json_add_invoice doesn't actually match the normal pattern,
so fix that.
```
E ValueError:
E Node errors:
E Global errors:
E - Node /tmp/ltests-_vbj8az8/test_pay_fail_unconfirmed_channel_1/lightning-2/ has memory leaks: [
E {
E "backtrace": [
E "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E "wallet/invoices.c:81 (wallet_stmt2invoice_details)",
E "wallet/invoices.c:697 (invoices_get_details)",
E "wallet/wallet.c:2946 (wallet_invoice_details)",
E "lightningd/invoice.c:1296 (json_add_invoices)",
E "lightningd/invoice.c:1370 (json_listinvoices)",
E "lightningd/jsonrpc.c:625 (command_exec)",
E "lightningd/jsonrpc.c:753 (rpc_command_hook_final)",
E "lightningd/plugin_hook.c:280 (plugin_hook_call_)",
E "lightningd/jsonrpc.c:841 (plugin_hook_call_rpc_command)",
E "lightningd/jsonrpc.c:938 (parse_request)",
E "lightningd/jsonrpc.c:1035 (read_json)",
E "ccan/ccan/io/io.c:59 (next_plan)",
E "ccan/ccan/io/io.c:407 (do_plan)",
E "ccan/ccan/io/io.c:417 (io_ready)",
E "ccan/ccan/io/poll.c:453 (io_loop)",
E "lightningd/io_loop_with_timers.c:22 (io_loop_with_timers)",
E "lightningd/lightningd.c:1194 (main)",
E "../csu/libc-start.c:308 (__libc_start_main)"
E ],
E "label": "wallet/invoices.c:81:struct invoice_details",
E "parents": [
E "common/json_stream.c:41:struct json_stream",
E "ccan/ccan/io/io.c:91:struct io_conn **NOTLEAK**",
E "lightningd/lightningd.c:107:struct lightningd"
E ],
E "value": "0x55a57a77de08"
E }
E ]
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were double counting channel lease fees because we were double firing
the channel open event sequence (so to speak). If we don't report
balances for unopened channels, we don't have this problem?
Changelog-Changed: Plugins: `balance_snapshot` notification does not send balances for channels that aren't locked-in/opened yet
Be more graceful in shutting down: this should fix the issue where
bookkeeper gets upset that its commands are rejected during shutdown,
and generally make things more graceful.
1. Stop any new RPC connections.
2. Stop any per-peer daemons (channeld, etc).
3. Shut down plugins.
4. Stop all existing RPC connections.
5. Stop global daemons.
6. Free up peer, chanen HTLC datastructures.
7. Close database.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: RPC operations are now still available during shutdown.
This includes the recommendation that we use 10 minute grace period,
so add quotes to where we use that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This alters the billboard, but that's a human-readable thing so not
noted in CHANGELOG.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `listpeers` `status` now refers to "channel ready" rather than "funding locked" (BOLT language change for zeroconf channels)
Changelog-Added: JSON-RPC: `channel_opened` notification `channel_ready` flag.
Changelog-Deprecated: JSON-RPC: `channel_opened` notification `funding_locked` flag (use `channel_ready`: BOLTs namechange).
This contains the zeroconf stuff, with funding_locked renamed to
channel_ready. I change that everywhere, and try to fix up the
comments.
Also the `alias` field is called `short_channel_id`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `funding_locked` is now called `channel_ready` as per latest BOLTs.
