They're not logically connected: we can know where they wanted to
go, but we didn't send it.
Where possible, it's the scid *they asked for*; otherwise, it's the
scid or fallback to the alias, but do this in the *caller*, not by
overriding inside wallet_forwarded_payment_add.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not only can the outgoing edge be a zeroconf channel, it can also be
the incoming channel. So we revert to the usual trick of using the
local alias if the short_channel_id isn't known yet.
We use the LOCAL alias instead of the REMOTE alias even though the
sender likely used the REMOTE alias to refer to the channel. This is
because we control the LOCAL alias, and we keep it stable during the
lifetime of the channel, whereas the REMOTE one could change or not be
there yet.
`alias_local` is generated locally and sent to the peer so it knows
what we're calling the channel, while `alias_remote` is received by
the peer so we know what to include in routehints when generating
invoices.
We had json_add_amount_msat_only(), which was designed to be used to
print out msat fields, if we had sats.
However, we misused it, so split it into the three different cases:
1. json_add_amount_sat_msat: We are using it correctly, with a field called
xxx_msat.
2. json_add_amount_sats_deprecated: We were using it wrong, so deprecate
the old field and create a new one which does end in _msat.
3. json_add_sats: we were using it to hand sats as a JSON parameter to an
interface, where "XXXsat".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Plugins: `rbf_channel` and `openchannel2` hooks `their_funding` (use `their_funding_msat`)
Changelog-Deprecated: Plugins: `openchannel2` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `funding_satoshis` (use `funding_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `dust_limit_satoshis` (use `dust_limit_msat`)
Changelog-Deprecated: Plugins: `openchannel` hook `channel_reserve_satoshis` (use `channel_reserve_msat`)
Changelog-Deprecated: Plugins: `channel_opened` notification `amount` (use `funding_msat`)
Changelog-Deprecated: JSON-RPC: `listtransactions` `msat` (use `amount_msat`)
Changelog-Deprecated: Plugins: `htlc_accepted` `forward_amount` (use `forward_msat`)
Per BIP-0171, the signature map is of pubkey to "The signature as would
be pushed to the stack from a scriptSig or witness".
Fixes 5298
Changelog-Fixed: PSBT: Fix signature encoding to comply with BIP-0171.
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Make it always a number; this makes the JSON request specification
simpler. We allowed a number since v0.10.1.
(reserve=True is the default anyway, so usually it can be omitted:
reserve=False becomes reserve=0).
Changelog-Deprecated: JSON-RPC: `fundpsbt`/`utxopsbt` `reserve` must be a number, not bool (for `true` use 72/don't specify, for `false` use 0). Numbers have been allowed since v0.10.1.
Regenerate from current BOLTS via `make extract-bolt-csv`
1. The remote_addr field was added manually into peer_wire.csv: this
needs to be a patch otherwise it vanishes on regen.
2. We never brought into the channel_disabled fields, because it was
too much hassle (we never actually generate this!). Do it now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Suggested-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listforwards` has new entry `style`, currently "legacy" or "tlv".
Means that field is now optional in JSON output.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `delinvoice` has a new parameter `desconly` to remove description.
It was tlv_fields_valid that wanted a non-const: now that's gone, we
can make this correctly const.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
More efficient to search a known peer than the whole set.
Also, move find_channel_by_id() from channel_control.c into channel.c
where we'd expect it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sure, we want to connect (usually) because of an active channel, but
it's not specific to the channel itself.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than intuiting whether this is a new channel / active channel,
use the channel_id. This simplifies things and makes them explicit,
and prepares for multiple live channels per peer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we always have it (either extracted from an unsolicited message,
or told to us by lightningd when it tells us it wants to talk), we can
always send it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't need to hand it to channeld: it will read it! We simply
need to tell it to expect it.
Similarly, openingd/dualopend will never see it, so remove that logic.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Either because lightningd tells us it wants to talk, or because the peer
says something about a channel.
We also introduce a behavior change: we disconnect after a failed open.
We might want to modify this later, but we it's a side-effect of openingd
not holding onto idle connections.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The message from lightningd simply acknowleges that we are allowed to
discard the peer (because no subdaemons are talking to it anymore).
This difference becomes more stark once connectd holds on to idle
peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Suggested by @m-schmook, I realized that if we append it later I'll
never get it right: I expect parameters min and max, not max and min!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: you can now alter the `htlc_minimum_msat` and `htlc_maximum_msat` your node advertizes.
As per proposal in https://github.com/lightning/bolts/pull/962
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
This is the cheapest algo I came up with that simply checks that the
same `remote_addr` has been report by two different peers. Can be
improved in many ways:
- Check by connecting to a radonm peers in the network
- Check for more than two confirmations or a certain fraction
- ...
Changelog-Added: Send updated node_annoucement when two peers report the same remote_addr.
Instead of doing this weird chaining, just call them all at once and
use a reference counter.
To make it simpler, we return the subd_req so we can hang a destructor
off it which decrements after the request is complete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There is no "wallet_lib_headers" variable in wallet/Makefile
Likewise, there were two "lightningd_headers", a couple of unused
variables and some other nonsene in lightningd/Makefile
This is neater than what we had before, and slightly more general.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON_RPC: `sendcustommsg` now works with any connected peer, even when shutting down a channel.
We also no longer strip the type off: everyone handles both forms, and
Eclair doesn't strip (and it's easier!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With this change, we get more fine-grained error messages if something
goes wrong in the course of communicating with the SQLite database. To
pick some random examples, the error codes SQLITE_IOERR_NOMEM,
SQLITE_IOERR_CORRUPTFS or SQLITE_IOERR_FSYNC are way more specific
than just a plain SQLITE_IOERR, and the corresponding error messages
generated by sqlite3_errstr() will hence give a better hint to the
user (or also to the developers, if an error report is sent) what the
cause for a failure is.
Changelog-None
The blockheight is zero though, since these aren't included in a block
yet.
We also don't issue an 'external' deposit event if we can tell that the
address you're sending to actually belongs to our wallet (we'll issue a
deposit event when it gets included in a block)
It's weird to have connectd ask gossipd, when lightningd can just do it
and hand all the addresses together.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now let gossipd do it.
This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. tal_strndup(.., str, strlen(str)) == tal_strdup()
2. tal_strdup also takes(), so document that.
3. Avoid passing 'struct sha256' on the stack: use ptr.
4. Generally, structures shouldn't keep pointers to things they don't own.
In this case, mvt->node_id.
5. Make payment_hash a pointer, since NULL is more natural than an all-zero
hash.
And add NON_NULL_ARGS() to the functions; it's cumbersome, but make it
fairly clear what params are optional.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Freeing an unconfirmed channel already releases the subd, so don't
do that explicitly.
2. Use channel->owner to transfer ownership where possible, using
channel_set_owner() which handles all the cases.
This simplifies the code and makes it more readable, IMHO.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to stash/save the amount of the lease fees on a leased channel,
we do this by re-using the 'push' amount field on channel (which is
technically correct, since we're essentially pushing the fee amount to
the peer).
Also updates a bit of how the pushes are accounted for (pushed to now
has an event; their channel will open at zero but then they'll
immediately register a push event).
Leases fees are treated exactly the same as pushes, except labeled
differently.
Required adding a 'lease_fee' field to the inflights so we keep track of
the fee for the lease until the open happens.
If we initialized the payment, the fees are the entire fee-chain
(final hop amount - starting hop amount)
If it's a payment we routed, the fees are the diff between the
inbound htlc and the outbound (net gain by this routing)
Added to database so data persists nicely.
we used this originally to suppress duplicate issuance of coin-move
events; we're assuming that any plugin expects duplicate events though
(and knows how to de-dupe them), so we no longer need this logic.
The old model of coin movements attempted to compute fees etc and log
amounts, not utxos. This is not as robust, as multi-party opens and dual
funded channels make it hard to account for fees etc correctly.
Instead, we move towards a 'utxo' view of the onchain events. Every
event is either the creation or 'destruction' of a utxo. For cases where
the value of the utxo is not (fully) debited/credited to our account, we
also record the output_value. E.g. channel closings spend a utxo who's
entire value we may not own.
Since we're now tracking UTXOs onchain, we can now do more complex
assertions about the onchain footprint of them. The integration tests
have been updated to now use more 'chain aware' assertions about the
ending state.
The only thing that needs ld->wallet after this is destroy_invoices_waiter (off jsonrpc)
Could not find any other destructors (destroy_*) that need wallet or db access after this.
Any db access would now segfault.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).
config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Various unit tests were creating temporary files unconditionally in /tmp
and were not cleaning up after themselves. Introduce a new variant of
mkstemp(3p) that respects the TMPDIR environment variable, and use it in
the offending unit tests. This allows each test run to use a dedicated
TMPDIR that can be cleaned up after the run.
Changelog-None
Signed-off-by: Matt Whitlock <c-lightning@mattwhitlock.name>
Closes: #4901
Tested by `EXPLAIN QUERY PLAN` on sqlite3; #4901 shows the result from
@whitslack doing a similar partial index on PostgreSQL on his ~1000 chan
node.
ChangeLog-Added: db: Speed up loading of pending HTLCs during startup by using a partial index.
since PR #3867 utxos are unreserved by height, destroy_utxos and
related functions are not used anymore so clean them up also
However free(ld->jsonrpc) still needs to happen before free(ld) because its
destructors need list_head pointers from ld
because:
- shutdown_subdaemons can trigger db write, comments in that function say so at least
- resurrecting the main event loop with subdaemons still running is counter productive
in shutting down activity (such as htlc's, hook_calls etc.)
- custom behavior injected by plugins via hooks should be consistent, see test
in previous commmit
IDEA:
in shutdown_plugins, when starting new io_loop:
- A plugin that is still running can return a jsonrpc_request response, this triggers
response_cb, which cannot be handled because subdaemons are gone -> so any response_cb should be blocked/aborted
- jsonrpc is still there, so users (such as plugins) can make new jsonrpc_request's which
cannot be handled because subdaemons are gone -> so new rpc_request should also be blocked
- But we do want to send/receive notifications and log messages (handled in jsonrpc as jsonrpc_notification)
as these do not trigger subdaemon calls or db_write's
Log messages and notifications do not have "id" field, where jsonrpc_request *do* have an "id" field
PLAN (hypothesis):
- hack into plugin_read_json_one OR plugin_response_handle to filter-out json with
an "id" field, this should
block/abandon any jsonrpc_request responses (and new jsonrpc_requests for plugins?)
Q. Can internal (so not via plugin) jsonrpc_requests called in the main io_loop return/revive in
the shutdown io_loop?
A. No. All code under lightningd/ returning command_still_pending depends on either a subdaemon, timer or
plugin. In shutdown loop the subdaemons are dead, timer struct cleared and plugins will be taken
care of (in next commits).
fixup: we can only io_break the main io_loop once
Because db->conn is a void *, changing it (from a direct pointer to
a pointer to a pair of pointers) did not break compile if one place hadn't
been update.
The result was a confusing failure: sqlite3 complaining about API misuse,
since the db->conn pointer was not a valid db handle any more.
This is one case where avoiding a void * is hard: we might not even
have the postgresql types, since it might not be installed. But a union
would have been superior here.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ChangeLog-Added: With the `sqlite3://` scheme for `--wallet` option, you can now specify a second file path for real-time database backup by separating it from the main file path with a `:` character.
1. db_col_text becomes db_col_strdup, which is what is usually wanted.
2. db_col_short_channel_id becomes db_col_short_channel_id_str, to emphasize
that it stores in string form. Modern versions should store u64.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This simplistically maps names to numbers, eg:
SELECT foo, bar FROM tbl;
'foo' -> 0
'bar' -> 1
If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we're over the dust limit, we fail it immediatey *after* commiting
it, but we need a way to signal this throughout the lifecycle, so we add
it to htlc_in struct and persist it through to the database.
If it's supposed to be failed, we fail after the commit cycle is
completed.
To reduce the surface area of amount of a channel balance that can be
eaten up as htlc dust, we introduce a new config
'--max-dust-htlc-exposure-msat', which sets the max amount that any
channel's balance can be added as dust
Changelog-Added: config: new option --max-dust-htlc-exposure-msat, which limits the total amount of sats to be allowed as dust on a channel
Closes: #4860
ChangeLog-Added: With `sqlite3` db backend we now use a 60-second busy timer, to allow backup processes like `litestream` to operate safely.
```
[gw1] [ 98%] PASSED tests/test_wallet.py::test_hsmtool_dump_descriptors
tests/test_wallet.py::test_fundchannel_listtransaction
[gw0] [ 98%] PASSED tests/test_plugin.py::test_channel_opened_notification
tests/test_wallet.py::test_hsmtool_generatehsm
[gw0] [ 98%] PASSED tests/test_wallet.py::test_hsmtool_generatehsm
tests/test_wallet.py::test_withdraw_nlocktime_fuzz
[gw1] [ 98%] ERROR tests/test_wallet.py::test_fundchannel_listtransaction
tests/test_wallet.py::test_fundchannel_listtransaction
tests/test_wallet.py::test_withdraw_nlocktime_fuzz
tests/test_wallet.py::test_fundchannel_listtransaction
[gw0] [ 99%] ERROR tests/test_wallet.py::test_withdraw_nlocktime_fuzz
tests/test_wallet.py::test_multiwithdraw_simple
[gw1] [ 99%] ERROR tests/test_wallet.py::test_fundchannel_listtransaction
tests/test_wallet.py::test_withdraw_nlocktime
tests/test_wallet.py::test_multiwithdraw_simple
tests/test_wallet.py::test_withdraw_nlocktime
tests/test_wallet.py::test_multiwithdraw_simple
tests/test_wallet.py::test_withdraw_nlocktime
[gw0] [ 99%] ERROR tests/test_wallet.py::test_multiwithdraw_simple
tests/test_wallet.py::test_repro_4258
[gw1] [ 99%] ERROR tests/test_wallet.py::test_withdraw_nlocktime
...
2021-10-12 06:36:09.203 UTC [224552] STATEMENT: SELECT version FROM version LIMIT 1
2021-10-12 06:36:09.566 UTC [224523] PANIC: could not write to file "pg_wal/xlogtemp.224523": No space left on device
2021-10-12 06:36:09.566 UTC [224523] STATEMENT: VACUUM FULL;
Error vacuuming db: BEGIN command failed: PANIC: could not write to file "pg_wal/xlogtemp.224523": No space left on device
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
```
This is particularly useful after our recent field deletion:
before: 362,573,824 bytes
after: 124,190,720 bytes
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: db: removal of old HTLC information and vacuuming shrinks large lightningd.sqlite3 by a factor of 2-3.
