Commit Graph

516 Commits

Author SHA1 Message Date
Rusty Russell
e96eb07ef4 lightningd: test that hsm_secret is as expected, at startup.
If you get the wrong hsm_secret, your node_id will change, and
peers won't know who you are, bitcoind will reject your transaction
signatures, and other madness.

Catch this as soon as it happens, by storing our node_id in the db.

Suggested-by: @cdecker, @fiatjaf
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Config: `lightningd` will refuse to start with the wrong node_id (i.e. hsm_secret changes).
2022-07-20 19:28:33 +09:30
Rusty Russell
912ac25270 lightningd: remove 'connected' flag from channel structure.
It's directly a product of "does it have a current owner subdaemon"
and "does that subdaemon talk to peers", so create a helper function
which just evaluates that instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
adi2011
6ba8abb0de lightningd: Add RPC for populating DB with stub channels and set an error on reconnecting 2022-07-14 12:24:48 -05:00
adi2011
eca844eb36 channel: Add struct scb_chan in channel and making last tx optional. 2022-07-14 12:24:48 -05:00
Rusty Russell
3a1a7eb93f wallet: allow saving forwarding scid even if we don't have amount.
They're not logically connected: we can know where they wanted to
go, but we didn't send it.

Where possible, it's the scid *they asked for*; otherwise, it's the
scid or fallback to the alias, but do this in the *caller*, not by
overriding inside wallet_forwarded_payment_add.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-12 06:38:11 +09:30
Christian Decker
252ccfa7ab db: Store the local alias for forwarded incoming payments
Not only can the outgoing edge be a zeroconf channel, it can also be
the incoming channel. So we revert to the usual trick of using the
local alias if the short_channel_id isn't known yet.

We use the LOCAL alias instead of the REMOTE alias even though the
sender likely used the REMOTE alias to refer to the channel. This is
because we control the LOCAL alias, and we keep it stable during the
lifetime of the channel, whereas the REMOTE one could change or not be
there yet.
2022-07-04 22:14:06 +02:00
Christian Decker
2dc86bf29b db: Store the alias if that's all we got in a forward 2022-07-04 22:14:06 +02:00
Christian Decker
de1c0b51f0 zeroconf: Add alias_remote and alias_local to channel and DB
`alias_local` is generated locally and sent to the peer so it knows
what we're calling the channel, while `alias_remote` is received by
the peer so we know what to include in routehints when generating
invoices.
2022-07-04 22:14:06 +02:00
Vincenzo Palazzo
7ff62b4a00 lightnind: removeDEFAULT_PORT global definition
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-06-28 06:09:01 +09:30
Rusty Russell
37403e471c pytest: add (failing) test for db with old Tor v2 address.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-24 10:24:39 +09:30
Rusty Russell
d18817a525 wallet: don't get so upset on orphaned HTLC.
User grubman on IRC reported a crash due to new HTLC checks:

```
2022-04-05T01:15:17.707Z **BROKEN** wallet: Missing preimage for orphaned HTLC; replacing with zeros
...
2022-04-05T01:15:44.950Z **BROKEN** lightningd: check_already_failed:Both failed and succeeded?
2022-04-05T01:15:45.020Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v0.11.0rc1-modded)
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0xaaaab8f7f8bb
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: common/daemon.c:46 (crashdump) 0xaaaab8f7f91b
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffa4a315bf
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffa45c6cd8
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffa45b3a27
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/log.c:821 (fatal_vfmt) 0xaaaab8f3e1b7
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/log.c:829 (fatal) 0xaaaab8f3e24b
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/htlc_end.c:87 (corrupt) 0xaaaab8f325d3
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/htlc_end.c:175 (htlc_out_check) 0xaaaab8f32ee3
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1471 (check_already_failed) 0xaaaab8f53ea7
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1575 (onchain_failed_our_htlc) 0xaaaab8f54aab
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:411 (handle_missing_htlc_output) 0xaaaab8f4037b
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:544 (onchain_msg) 0xaaaab8f409bf
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/subd.c:556 (sd_msg_read) 0xaaaab8f5f8ef
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0xaaaab8fda197
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0xaaaab8fda76f
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0xaaaab8fda82f
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0xaaaab8fdc77f
2022-04-05T01:15:45.020Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0xaaaab8f37857
2022-04-05T01:15:45.021Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1181 (main) 0xaaaab8f3bcff
2022-04-05T01:15:45.021Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffa45b3d4f
2022-04-05T01:15:45.021Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xaaaab8f1ed33
```

Since no HTLCs are likely to still be pending since v0.6.1, we can remove that compat
code.  Indeed, this happens on my node: we've not loaded the incoming HTLC because
it's resolved.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-05 15:39:38 +09:30
Rusty Russell
e47786da04 pay/sendpay: also store description in case bolt11 uses description_hash.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-04 20:26:04 +09:30
Rusty Russell
7e789be0ea doc: update BOLTs to latest master.
Just typo fixes and the like.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell
2f7f7ec255 lightningd: add "style" to listforwards.
Suggested-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listforwards` has new entry `style`, currently "legacy" or "tlv".
2022-03-31 19:38:05 +10:30
Rusty Russell
aad4495f56 delinvoice: allow desconly arg to only remove the description.
Means that field is now optional in JSON output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `delinvoice` has a new parameter `desconly` to remove description.
2022-03-29 10:04:16 +10:30
Rusty Russell
999c734bb5 setchannel: add minhtlc
Suggested by @m-schmook, I realized that if we append it later I'll
never get it right: I expect parameters min and max, not max and min!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: you can now alter the `htlc_minimum_msat` and `htlc_maximum_msat` your node advertizes.
2022-03-22 18:45:41 +10:30
Rusty Russell
6fdcc86f9d lightningd: store htlc_maximum_msat for channel in the db.
We currently don't allow setting it, but it's been requested.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
niftynei
453ef7f0ce db: mark that column is unused/ignored
2022-01-25T23:41:45.2994844Z ----------------------------- Captured stderr call -----------------------------
2022-01-25T23:41:45.2995230Z lightningd: Never accessed column 1 in query SELECT 1  FROM offers WHERE offer_id = ?;
2022-03-05 15:03:34 +10:30
niftynei
ce12d2b8a9 database: pull out database code into a new module
We're going to reuse the database controllers for the accounting plugin
2022-03-05 15:03:34 +10:30
niftynei
03c950bae8 db: decouple fatal reliance, have as impl defined function
`fatal` is defined in lightningd and has logfile dependencies etc.

Make it more generic by allowing declaration in the use file (wallet.c)
2022-03-05 15:03:34 +10:30
niftynei
36ca175ec7 wallet: was erroring out, saving to null field
Add missing field to first write
2022-02-07 13:02:09 +10:30
niftynei
8225a9decf coin_mvt: log events for pushes/lease_fees for leased channels
We need to stash/save the amount of the lease fees on a leased channel,
we do this by re-using the 'push' amount field on channel (which is
technically correct, since we're essentially pushing the fee amount to
the peer).

Also updates a bit of how the pushes are accounted for (pushed to now
has an event; their channel will open at zero but then they'll
immediately register a push event).

Leases fees are treated exactly the same as pushes, except labeled
differently.

Required adding a 'lease_fee' field to the inflights so we keep track of
the fee for the lease until the open happens.
2021-12-28 04:42:42 +10:30
niftynei
bddd3694fa coin_mvt: record fees for an outbound htlc
If we initialized the payment, the fees are the entire fee-chain
(final hop amount - starting hop amount)

If it's a payment we routed, the fees are the diff between the
inbound htlc and the outbound (net gain by this routing)

Added to database so data persists nicely.
2021-12-28 04:42:42 +10:30
niftynei
d2c4d4aec2 coin_mvts: rewrite how onchain events are recorded, update tests
The old model of coin movements attempted to compute fees etc and log
amounts, not utxos. This is not as robust, as multi-party opens and dual
funded channels make it hard to account for fees etc correctly.

Instead, we move towards a 'utxo' view of the onchain events. Every
event is either the creation or 'destruction' of a utxo. For cases where
the value of the utxo is not (fully) debited/credited to our account, we
also record the output_value. E.g. channel closings spend a utxo who's
entire value we may not own.

Since we're now tracking UTXOs onchain, we can now do more complex
assertions about the onchain footprint of them. The integration tests
have been updated to now use more 'chain aware' assertions about the
ending state.
2021-12-28 04:42:42 +10:30
Ken Sedgwick
e8f43ef6ca wallet: make wallet_can_spend non-static because needed 2021-12-14 11:24:52 +10:30
Rusty Russell
484222b0a1 daemons: remove unused functions or make static.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell
4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
ZmnSCPxj jxPCSnmZ
3433ff5e15 wallet/db.c, wallet/wallet.c: Add a partial index to speed up startup.
Closes: #4901

Tested by `EXPLAIN QUERY PLAN` on sqlite3; #4901 shows the result from
@whitslack doing a similar partial index on PostgreSQL on his ~1000 chan
node.

ChangeLog-Added: db: Speed up loading of pending HTLCs during startup by using a partial index.
2021-12-02 11:54:57 +01:00
Simon Vrouwe
63bd569bf6 lightningd: cleanup, freeing jsonrpc in shutdown cannot trigger db write's anymore
since PR #3867 utxos are unreserved by height, destroy_utxos and
related functions are not used anymore so clean them up also

However free(ld->jsonrpc) still needs to happen before free(ld) because its
destructors need list_head pointers from ld
2021-11-30 13:34:44 +10:30
Rusty Russell
2e6df89274 wallet: remove redundant field counters, ignored fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
c462ccae1a wallet: have db track what columns are accessed in DEVELOPER mode.
And add db_col_ignore helper for cases where it's deliberate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
b8a240bd15 wallet: convert wallet.c to db_col_ APIs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
787fbb1228 db: create simple hashtable of fields in SELECT.
This simplistically maps names to numbers, eg:

	SELECT foo, bar FROM tbl;

'foo' -> 0
'bar' -> 1

If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
niftynei
42e40c1ced htlcs: add flag to 'fail immediately'
If we're over the dust limit, we fail it immediatey *after* commiting
it, but we need a way to signal this throughout the lifecycle, so we add
it to htlc_in struct and persist it through to the database.

If it's supposed to be failed, we fail after the commit cycle is
completed.
2021-10-23 12:59:13 +02:00
niftynei
1fe829c546 lightningd: new option for htlc dust limit
To reduce the surface area of amount of a channel balance that can be
eaten up as htlc dust, we introduce a new config
'--max-dust-htlc-exposure-msat', which sets the max amount that any
channel's balance can be added as dust

Changelog-Added: config: new option --max-dust-htlc-exposure-msat, which limits the total amount of sats to be allowed as dust on a channel
2021-10-23 12:59:13 +02:00
Rusty Russell
da03985996 wallet: only hand onchaind the HTLCs it needs to know.
This will make closing long-lived channels more efficient, and it's
just nicer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
c503232cde common: use bitcoin_outpoint.
I started pulling this thread, and the entire codebase got unravelled.

Oh well, it's done now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
433e61fcb9 wallet: remove unnecessary data from channel_htlcs when htlc is dead.
In particular, the onion and errors can be large, but now we'll never
need to retransmit them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
8a85bf6880 lightningd: populate min/max commit fields in db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
bdaec48400 wallet: wrap htlc_state enum in db function.
All enums in the db should be wrapped this way on reading/writing them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Christian Decker
5f260840ab jsonrpc: Add groupid to waitsendpay 2021-10-13 13:41:18 +10:30
Christian Decker
817b1b361e db: Add groupid to HTLCs 2021-10-13 13:41:18 +10:30
Christian Decker
0bb0d8a707 wallet: Add function to retrieve the latest groupid for a payment
When doing things like `waitsendpay` without specifying the `groupid`
we likely want to use the latest `groupid` we created, since that's
the one in flight. This adds a function to quickly retrieve that.
2021-10-13 13:41:18 +10:30
Christian Decker
ba4be0313b db: Add groupid to the payments table 2021-10-13 13:41:18 +10:30
Christian Decker
c963e6b03f db: Don't switch between text and blob fields
The database will yell at you if you try.
2021-10-10 11:05:01 +10:30
Christian Decker
b9c602c7e4 db: Fix a syntax error with the optional parameters
SQL doesn't really allow `a OR 1` as a clause since `1` is not a
boolean expression. Moving it into `a OR 1=1` however is valid again.
2021-10-10 11:05:01 +10:30
Vincenzo Palazzo
eb103c15df review 2/2: Use generic query to make the code more readable.
Suggested by @cdecker

P.S: Also this include an API refactoring from my previous solution, also this it is suggested by @cdecker.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo
ee8c7252a3 wallet db: Support the query on database with status and payment hash.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo
fd33aed4b5 rpc: Integrate the status flow in the listsendpays command
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Rusty Russell
7401b26824 cleanup: remove unneeded includes in C files.
Before:
 Ten builds, laptop -j5, no ccache:

```
real	0m36.686000-38.956000(38.608+/-0.65)s
user	2m32.864000-42.253000(40.7545+/-2.7)s
sys	0m16.618000-18.316000(17.8531+/-0.48)s
```

 Ten builds, laptop -j5, ccache (warm):

```
real	0m8.212000-8.577000(8.39989+/-0.13)s
user	0m12.731000-13.212000(12.9751+/-0.17)s
sys	0m3.697000-3.902000(3.83722+/-0.064)s
```

After:
 Ten builds, laptop -j5, no ccache: 8% faster

```
real	0m33.802000-35.773000(35.468+/-0.54)s
user	2m19.073000-27.754000(26.2542+/-2.3)s
sys	0m15.784000-17.173000(16.7165+/-0.37)s
```

 Ten builds, laptop -j5, ccache (warm): 1% faster

```
real	0m8.200000-8.485000(8.30138+/-0.097)s
user	0m12.485000-13.100000(12.7344+/-0.19)s
sys	0m3.702000-3.889000(3.78787+/-0.056)s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30