Commit Graph

1214 Commits

Author SHA1 Message Date
niftynei
ea36c3a938 db: collapse the db_queries files into the generated ones
There's no reason to have these be independent of the generated files,
all the data contained within them is 'formulaic'
2022-03-05 15:03:34 +10:30
niftynei
453ef7f0ce db: mark that column is unused/ignored
2022-01-25T23:41:45.2994844Z ----------------------------- Captured stderr call -----------------------------
2022-01-25T23:41:45.2995230Z lightningd: Never accessed column 1 in query SELECT 1  FROM offers WHERE offer_id = ?;
2022-03-05 15:03:34 +10:30
niftynei
ce12d2b8a9 database: pull out database code into a new module
We're going to reuse the database controllers for the accounting plugin
2022-03-05 15:03:34 +10:30
niftynei
03c950bae8 db: decouple fatal reliance, have as impl defined function
`fatal` is defined in lightningd and has logfile dependencies etc.

Make it more generic by allowing declaration in the use file (wallet.c)
2022-03-05 15:03:34 +10:30
niftynei
b0829fc52a lightningd/Make: cleanup lightningd+wallet headers
There is no "wallet_lib_headers" variable in wallet/Makefile

Likewise, there were two "lightningd_headers", a couple of unused
variables and some other nonsene in lightningd/Makefile
2022-03-05 15:03:34 +10:30
Michael Schmoock
38e2abf68a peer_exchange: set, read and log remote_addr
Changelog-Added: Protocol: set remote_addr on init tlvs
2022-02-22 05:45:47 +10:30
Rusty Russell
727b486d49 connectd: don't received useless peer fd if we're told to send final msg.
We don't need the connection to ourselves, just to free it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell
3c5d27e3e9 subdaemons: remove gossipd fd from per-peer daemons.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Rusty Russell
1c71c9849b connectd: handle custom messages.
This is neater than what we had before, and slightly more general.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON_RPC: `sendcustommsg` now works with any connected peer, even when shutting down a channel.
2022-02-08 11:15:52 +10:30
Rusty Russell
5065bd6fc2 lightningd: use our cached channel_update for errors instead of asking gossipd.
We also no longer strip the type off: everyone handles both forms, and
Eclair doesn't strip (and it's easier!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-02-08 11:15:52 +10:30
Sebastian Falbesoner
8309a049eb db: enable SQLite extended result codes
With this change, we get more fine-grained error messages if something
goes wrong in the course of communicating with the SQLite database. To
pick some random examples, the error codes SQLITE_IOERR_NOMEM,
SQLITE_IOERR_CORRUPTFS or SQLITE_IOERR_FSYNC are way more specific
than just a plain SQLITE_IOERR, and the corresponding error messages
generated by sqlite3_errstr() will hence give a better hint to the
user (or also to the developers, if an error report is sent) what the
cause for a failure is.

Changelog-None
2022-02-07 13:59:10 +10:30
niftynei
36ca175ec7 wallet: was erroring out, saving to null field
Add missing field to first write
2022-02-07 13:02:09 +10:30
niftynei
4dafeede5c coin moves: notify when we make deposits to external accounts
The blockheight is zero though, since these aren't included in a block
yet.

We also don't issue an 'external' deposit event if we can tell that the
address you're sending to actually belongs to our wallet (we'll issue a
deposit event when it gets included in a block)
2022-01-26 13:34:45 +10:30
Rusty Russell
39c93ee6e5 connectd: get addresses from lightningd, not gossipd.
It's weird to have connectd ask gossipd, when lightningd can just do it
and hand all the addresses together.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell
741f44725a patch lightningd-peer-fds.patch 2022-01-20 15:24:06 +10:30
Rusty Russell
6115ed02e8 subdaemons: don't stream gossip_store at all.
We now let gossipd do it.

This also means there's nothing left in 'struct per_peer_state' to
send across the wire (the fds are sent separately), so that gets
removed from wire messages too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell
425a7af512 common/coin_mvt: clean up API a little.
1. tal_strndup(.., str, strlen(str)) == tal_strdup()
2. tal_strdup also takes(), so document that.
3. Avoid passing 'struct sha256' on the stack: use ptr.
4. Generally, structures shouldn't keep pointers to things they don't own.
   In this case, mvt->node_id.
5. Make payment_hash a pointer, since NULL is more natural than an all-zero
   hash.

And add NON_NULL_ARGS() to the functions; it's cumbersome, but make it
fairly clear what params are optional.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 14:36:55 +10:30
Rusty Russell
90b669857e lightningd: handle channel cleanups more explicitly.
1. Freeing an unconfirmed channel already releases the subd, so don't
   do that explicitly.
2. Use channel->owner to transfer ownership where possible, using
   channel_set_owner() which handles all the cases.

This simplifies the code and makes it more readable, IMHO.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-30 09:50:40 +10:30
niftynei
8225a9decf coin_mvt: log events for pushes/lease_fees for leased channels
We need to stash/save the amount of the lease fees on a leased channel,
we do this by re-using the 'push' amount field on channel (which is
technically correct, since we're essentially pushing the fee amount to
the peer).

Also updates a bit of how the pushes are accounted for (pushed to now
has an event; their channel will open at zero but then they'll
immediately register a push event).

Leases fees are treated exactly the same as pushes, except labeled
differently.

Required adding a 'lease_fee' field to the inflights so we keep track of
the fee for the lease until the open happens.
2021-12-28 04:42:42 +10:30
niftynei
bddd3694fa coin_mvt: record fees for an outbound htlc
If we initialized the payment, the fees are the entire fee-chain
(final hop amount - starting hop amount)

If it's a payment we routed, the fees are the diff between the
inbound htlc and the outbound (net gain by this routing)

Added to database so data persists nicely.
2021-12-28 04:42:42 +10:30
niftynei
8098a4cd21 onchaind: remove 'is_replay' logic
we used this originally to suppress duplicate issuance of coin-move
events; we're assuming that any plugin expects duplicate events though
(and knows how to de-dupe them), so we no longer need this logic.
2021-12-28 04:42:42 +10:30
niftynei
d2c4d4aec2 coin_mvts: rewrite how onchain events are recorded, update tests
The old model of coin movements attempted to compute fees etc and log
amounts, not utxos. This is not as robust, as multi-party opens and dual
funded channels make it hard to account for fees etc correctly.

Instead, we move towards a 'utxo' view of the onchain events. Every
event is either the creation or 'destruction' of a utxo. For cases where
the value of the utxo is not (fully) debited/credited to our account, we
also record the output_value. E.g. channel closings spend a utxo who's
entire value we may not own.

Since we're now tracking UTXOs onchain, we can now do more complex
assertions about the onchain footprint of them. The integration tests
have been updated to now use more 'chain aware' assertions about the
ending state.
2021-12-28 04:42:42 +10:30
Ken Sedgwick
bb574be839 hsmd: Add hsmd_new_channel 2021-12-14 11:24:52 +10:30
Ken Sedgwick
e8f43ef6ca wallet: make wallet_can_spend non-static because needed 2021-12-14 11:24:52 +10:30
ZmnSCPxj jxPCSnmZ
d088288daa bitcoin/chainparams.h: Split BIP173 name into onchain and Lightning HRPs.
Fixes: #4937
2021-12-14 11:21:35 +10:30
Simon Vrouwe
f936fa926f plugins: simplify shutdown loop, simply close the db
The only thing that needs ld->wallet after this is destroy_invoices_waiter (off jsonrpc)
Could not find any other destructors (destroy_*) that need wallet or db access after this.
Any db access would now segfault.
2021-12-14 09:33:10 +10:30
Andrew Toth
78fb78478b make: remove generated files when running clean 2021-12-06 10:06:25 +10:30
Rusty Russell
484222b0a1 daemons: remove unused functions or make static.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell
4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Matt Whitlock
1f79aad830 common/utils: introduce tmpdir_mkstemp; use it
Various unit tests were creating temporary files unconditionally in /tmp
and were not cleaning up after themselves. Introduce a new variant of
mkstemp(3p) that respects the TMPDIR environment variable, and use it in
the offending unit tests. This allows each test run to use a dedicated
TMPDIR that can be cleaned up after the run.

Changelog-None

Signed-off-by: Matt Whitlock <c-lightning@mattwhitlock.name>
2021-12-06 09:20:27 +10:30
ZmnSCPxj jxPCSnmZ
3433ff5e15 wallet/db.c, wallet/wallet.c: Add a partial index to speed up startup.
Closes: #4901

Tested by `EXPLAIN QUERY PLAN` on sqlite3; #4901 shows the result from
@whitslack doing a similar partial index on PostgreSQL on his ~1000 chan
node.

ChangeLog-Added: db: Speed up loading of pending HTLCs during startup by using a partial index.
2021-12-02 11:54:57 +01:00
Simon Vrouwe
63bd569bf6 lightningd: cleanup, freeing jsonrpc in shutdown cannot trigger db write's anymore
since PR #3867 utxos are unreserved by height, destroy_utxos and
related functions are not used anymore so clean them up also

However free(ld->jsonrpc) still needs to happen before free(ld) because its
destructors need list_head pointers from ld
2021-11-30 13:34:44 +10:30
Simon Vrouwe
5f69674faa lightningd: shutdown plugins after subdaemons and assert no write access to db
because:
    - shutdown_subdaemons can trigger db write, comments in that function say so at least
    - resurrecting the main event loop with subdaemons still running is counter productive
      in shutting down activity (such as htlc's, hook_calls etc.)
    - custom behavior injected by plugins via hooks should be consistent, see test
      in previous commmit

    IDEA:

    in shutdown_plugins, when starting new io_loop:

    - A plugin that is still running can return a jsonrpc_request response, this triggers
      response_cb, which cannot be handled because subdaemons are gone -> so any response_cb should be blocked/aborted

    - jsonrpc is still there, so users (such as plugins) can make new jsonrpc_request's which
      cannot be handled because subdaemons are gone -> so new rpc_request should also be blocked

    - But we do want to send/receive notifications and log messages (handled in jsonrpc as jsonrpc_notification)
      as these do not trigger subdaemon calls or db_write's
      Log messages and notifications do not have "id" field, where jsonrpc_request *do* have an "id" field

    PLAN (hypothesis):
    - hack into plugin_read_json_one OR plugin_response_handle to filter-out json with
      an "id" field, this should
      block/abandon any jsonrpc_request responses (and new jsonrpc_requests for plugins?)

  Q. Can internal (so not via plugin) jsonrpc_requests called in the main io_loop return/revive in
     the shutdown io_loop?
  A. No. All code under lightningd/ returning command_still_pending depends on either a subdaemon, timer or
     plugin. In shutdown loop the subdaemons are dead, timer struct cleared and plugins will be taken
     care of (in next commits).

 fixup: we can only io_break the main io_loop once
2021-11-30 13:34:44 +10:30
Rusty Russell
6c9b752751 memleak: make notleak use the tal name instead of a boutique struct.
This lets us mark it directly.

Get rid of long-unused "notleaks" member of struct lightningd too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-27 10:38:13 +10:30
Rusty Russell
efeb1bc65b wallet: fix sqlite3 column renaming since db backup merge.
Because db->conn is a void *, changing it (from a direct pointer to
a pointer to a pair of pointers) did not break compile if one place hadn't
been update.

The result was a confusing failure: sqlite3 complaining about API misuse,
since the db->conn pointer was not a valid db handle any more.

This is one case where avoiding a void * is hard: we might not even
have the postgresql types, since it might not be installed.  But a union
would have been superior here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-18 12:53:17 +10:30
ZmnSCPxj jxPCSnmZ
a294683675 wallet/db_sqlite3.c: Support direct replication of SQLITE3 backends.
ChangeLog-Added: With the `sqlite3://` scheme for `--wallet` option, you can now specify a second file path for real-time database backup by separating it from the main file path with a `:` character.
2021-11-17 12:10:07 +10:30
Rusty Russell
6c34e522dd wallet: db column manipulation helpers.
Removing columns and renaming them is easy in Postgres, hard in
sqlite3.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
fcf3d0ce6c db: turn generated queries array into a simple hash table.
Since we have that functionality, let's use it.  Also, make table
const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
53c9d9853d wallet: remove db_column_ functions.
Keep some as internal helpers only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
2e6df89274 wallet: remove redundant field counters, ignored fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
c462ccae1a wallet: have db track what columns are accessed in DEVELOPER mode.
And add db_col_ignore helper for cases where it's deliberate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
663c8c6c02 wallet: convert db internal routines to db_col_
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
b8a240bd15 wallet: convert wallet.c to db_col_ APIs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
c2a4285a7c wallet: use db_col_ accessors in wallet/invoices.c
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
befab73070 db: improve db_col_* APIs.
1. db_col_text becomes db_col_strdup, which is what is usually wanted.
2. db_col_short_channel_id becomes db_col_short_channel_id_str, to emphasize
   that it stores in string form.  Modern versions should store u64.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
5b482eb04b db: db_col_ variants for accessing SELECT statements by name.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
787fbb1228 db: create simple hashtable of fields in SELECT.
This simplistically maps names to numbers, eg:

	SELECT foo, bar FROM tbl;

'foo' -> 0
'bar' -> 1

If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
57328fe59e tests: use common_setup/common_shutdown to avoid leaks.
It also does more checks (like taken() checks).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Christian Decker
344579b233 wallet: Re-sync the payments.id sequence with postgresql
Fixes #4883
Fixes #4879
Changelog-None: Unreleased bug being fixed.
2021-10-31 13:11:34 +01:00
ZmnSCPxj jxPCSnmZ
d330e6e5b4 wallet/db_postgres.c: Fix obvious typo in error message.
ChangeLog-None
2021-10-28 15:51:25 +02:00
niftynei
42e40c1ced htlcs: add flag to 'fail immediately'
If we're over the dust limit, we fail it immediatey *after* commiting
it, but we need a way to signal this throughout the lifecycle, so we add
it to htlc_in struct and persist it through to the database.

If it's supposed to be failed, we fail after the commit cycle is
completed.
2021-10-23 12:59:13 +02:00
niftynei
1fe829c546 lightningd: new option for htlc dust limit
To reduce the surface area of amount of a channel balance that can be
eaten up as htlc dust, we introduce a new config
'--max-dust-htlc-exposure-msat', which sets the max amount that any
channel's balance can be added as dust

Changelog-Added: config: new option --max-dust-htlc-exposure-msat, which limits the total amount of sats to be allowed as dust on a channel
2021-10-23 12:59:13 +02:00
ZmnSCPxj jxPCSnmZ
ab026fb4de wallet/db_sqlite3.c: Increase busy timeout to 60 seconds.
Closes: #4860

ChangeLog-Added: With `sqlite3` db backend we now use a 60-second busy timer, to allow backup processes like `litestream` to operate safely.
2021-10-17 23:58:39 +00:00
Rusty Russell
c012a71dec CI: suppress postgres vaccuuming.
```
[gw1] [ 98%] PASSED tests/test_wallet.py::test_hsmtool_dump_descriptors 
tests/test_wallet.py::test_fundchannel_listtransaction 
[gw0] [ 98%] PASSED tests/test_plugin.py::test_channel_opened_notification 
tests/test_wallet.py::test_hsmtool_generatehsm 
[gw0] [ 98%] PASSED tests/test_wallet.py::test_hsmtool_generatehsm 
tests/test_wallet.py::test_withdraw_nlocktime_fuzz 
[gw1] [ 98%] ERROR tests/test_wallet.py::test_fundchannel_listtransaction 
tests/test_wallet.py::test_fundchannel_listtransaction 
tests/test_wallet.py::test_withdraw_nlocktime_fuzz 
tests/test_wallet.py::test_fundchannel_listtransaction 
[gw0] [ 99%] ERROR tests/test_wallet.py::test_withdraw_nlocktime_fuzz 
tests/test_wallet.py::test_multiwithdraw_simple 
[gw1] [ 99%] ERROR tests/test_wallet.py::test_fundchannel_listtransaction 
tests/test_wallet.py::test_withdraw_nlocktime 
tests/test_wallet.py::test_multiwithdraw_simple 
tests/test_wallet.py::test_withdraw_nlocktime 
tests/test_wallet.py::test_multiwithdraw_simple 
tests/test_wallet.py::test_withdraw_nlocktime 
[gw0] [ 99%] ERROR tests/test_wallet.py::test_multiwithdraw_simple 
tests/test_wallet.py::test_repro_4258 
[gw1] [ 99%] ERROR tests/test_wallet.py::test_withdraw_nlocktime
...
2021-10-12 06:36:09.203 UTC [224552] STATEMENT:  SELECT version FROM version LIMIT 1
2021-10-12 06:36:09.566 UTC [224523] PANIC:  could not write to file "pg_wal/xlogtemp.224523": No space left on device
2021-10-12 06:36:09.566 UTC [224523] STATEMENT:  VACUUM FULL;
Error vacuuming db: BEGIN command failed: PANIC:  could not write to file "pg_wal/xlogtemp.224523": No space left on device
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
```
2021-10-15 12:09:36 +02:00
Rusty Russell
da03985996 wallet: only hand onchaind the HTLCs it needs to know.
This will make closing long-lived channels more efficient, and it's
just nicer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
c503232cde common: use bitcoin_outpoint.
I started pulling this thread, and the entire codebase got unravelled.

Oh well, it's done now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
2bb13bacc2 db: vacuum after a db upgrade.
This is particularly useful after our recent field deletion:

before: 362,573,824 bytes
after: 124,190,720 bytes

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: db: removal of old HTLC information and vacuuming shrinks large lightningd.sqlite3 by a factor of 2-3.
2021-10-15 12:09:36 +02:00
Rusty Russell
f986549cea db: clear our unneeded htlc fields in old terminated HTLCs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
433e61fcb9 wallet: remove unnecessary data from channel_htlcs when htlc is dead.
In particular, the onion and errors can be large, but now we'll never
need to retransmit them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
8a85bf6880 lightningd: populate min/max commit fields in db.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
36b66871a2 db: add min/max commitnum fields to channel_htlcs.
And initialize max to current height max when htlcs are already dead.
Turns out (thanks CI!) that MAX() of multiple columns is GREATEST() in
Postgres.  That's clearer (MAX is used elsewhere for single columns),
so translate on the sqlite3 side.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
bdaec48400 wallet: wrap htlc_state enum in db function.
All enums in the db should be wrapped this way on reading/writing them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Christian Decker
5f260840ab jsonrpc: Add groupid to waitsendpay 2021-10-13 13:41:18 +10:30
Christian Decker
817b1b361e db: Add groupid to HTLCs 2021-10-13 13:41:18 +10:30
Christian Decker
0bb0d8a707 wallet: Add function to retrieve the latest groupid for a payment
When doing things like `waitsendpay` without specifying the `groupid`
we likely want to use the latest `groupid` we created, since that's
the one in flight. This adds a function to quickly retrieve that.
2021-10-13 13:41:18 +10:30
Christian Decker
ba4be0313b db: Add groupid to the payments table 2021-10-13 13:41:18 +10:30
Christian Decker
c963e6b03f db: Don't switch between text and blob fields
The database will yell at you if you try.
2021-10-10 11:05:01 +10:30
Christian Decker
b9c602c7e4 db: Fix a syntax error with the optional parameters
SQL doesn't really allow `a OR 1` as a clause since `1` is not a
boolean expression. Moving it into `a OR 1=1` however is valid again.
2021-10-10 11:05:01 +10:30
Rusty Russell
79e09b92ef Makefile: remove generated files.
By popular merge-hell demand.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Build: Python is now required to build, as generated files are no longer checked into the repository.
2021-09-22 15:25:14 +02:00
Rusty Russell
3dbf3b057a Makefile: remove NO_PYTHON flag in anticipation of making it compulsory.
And note the EXPERIMENTAL_DUAL_FUND env var in HACKING.md.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-22 15:25:14 +02:00
Vincenzo Palazzo
eb103c15df review 2/2: Use generic query to make the code more readable.
Suggested by @cdecker

P.S: Also this include an API refactoring from my previous solution, also this it is suggested by @cdecker.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo
9d5c992fc8 plugin: Adding status to the pay plugin
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>

Rebase

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo
62ef403d20 doc: Update doc with the new parameter supported
Changelog-Added: Support to listpays the status parameter to filter the payments by status.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo
ee8c7252a3 wallet db: Support the query on database with status and payment hash.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Vincenzo Palazzo
fd33aed4b5 rpc: Integrate the status flow in the listsendpays command
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-09-22 16:23:24 +09:30
Rusty Russell
24536c5561 common/autodata: use instead of ccan/autodata
This means it needs to be linked ~everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-21 18:04:43 +02:00
Rusty Russell
7401b26824 cleanup: remove unneeded includes in C files.
Before:
 Ten builds, laptop -j5, no ccache:

```
real	0m36.686000-38.956000(38.608+/-0.65)s
user	2m32.864000-42.253000(40.7545+/-2.7)s
sys	0m16.618000-18.316000(17.8531+/-0.48)s
```

 Ten builds, laptop -j5, ccache (warm):

```
real	0m8.212000-8.577000(8.39989+/-0.13)s
user	0m12.731000-13.212000(12.9751+/-0.17)s
sys	0m3.697000-3.902000(3.83722+/-0.064)s
```

After:
 Ten builds, laptop -j5, no ccache: 8% faster

```
real	0m33.802000-35.773000(35.468+/-0.54)s
user	2m19.073000-27.754000(26.2542+/-2.3)s
sys	0m15.784000-17.173000(16.7165+/-0.37)s
```

 Ten builds, laptop -j5, ccache (warm): 1% faster

```
real	0m8.200000-8.485000(8.30138+/-0.097)s
user	0m12.485000-13.100000(12.7344+/-0.19)s
sys	0m3.702000-3.889000(3.78787+/-0.056)s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
ea30c34d82 cleanup: remove unneeded includes in header files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
789b330338 lightningd/closing_control.c: move json_close here from peer_control.c
They share some code, but not much: command_find_channel is made
non-static.  Rest is move-only.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-15 18:23:30 +02:00
Rusty Russell
183fe107e8 lightningd: use channel_type, pass to-and-from channeld.
Instead of explicit option_static_remotekey and option_anchor_outputs flags.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell
6ee8c40b29 closing: add option to set closing range.
This affects the range we offer even without quick-close, but it's
more critical for quick-close.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSONRPC: `close` now takes a `feerange` parameter to set min/max fee rates for mutual close.
2021-09-09 12:04:48 +09:30
Rusty Russell
79d7e83f51 --experimental-quick-close to enable quick-close negotiation
Based on a commit by @niftynei, but:
- Separated quickclose logic from main loop.
- I made it indep of anchor_outputs, use and option instead.
- Disable if they've specified how to negotiate.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-09 12:04:48 +09:30
Rusty Russell
fb4edc2938 Makefile: update bolt version to include option_anchors_zero_fee_htlc_tx.
This touches a lot of text, mainly to change "if `option_anchor_outputs`"
to "if `option_anchors`"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-09 12:04:48 +09:30
Rusty Russell
d6192c2cb0 db: simple index addition to speed listpeers.
Fixes: #4718
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-04 13:41:35 +02:00
Rusty Russell
5bf2b5c0ba wallet: set a timeout if the db is busy.
This is recommended for litestream, which allows for easy async backup,
and harmless otherwise.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: db: we now set a busy timeout to safely allow others to access sqlite3 db (e.g. litestream)
2021-08-26 12:44:02 +09:30
Rusty Russell
fe86c117d9 datastore: turn keys into arrays
After some discussion with @shesek, and my own usage, we agreed that
a more comprehensive interface, which explicitly supports grouping,
is desirable.

Thus keys are now arrays, with the semantic that a key is either a
parent or has a value, never both.

For convenience in the JSON schema, we always return them as arrays,
though we accept simple strings as arguments.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-25 10:06:31 +02:00
Rusty Russell
533571a655 datastore: add generation, simple atomicity.
We add a generation counter, and allow update or del conditional
on a given generation.

Formalizes error codes, too, since we have more now.

Suggested-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-25 10:06:31 +02:00
Rusty Russell
432508e65e datastore: allow replace/append.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-25 10:06:31 +02:00
Rusty Russell
2fb8735f04 wallet: add datastore access routines.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-25 10:06:31 +02:00
Rusty Russell
df17387e05 wallet: add datastore for plugins to use.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-25 10:06:31 +02:00
Sebastian Falbesoner
48473eb3e9 wallet-test: fix segfault due to uninitialized block
The variable `block` (instace of `struct block`) is
allocated on the stack without being initialized, i.e. its
member `prev` points to nowhere. This causes a segmentation
fault on my machine on the binding of "prev_hash" on running
`wallet_block_add`, as the following core-dump analysis
shows:

    $ egdb ./wallet/test/run-wallet ./run-wallet.core
    [...]
    Core was generated by `run-wallet'.
    Program terminated with signal SIGSEGV, Segmentation fault.
    ---Type <return> to continue, or q <return> to quit---
    #0  0x000008f67a04b660 in memcpy (dst0=<optimized out>, src0=0x100007f8c, length=32) at /usr/src/lib/libc/string/memcpy.c:97
    97                      TLOOP1(*dst++ = *src++);
    (gdb) bt
    #0  0x000008f67a04b660 in memcpy (dst0=<optimized out>, src0=0x100007f8c, length=32) at /usr/src/lib/libc/string/memcpy.c:97
    #1  0x000008f73e838f60 in sqlite3VdbeMemSetStr () from /usr/local/lib/libsqlite3.so.37.12
    #2  0x000008f73e83cb11 in bindText () from /usr/local/lib/libsqlite3.so.37.12
    #3  0x000008f44bc91345 in db_sqlite3_query (stmt=0x8f6845bf028) at wallet/db_sqlite3.c:77
    #4  0x000008f44bc91122 in db_sqlite3_exec (stmt=0x8f6845bf028) at wallet/db_sqlite3.c:110
    #5  0x000008f44bcbb3b2 in db_exec_prepared_v2 (stmt=0x8f6845bf028) at ./wallet/db.c:2055
    #6  0x000008f44bcc6890 in wallet_block_add (w=0x8f688b5bba8, b=0x7f7ffffca788) at ./wallet/wallet.c:3556
    #7  0x000008f44bce2607 in test_wallet_outputs (ld=0x8f6a35a7828, ctx=0x8f6a35c0268) at wallet/test/run-wallet.c:1104
    #8  0x000008f44bcddec0 in main (argc=1, argv=0x7f7ffffcaaf8) at wallet/test/run-wallet.c:1930

Fix by explicitely setting the whole structure to zero.

[ Rebuilt generated files, too --RR ]
2021-08-23 19:46:04 +09:30
niftynei
4514d2a180 wallet-test: print better error message on failure 2021-08-17 13:20:00 +09:30
niftynei
773aa26321 utxos: more comprehensive fix for csv check
we only 'need' to check the CSV for 'option_anchor_outputs' (and even
then, really only ones with csv's greater than 1 but...)

Reported-In: #4697
2021-08-17 13:20:00 +09:30
Rusty Russell
66af5f8a28 chaintopology: tell gossipd that channels no longer exist on reorg.
This actually caused the flake in test_funding_reorg_private, where
l1 and l2 might not mark the original channel disabled.  In fact, they
should *remove* it as it gets reorged out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-14 12:07:38 +09:30
Antoine Poinsot
8157e378f4 wallet: correct CSV check in deep_enough
Return false if the timelock didn't mature yet, not the other way
around.

Also, the check shouldn't be strict: if the CSV is 1 it is valid
at utxo->blockheight + 1.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-08-09 13:57:36 -05:00
Rusty Russell
44c469d52b disableoffer: fix disabling of already-used offers.
Turns out we didn't actually test this at all, and next commit does :(

    offer_status_in_db: 4 is invalid
    lightningd: FATAL SIGNAL 6 (version v0.10.0-459-g48fbd45-modded)
    0x5608cd360855 send_backtrace
	common/daemon.c:39
    0x5608cd3608ff crashdump
	common/daemon.c:52
    0x7f9af1dae20f ???
	???:0
    0x7f9af1dae18b ???
	???:0
    0x7f9af1d8d858 ???
	???:0
    0x5608cd30a47e fatal
	lightningd/log.c:819
    0x5608cd3430c5 offer_status_in_db
	wallet/wallet.h:1424
    0x5608cd34f1f3 wallet_offer_disable
	wallet/wallet.c:4494
    0x5608cd33ae2e json_disableoffer
	lightningd/offer.c:256
    0x5608cd3038fc command_exec
	lightningd/jsonrpc.c:643

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
niftynei
03ccebc296 utxo: don't add outputs that aren't currently eligible to be spent
If an output's CSV lock hasn't been surpassed yet, don't try to
include it in a transaction
2021-07-20 13:28:38 -04:00
niftynei
86f7a179a9 channel utxo: persist the 'csv' lock value to database
Channel leases modify the CSV height that an output is eligible for
being spent at,  persist this to the database
2021-07-20 13:28:38 -04:00
niftynei
67b8a22aa7 channel: wire blockheight updates from channeld in to database 2021-07-20 13:28:38 -04:00
niftynei
265f960cfe liquidity-ads: persist channel blockheight states to disk
Adds new tables to database, backfills, basically copies the fee_rates
state machine for channeld.
2021-07-20 13:28:38 -04:00