And initialize max to current height max when htlcs are already dead.
Turns out (thanks CI!) that MAX() of multiple columns is GREATEST() in
Postgres. That's clearer (MAX is used elsewhere for single columns),
so translate on the sqlite3 side.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When doing things like `waitsendpay` without specifying the `groupid`
we likely want to use the latest `groupid` we created, since that's
the one in flight. This adds a function to quickly retrieve that.
By popular merge-hell demand.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Build: Python is now required to build, as generated files are no longer checked into the repository.
Suggested by @cdecker
P.S: Also this include an API refactoring from my previous solution, also this it is suggested by @cdecker.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Added: Support to listpays the status parameter to filter the payments by status.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Before:
Ten builds, laptop -j5, no ccache:
```
real 0m36.686000-38.956000(38.608+/-0.65)s
user 2m32.864000-42.253000(40.7545+/-2.7)s
sys 0m16.618000-18.316000(17.8531+/-0.48)s
```
Ten builds, laptop -j5, ccache (warm):
```
real 0m8.212000-8.577000(8.39989+/-0.13)s
user 0m12.731000-13.212000(12.9751+/-0.17)s
sys 0m3.697000-3.902000(3.83722+/-0.064)s
```
After:
Ten builds, laptop -j5, no ccache: 8% faster
```
real 0m33.802000-35.773000(35.468+/-0.54)s
user 2m19.073000-27.754000(26.2542+/-2.3)s
sys 0m15.784000-17.173000(16.7165+/-0.37)s
```
Ten builds, laptop -j5, ccache (warm): 1% faster
```
real 0m8.200000-8.485000(8.30138+/-0.097)s
user 0m12.485000-13.100000(12.7344+/-0.19)s
sys 0m3.702000-3.889000(3.78787+/-0.056)s
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This affects the range we offer even without quick-close, but it's
more critical for quick-close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSONRPC: `close` now takes a `feerange` parameter to set min/max fee rates for mutual close.
Based on a commit by @niftynei, but:
- Separated quickclose logic from main loop.
- I made it indep of anchor_outputs, use and option instead.
- Disable if they've specified how to negotiate.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This touches a lot of text, mainly to change "if `option_anchor_outputs`"
to "if `option_anchors`"
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is recommended for litestream, which allows for easy async backup,
and harmless otherwise.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: db: we now set a busy timeout to safely allow others to access sqlite3 db (e.g. litestream)
After some discussion with @shesek, and my own usage, we agreed that
a more comprehensive interface, which explicitly supports grouping,
is desirable.
Thus keys are now arrays, with the semantic that a key is either a
parent or has a value, never both.
For convenience in the JSON schema, we always return them as arrays,
though we accept simple strings as arguments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We add a generation counter, and allow update or del conditional
on a given generation.
Formalizes error codes, too, since we have more now.
Suggested-by: @shesek
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The variable `block` (instace of `struct block`) is
allocated on the stack without being initialized, i.e. its
member `prev` points to nowhere. This causes a segmentation
fault on my machine on the binding of "prev_hash" on running
`wallet_block_add`, as the following core-dump analysis
shows:
$ egdb ./wallet/test/run-wallet ./run-wallet.core
[...]
Core was generated by `run-wallet'.
Program terminated with signal SIGSEGV, Segmentation fault.
---Type <return> to continue, or q <return> to quit---
#0 0x000008f67a04b660 in memcpy (dst0=<optimized out>, src0=0x100007f8c, length=32) at /usr/src/lib/libc/string/memcpy.c:97
97 TLOOP1(*dst++ = *src++);
(gdb) bt
#0 0x000008f67a04b660 in memcpy (dst0=<optimized out>, src0=0x100007f8c, length=32) at /usr/src/lib/libc/string/memcpy.c:97
#1 0x000008f73e838f60 in sqlite3VdbeMemSetStr () from /usr/local/lib/libsqlite3.so.37.12
#2 0x000008f73e83cb11 in bindText () from /usr/local/lib/libsqlite3.so.37.12
#3 0x000008f44bc91345 in db_sqlite3_query (stmt=0x8f6845bf028) at wallet/db_sqlite3.c:77
#4 0x000008f44bc91122 in db_sqlite3_exec (stmt=0x8f6845bf028) at wallet/db_sqlite3.c:110
#5 0x000008f44bcbb3b2 in db_exec_prepared_v2 (stmt=0x8f6845bf028) at ./wallet/db.c:2055
#6 0x000008f44bcc6890 in wallet_block_add (w=0x8f688b5bba8, b=0x7f7ffffca788) at ./wallet/wallet.c:3556
#7 0x000008f44bce2607 in test_wallet_outputs (ld=0x8f6a35a7828, ctx=0x8f6a35c0268) at wallet/test/run-wallet.c:1104
#8 0x000008f44bcddec0 in main (argc=1, argv=0x7f7ffffcaaf8) at wallet/test/run-wallet.c:1930
Fix by explicitely setting the whole structure to zero.
[ Rebuilt generated files, too --RR ]
This actually caused the flake in test_funding_reorg_private, where
l1 and l2 might not mark the original channel disabled. In fact, they
should *remove* it as it gets reorged out.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Return false if the timelock didn't mature yet, not the other way
around.
Also, the check shouldn't be strict: if the CSV is 1 it is valid
at utxo->blockheight + 1.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Turns out we didn't actually test this at all, and next commit does :(
offer_status_in_db: 4 is invalid
lightningd: FATAL SIGNAL 6 (version v0.10.0-459-g48fbd45-modded)
0x5608cd360855 send_backtrace
common/daemon.c:39
0x5608cd3608ff crashdump
common/daemon.c:52
0x7f9af1dae20f ???
???:0
0x7f9af1dae18b ???
???:0
0x7f9af1d8d858 ???
???:0
0x5608cd30a47e fatal
lightningd/log.c:819
0x5608cd3430c5 offer_status_in_db
wallet/wallet.h:1424
0x5608cd34f1f3 wallet_offer_disable
wallet/wallet.c:4494
0x5608cd33ae2e json_disableoffer
lightningd/offer.c:256
0x5608cd3038fc command_exec
lightningd/jsonrpc.c:643
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were getting off-by-one for the total amount that the change is for,
since it rounds the fee *down*, independent of the total weight of the
entire tx.
We fix this by using the diff btw the fee of the total weight (w/ and
w/o the change output)
And add the local_offer_id to the schemas too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSONRPC: `listoffers` now shows `local_offer_id` when listing all offers.
This supports reestablish on a closed channel: we tell channeld to
respond to the reestablish message appropriately, then close the
channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It handles all the cases of retransmission, and in the normal case
retransmits shutdown and immediately returns for us to run closingd.
This is actually far simpler and reduces code duplication.
[ Includes fixup to stop warn_unused_result from Christian ]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: We could get stuck on signature exchange if we needed to retransmit the final revoke_and_ack.
That's a terrible, terrible idea. (Documentation comes in later patch
which has the schema).
Also, blockheight is a u32, so simplify.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `listtransactions` `outputs` `satoshis` field (use `msat` instead).
Prior to this, sending a v1 address (or, in fact, any random crap!)
would cause the unsupporting node to unilaterally close.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This commit introduces the code cleanup suggested by the TODO comment in the code.
Basically, it moves the code from the if-else statement to a switch statement without the default case. I used the basic idea of the code used in PR #4507.
Changelog-Changed: None.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
For markdown, there's no simple comment prefix: we need a postfix too.
We also need to use "" since we want to use ' in some of the Makefiles
in future when V=1.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Not an API break: reserve=true|false still works for fundpsbt and utxopsbt,
but we also allow a raw number in there.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We already have this field in reserveinputs and unreserveinputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listfunds` has a new `reserved_to_block` field.
Tor v2 hidden services have been deprecated for a while:
https://blog.torproject.org/v2-deprecation-timeline .
This prevents user from being able to set them in the configuration
and to connect to them while still letting us be able to parse them
for gossip.
Changelog-Deprecated: lightningd: v2 Tor addresses. Use v3. See https://blog.torproject.org/v2-deprecation-timeline.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
If the peer is offline when we see the funding txid, we don't actually
update the channel's info. Here, we move it up to where the scid is set,
so that we always update the channel's funding_txid to the correct
(mined) information.
if the utxo can't pay for its own fees, dont put it in the tx
Changelog-Changed: JSONRPC: fundpsbt will not include UTXOs that aren't economic (can't pay for their own fees), unless 'all'
If you do update-mocks in a dirty tree, the recursive make that it
uses will try to rebuild things! Suppress that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Trying to put all the disconnect logic into the same path was a dumb
idea. If you asked to reconnect but passed in an 'unsaved' channel, we
would not call the 'reconnect' code.
Instead, we make a differentiation between "unsaved" channels
(ones that we haven't received commitment tx for) and handle the
disconnect for these separate from where we want to do a reconnect.
This means remembering the connection direction. We also use the address to try
to reconnect, which we shouldn't bother with if they connect to us.
For peers from the database, we currently always save the addr: we shouldn't really
do this if they connected to us, since it's not useful for reconnecting (we don't
show the addr in JSON reply to listpeers unless we're connected, so it's only an
internal issue). This is left for future work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This matters: if we connected, the address is probably usable for future connections.
But if they connected, the port is probably not (but the IP address may be).
Changelog-Added: JSON-RPC: `connect` returns "direction" ("in": they iniatated, or "out": we initiated)
Changelog-Added: plugins: `peer_connected` hook and `connect` notifications have "direction" field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise, we might find an address other than the one given and
the user might think that address worked.
Fixes: #4185
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `connect` returns `address` it actually connected to
Changelog-Added: lightningd: experimental-shutdown-wrong-funding to allow remote nodes to close incorrectly opened channels.
Changelog-Added: JSON-RPC: close has a new `wrong_funding` option to try to close out unused channels where we messed up the funding tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Allows us to clean up an in-progress open that we won't be completing
Changelog-Added: EXPERIMENTAL JSON-RPC: Permit user-initiated aborting of in-progress opens. Only valid for not-yet-committed opens and RBF-attempts
We move over to the new "warning" paradigm, instead of using
an "rbf_fail" message.
Every failure is either a warning or an error; on warnings we
hang up and reconnect later, effectively resetting the state.
Users have no idea what they would pay for unilateral closes.
At least this gives them a clue!
Reported-by: @az0re on IRC.
Changelog-Added: JSON-RPC: `listpeers` now shows latest feerate and unilaral close fee.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were always prefixing the `message` field with the internal type
prefix 0x0407, followed by the length prefix. Neither is needed since
the type being constant is of no interest to the plugin and the length
being implicit due to the JSON-encoding.
Reported-by: Ilya Evdokimov
Changelog-Fixed: plugin: The `custommsg` hook no longer includes the internal type prefix and length prefix in its `payload`
Changelog-Deprecated: plugin: The `message` field on the `custommsg` hook is deprecated in favor of the `payload` field, which skips the internal prefix.
We used to only set it for single-use offers (where it's required),
but it's still interesting for multi-use offers, so let's keep it
there.
We also put this field in the documentation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This ensures that after the migration in the previous commit we never
insert a new htlc with a null value.
Fixes: #4363
Reported-by: Zoltán Gálli <@gallizoltan>
Changelog-Fixed: db: Fixed an access to a NULL-field in the `channel_htlcs` table and resulting warning.
The leak exists if we `tal_free` the result array onto another parent,
but the `ctx` we allocated on is still valid. This leads to a
temporary gap in the ownership tree which is then reported as the
following error:
```text
- Node /tmp/ltests-ufn3ox3p/test_htlc_out_timeout_1/lightning-1/ has memory leaks: [
{
"backtrace": [
"ccan/ccan/tal/tal.c:442 (tal_alloc_)",
"ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)",
"ccan/ccan/tal/tal.c:799 (tal_dup_)",
"ccan/ccan/tal/str/str.c:18 (tal_strdup_)",
"wallet/wallet.c:1652 (wallet_state_change_get)",
"lightningd/peer_control.c:869 (json_]add_channel)",
"lightningd/peer_control.c:1319 (json_add_peer)",
"lightningd/peer_control.c:1348 (json_listpeers)",
"lightningd/jsonrpc.c:643 (command_exec)",
"lightningd/jsonrpc.c:753 (rpc_command_hook_callback)",
"lightningd/plugin_hook.c:288 (plugin_hook_call_)",
"lightningd/jsonrpc.c:808 (plugin_hook_call_rpc_command)",
"lightningd/jsonrpc.c:888 (parse_request)",
"lightningd/jsonrpc.c:979 (read_json)",
"ccan/ccan/io/io.c:59 (next_plan)",
"ccan/ccan/io/io.c:435 (io_do_always)",
"ccan/ccan/io/poll.c:300 (handle_always)",
"ccan/ccan/io/poll.c:377 (io_loop)",
"lightningd/io_loop_with_timers.c:24 (io_loop_with_timers)",
"lightningd/lightningd.c:1016 (main)"
],
"label": "wallet/wallet.c:1652:char[]",
"parents": [
"common/json_stream.c:29:struct json_stream",
"ccan/ccan/io/io.c:91:struct io_conn",
"lightningd/lightningd.c:116:struct lightningd"
],
"value": "0x556b0856ab68"
},
```
Changelog-None
In the case where you want a PSBT and also want the output to be added
as a change address, use `excess_as_change` = true.
Generates a change address to use. If you want to pay the excess
elsewhere, you will have to add separately.
Changelog-Added: JSON-RPC: Add new parameter `excess_as_change` to fundpsbt+utxopsbt
No more sending "all-channel" errors; in particular, gossipd now only
sends warnings (which make us hang up), not errors, and peer_connected
rejections are warnings (and disconnect), not errors.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Plugins: `peer_connected` rejections now send a warning, not an error, to the peer.
Prior to this, all reconnect logic lived in channeld. If you
disconnected before we finished building a funding transaction, that was
no big deal. Now, however, we're waiting for the funding to lock in in
dualopend, instead of handing straight to channeld to wait.
So we need a way to restart dualopend.
We need to know if they've sent us their sigs message yet. Ideally, we'd
be able to check the 'finalness' of the PSBT, however if the peer
doesn't have any inputs to the channel this doesn't work.
This reverts commit c239a7161b.
The goal of c239a716 was to reduce the memory footprint of our
internal UTXO set tracking, and testing against the sqlite3 backend
showed no performance impact. We have since found that the added
roundtrips to the DB server with postgres was having a considerable
performance impact, and backups would also bloat due to the increased
number of queries.
Undoing this change skips the noop updates that were causing this
regression.
Changelog-Fixed: db: Fixed a performance regression during block sync, resulting in many more queries against the DB than necessary.
1. Hoist 7200 constant into the bolt12 heade2.
2. Make preimage the last createinvoice arg, so we could make it optional.
3. Check the validity of the preimage in createinvoice.
4. Always output used flag in listoffers.
5. Rename wallet offer iterators to offer_id iterators.
6. Fix paramter typos.
7. Rename `local_offer_id` parameter to `localofferid`.
8. Add reference constraints on local_offer_id db fields.
9. Remove cut/paste comment.
10. Clarify source of fatal() messages in wallet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This takes an unsigned bolt11 (or bolt12 if EXPERIMENTAL_FEATURES) string
and signs it and puts it in the database.
The invoice command could now be moved out to a plugin, in fact.
Changelog-Added: JSON-RPC: `createinvoice` new low-level invoice creation API.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is for offers which have `send_invoice`: we need to associate the
payment with the original offer, in (the usual) case where it is a single
use offer. We mark it used when it's paid, to avoid a race.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows us to mark an offer used when an invoice derived from it
is paid, and importantly, avoid any other invoices for the offer being
paid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Still asserts that it's the standard size, but makes it a dynamic
member. For simpliciy, changes the parse_onionpacket API (it must be
a tal object now, so we might as well allocate it here to catch all
the callers).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's actually a (very unlikely) race here: we would previously have
crashed with an assertion in invoices_resolve.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Needed for v2 of channel opens, where the minimum weight is 110; a
'simple utxo' (sig + key) weighs in at 107, so we a need a way to
establish a floor for this case.
Changelog-Added: JSON-RPC: fundpsbt/utxopsbt have new param, `min_witness_utxo`, which sets a floor for the weight calculation of an added input
Avoids much cut & paste. Some tests don't need any of it, but most
want at least some of this infrastructure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a `state_change` 'cause' to a channel.
A 'cause' is some initial 'reason' a channel was created or closed by:
/* Anything other than the reasons below. Should not happen. */
REASON_UNKNOWN,
/* Unconscious internal reasons, e.g. dev fail of a channel. */
REASON_LOCAL,
/* The operator or a plugin opened or closed a channel by intention. */
REASON_USER,
/* The remote closed or funded a channel with us by intention. */
REASON_REMOTE,
/* E.g. We need to close a channel because of bad signatures and such. */
REASON_PROTOCOL,
/* A channel was closed onchain, while we were offline. */
/* Note: This is very likely a conscious remote decision. */
REASON_ONCHAIN
If a 'cause' is known and a subsequent state change is made with
`REASON_UNKNOWN` the preceding cause will be used as reason, since a lot
(all `REASON_UNKNOWN`) state changes are a subsequent consequences of a prior
cause: local, user, remote, protocol or onchain.
Changelog-Added: Plugins: Channel closure resaon/cause to channel_state_changed notification
== is a bash extension; in shell it's a single =:
```
/bin/sh: 1: [: unexpected operator
sql-rewrite wallet/db_postgres_sqlgen.c
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For compatibility, we only do this if `allow-deprecated-apis` is false
for now. Otherwise scripts parsing should use `grep -v '^# '` or
start using `-N none`.
Changelog-Added: JSON-RPC: `close` now sends notifications for slow closes (if `allow-deprecated-apis`=false)
Changelog-Deprecated: cli: scripts should filter out '^# ' or use `-N none`, as commands will start returning notifications soon
Fixes: #3925
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need the PSBT to create the finalized tx from once the peer's
tx_signatures are received. Since we're passing the PSBT, we no longer
need the secondary message to be passed, as it was derived from the
PSBT.
Also removes now unused witness serialization code
We need this for dual funding, since the interactive tx construction
protocol requires the full tx to send places. We add it to all PSBTs (if
we have it), here
This adds an environment variable $NO_PYTHON check to Makefiles so that:
- It checks and runs into an defined error instead of some python hickup:
`ModuleNotFoundError: No module named 'mako'`
- makes it possible to manually export this environment variable NO_PYTHON=1
to run the same testcase that travis is running on job 1 which causes
so much pain ;)
Changelog-None
I discovered this accidentally when using the `tests/plugins/dblog.py`
plugin on another testcase: tests/test_connection.py::test_fail_unconfirmed
There the plugin/hook crashes because it can't execute the statement:
```json
{
"jsonrpc": "2.0",
"id": 34,
"error": {
"code": -32600,
"message": "Error while processing db_write: unrecognized token: \"174WHERE\"",
"traceback": "Traceback (most recent call last):\n File \"/home/will/projects/lightning.git/contrib/pyln-client/pyln/client/plugin.py\", line 535, in _dispatch_request\n result = self._exec_func(method.func, request)\n File \"/home/will/projects/lightning.git/contrib/pyln-client/pyln/client/plugin.py\", line 520, in _exec_func\n return func(*ba.args, **ba.kwargs)\n File \"/home/will/projects/lightning.git/tests/plugins/dblog.py\", line 45, in db_write\n plugin.conn.execute(c)\nsqlite3.OperationalError: unrecognized token: \"174WHERE\"\n"
}
}
```
Changelog-Fixed: plugin: Regression with SQL statement expansion that could result in invalid statements being passed to the `db_write` hook.
I got a corrupt file, which looked like multiple concurrent attempts
to build it. So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).
Also move into plugins/ where it logically belongs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We force use of tal_wally_start/tal_wally_end around every wally
allocation, and with "end" make the caller choose where to reparent
everything.
This is particularly powerful where we allocate a tx or a psbt: we
want that tx or psbt to be the parent of the other allocations, so
this way we can reparent the tx or psbt, then reparent everything
else onto it.
Implementing psbt_finalize (which uses a behavior flag antipattern)
was tricky, so I ended up splitting that into 'psbt_finalize' and
'psbt_final_tx', which I think also makes the callers clearer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are simple space-separated key-value pair sets of options instead of the
URI style DSNs, but they are also much more flexible allowing the user to
specify client SSL certificates, server certificates, compression and
encryption levels, and much more (see [1] for more information)
[1]: https://www.postgresql.org/docs/9.1/libpq-connect.html
Changelog-Added: db: Added support for key-value DSNs for postgresql, allowing for a wider variety of configurations and environments.
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This notification will be raised whenever a channel state changes.
The payload includes the channel and peer identifiers and the
old and the new state.
Example payload:
```
{
"channel_state_changed": {
"peer_id": "03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251",
"channel_id": "a2d0851832f0e30a0cf778a826d72f077ca86b69f72677e0267f23f63a0599b4",
"short_channel_id" : "561820x1020x1",
"old_state": "CHANNELD_NORMAL",
"new_state": "AWAITING_UNILATERAL"
}
}
```
Changelog-Added: Plugins: channel_state_changed notification
Greatly simplify the changeset API. Instead of 'diff' we simply generate
the changes.
Also pulls up the 'next message' method, as at some point the
interactive tx protocol will be used for other things as well
(splices/closes etc)
Suggested-By: @rustyrussell
dual funding needs the max-witness-len and utxo fields set for every
input. we should add them when we create a 'fundpsbt', so that every
psbt that c-lightning generates is dual-funding ready
v2 of channel establishment, in the accpeter case, now sends 2 messages
to our peer after saving the information to disk (our commitment
signatures and our funding transaction signatures)
v2 channel open uses a different method to derive the channel_id, so now
we save it to the database so that we dont have to remember how to
derive it for each.
includes a migration for existing channels
fundpsbt / utxopsbt create a (typically) output-less PSBT,
however for elements we require the fees to be encapsulated in an
output.
this patch updates fundpsbt / utxopsbt to add a fee output for elements
transactions. includes test updates.
Fixes#3998
We defer the notification to gossipd till the end of the spends. By itself not
a huge change, but it allows us to later migrate to doing updates blindly and
using the DB as our ground truth. It also allows us to simplify the
`wallet_outpoint_spend` semantics to not return two values in the next commit.
We're going to want this for bolt13 formation as well.
As a result of reworking the logic into "candidate selection" then
"route hint selection", we need to change the way round-robin works.
We use a simple incrementing index now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This removes the reservation cleanup at startup, too, now they're all
using 'reserved_til'.
This changes test_withdraw, since it asserted that outputs were marked
spent as soon as we broadcast a transaction: now they're reserved until
it's mined. Similarly, test_addfunds_from_block assumed we'd see funds
as soon as we broadcast the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` now randomizes input and output order, not BIP69.
Marking spent means if the transaction doesn't confirm for some
reason, the user will need to force a rescan to find the funds. Now
we have timed reservations, reserving for (an additional) 12 hours
should be sufficient.
We also take this opportunity (now we have our own callback path)
to record the tx in the wallet only on success.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some minor phrasing differences cause test changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: txprepare reservations stay across restarts: use fundpsbt/reservepsbt/unreservepsbt
Changelog-Removed: txprepare `destination` `satoshi` argument form removed (deprecated v0.7.3)
If all is set, diff may not be:
==1230518== Use of uninitialised value of size 8
==1230518== at 0x4C5781B: _itoa_word (_itoa.c:179)
==1230518== by 0x4C736F4: __vfprintf_internal (vfprintf-internal.c:1687)
==1230518== by 0x4C88119: __vsnprintf_internal (vsnprintf.c:114)
==1230518== by 0x1D44B6: do_vfmt (str.c:66)
==1230518== by 0x1D45A0: tal_vfmt_ (str.c:92)
==1230518== by 0x1D4401: tal_fmt_ (str.c:44)
==1230518== by 0x15D30F: fmt_amount_sat (amount.c:60)
==1230518== by 0x15D338: fmt_amount_sat_ (amount.c:62)
==1230518== by 0x178C45: type_to_string_ (type_to_string.c:35)
==1230518== by 0x1B8F75: json_fundpsbt (reservation.c:394)
==1230518== by 0x12D0EC: command_exec (jsonrpc.c:602)
==1230518== by 0x12D598: rpc_command_hook_callback (jsonrpc.c:712)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids overwriting the ones in git, and generally makes things neater.
We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Generating these requires python tooling which may not be present, so only
clean them in maintainer mode. Also remove `wallet/statements.po` which is no
longer used or generated at all.
And rename them so they're not cleared by `make clean`. We leave the
old rules in place so old files get cleaned still.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS. Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: build: On some operating systems the postgresql library would not get picked up. `./configure` now uses `pg_config` to locate the headers.
Note that other directories were explicitly depending on the generated
file, instead of relying on their (already existing) dependency on
$(LIGHTNINGD_HSM_CLIENT_OBJS), so we remove that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means some files get renamed, and I took the opportunity to clarify
our naming (the *d* is important!)
1. channeld/channel_wire.csv -> channeld/channeld_wire.csv
2. channeld/gen_channel_wire.h -> channeld/channeld_wiregen.h
3. enum channel_wire_type -> enum channeld_wire
4. WIRE_CHANNEL_FUNDING_DEPTH -> WIRE_CHANNELD_FUNDING_DEPTH.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise valgrind gets upset when we *run* the statements: better
to get a backtrace when we bind, so we can tell which field it is!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We've never hit this, we do check them on insert, and it's slowing
down some operations unnecessarily.
$ time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null
Before:
real 0m1.781s
user 0m0.127s
sys 0m0.013s
After:
real 0m1.545s
user 0m0.124s
sys 0m0.024s
Also, the raw listsendpays drops from 0.983s to 0.676s.
(With -O3 -flto, listsendpays is 0.416s, listpays 0.971s).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an extra safety check for dual funding, where we only want to sign
the inputs we provided!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `signpsbt` takes an optional `signonly` array to limit what inputs to sign.
Required for dual funding where the opener sets it.
Changelog-Added: JSON-RPC: `fundpsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This more closely mirrors fundpsbt (which will only select unreserved ones)
but you can turn it off if you want to e.g. rbf in future.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: New low-level command `utxopsbt` to create PSBT from existing utxos.
It's *possible* to do this using various RPC calls, but it's
unfriendly:
1. Call getinfo to get the current block height.
2. Call listfunds to map the UTXOs.
3. Create the PSBT and hope you get all the fields correct.
Instead, this presents an interface just like `fundpsbt`, with identical
returns.
I think it's different enough to justify a new command (though it
shares much internally, of course).
In particular, it's now quite simple to create a command which uses
specified utxos, and then adds more to meet any shortfall.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to remember this in the db (it's a P2WSH for option_anchor_outputs),
and we need to set nSequence to 1 to spend it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is best done by passing `struct bitcoin_signature` around instead
of raw signatures. We still save raw sigs to the db, and of course the
wire protocol uses them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
HTLC fees increase (larger weight), and the fee paid by the opener
has to include the anchor outputs (i.e. 660 sats).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the same way we handle option_static_remotekey, which
is also sticky (if negotiated at opening time, it always applies).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You'd need this if you ever wanted to make your own PSBT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `listfunds` now has a `redeemscript` field for p2sh-wrapped outputs.
This is what txsend does, only we have a psbt so we have
to change the db interface to take a wally_tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It keeps multiple different variables around the loop, but a simple
"are we done yet?" helper makes this clearer and reduces special
cases or all-vs-target.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Includes:
psbt: Use renamed functions for new wally version
psbt: Set the transaction directly to avoid script workarounds
psbt: Use low-S grinding when computing signatures
tx: Use wally_tx_clone from libwally now that its exported
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
the way we use PSBTs to sign things requires that we have the
scriptpubkey available on the utxo so we can populate the witness-utxo
field with it.
this causes problems if we don't already have the scriptpubkey cached in
the database, as in *some* cases we require a round trip to the HSM to
populate them
to get over this hump, we backfill any and all missing scriptpubkey
information for the utxo's that we hold in our wallet.
this will allow us to clean up the NULL handling of missing
scriptpubkeys.
we're about to add a migration that requires access to the bip32_key
in order to calculate missing scriptpubkeys.
prior to this patch, we don't have access to the bip32 key in the db
migration, as it's set on the wallet but after the db migrations are
run.
here we patch it through so that every migration can access it
fundpsbt forces the caller to manually add their weight * feerate
to the satoshis they ask for. That means no named feerates.
Instead, create a startweight parameter and do the calc for them
internally, and return the feerate we used (and, while we're at it,
the estimated final weight).
This API change is best done now, as it would otherwise have to
be appended as a parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're actually going to deprecate this, so don't add new features!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: ***REMOVE*** JSON-API: `txprepare` returns a psbt version of the created transaction
Technically, they could do this themselves, but it's much nicer to have one
place to do it (and it makes sure we get the required information into the
PSBT, which is actually not entirely accessible through listfunds, as that
doesn't want to consult with the HSM for close outputs).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON RPC: new low-level coin selection `fundpsbt` routine.
This is the normal case: you only want to reserve inputs which
are not already reserved. This saves you iterating through the
results and unreserving some if you weren't exclusive.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
reserveinputs marks UTXOs reserved for 12 hours, so we won't select them
for spending: unreserveinputs marks them available again.
Exposes param_psbt() for wider use.
Disabled the test_sign_and_send_psbt since we're altering the API;
the final patch restores it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a new fundamental routine to obtain UTXOs from the database.
It's not the most efficient approach, as it returns a single UTXO at a
time, but it can consolidate all our UTXO handling (becoming more
complex by the reservation timeout logic).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These keep the struct utxo in sync with the database, explicitly:
these will be the only places where utxo->status is set.
The old routines will be removed at the end.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
PSETs have a bit different requirements. The witness_utxo needs
the asset tag + values, and these should also be added to the PSET
struct separately as well. To do this, we create a new 'init' method for
elements inputs, which takes care of the elements specific things.
This is the only place outside the wallet code where we create
a 'struct utxo', so it makes sense for us to move that logic inside
the wallet.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It looked like we weren't printing the address on closing outputs.
But we are, because the 'scriptPubkey' field is in the 'outputs' db
table since 0.7.3 (66a47d2761).
So make the logic clearer, and remove a completely bogus comment (UTXOs
with closing_info are definitely spendable!).
We export the json_add_utxos() for future use, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're not allowed to command_fail() once we've started json_success.
That's OK, because encoding a known output can only fail if something is
badly, badly wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are pulled from wallet/wallet.c, with the fix now that we grind sigs.
This reduces the fees we pay slightly, as you can see in the coinmoves changes.
I now print out all the coin moves in suitable format before we match:
you only see this if the test fails, but it's really helpful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: `txprepare` now prepares transactions whose `nLockTime` is set to the tip blockheight, instead of using 0. `fundchannel` will use `nLockTime` set to the tip blockheight as well.
We were assuming `wallet_channel_insert` that there cannot be a matching peer
if our in-memory representation isn't bound to it (`dbid == 0`). If we then
attempt to create the peer, and we already had one it'd cause a unique
constraint violation. As far as I can tell this could end up happening if we
have an uncommitted channel, and then exited without cleanup (`tal_destructor`
on the uncommitted channel not running). This could then leave the peer in the
DB. This is because the constraint that every peer has at least one channel is
not enforce at DB level, but rather in destructors that may or may not run.
Changelog-Fixed: Fixed a failing assertion if we reconnect to a peer that we had a channel with before, and then attempt to insert the peer into the DB twice.
Changelog-Added: JSON-RPC: new call `signpsbt` which will add the wallet's signatures to a provided psbt
Changelog-Added: JSON-RPC: new call `sendpsbt` which will finalize and send a signed PSBT
Reserve and unreserve wallet UTXOs using a PSBT which includes those
inputs.
Note that currently we unreserve inputs everytime the node restarts.
This will be addressed in a future commit.
Changelog-Added: JSON-RPC: Adds two new rpc methods, `reserveinputs` and `unreserveinputs`, which allow for reserving or unreserving wallet UTXOs
Unused here, but we'll use it in the next commit so that we can always
pass back the effective / used feerate to the caller of `reserveinputs`
This makes opening a channel much easier if we've internally determined
the feerate
We're not using the change_outnum for withdraw tx's (and the way
we were calculating it was broken as of the addition of 'multiple
outputs'). This removes the change output knowhow from withdraw_tx
entirely, and pushes the responsibility up to the caller to
include the change output in the output set if desired.
Consequently, we also remove the change output knowhow from hsmd.
