This is for offers which have `send_invoice`: we need to associate the
payment with the original offer, in (the usual) case where it is a single
use offer. We mark it used when it's paid, to avoid a race.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows us to mark an offer used when an invoice derived from it
is paid, and importantly, avoid any other invoices for the offer being
paid.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Still asserts that it's the standard size, but makes it a dynamic
member. For simpliciy, changes the parse_onionpacket API (it must be
a tal object now, so we might as well allocate it here to catch all
the callers).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's actually a (very unlikely) race here: we would previously have
crashed with an assertion in invoices_resolve.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Needed for v2 of channel opens, where the minimum weight is 110; a
'simple utxo' (sig + key) weighs in at 107, so we a need a way to
establish a floor for this case.
Changelog-Added: JSON-RPC: fundpsbt/utxopsbt have new param, `min_witness_utxo`, which sets a floor for the weight calculation of an added input
Avoids much cut & paste. Some tests don't need any of it, but most
want at least some of this infrastructure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a `state_change` 'cause' to a channel.
A 'cause' is some initial 'reason' a channel was created or closed by:
/* Anything other than the reasons below. Should not happen. */
REASON_UNKNOWN,
/* Unconscious internal reasons, e.g. dev fail of a channel. */
REASON_LOCAL,
/* The operator or a plugin opened or closed a channel by intention. */
REASON_USER,
/* The remote closed or funded a channel with us by intention. */
REASON_REMOTE,
/* E.g. We need to close a channel because of bad signatures and such. */
REASON_PROTOCOL,
/* A channel was closed onchain, while we were offline. */
/* Note: This is very likely a conscious remote decision. */
REASON_ONCHAIN
If a 'cause' is known and a subsequent state change is made with
`REASON_UNKNOWN` the preceding cause will be used as reason, since a lot
(all `REASON_UNKNOWN`) state changes are a subsequent consequences of a prior
cause: local, user, remote, protocol or onchain.
Changelog-Added: Plugins: Channel closure resaon/cause to channel_state_changed notification
== is a bash extension; in shell it's a single =:
```
/bin/sh: 1: [: unexpected operator
sql-rewrite wallet/db_postgres_sqlgen.c
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For compatibility, we only do this if `allow-deprecated-apis` is false
for now. Otherwise scripts parsing should use `grep -v '^# '` or
start using `-N none`.
Changelog-Added: JSON-RPC: `close` now sends notifications for slow closes (if `allow-deprecated-apis`=false)
Changelog-Deprecated: cli: scripts should filter out '^# ' or use `-N none`, as commands will start returning notifications soon
Fixes: #3925
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need the PSBT to create the finalized tx from once the peer's
tx_signatures are received. Since we're passing the PSBT, we no longer
need the secondary message to be passed, as it was derived from the
PSBT.
Also removes now unused witness serialization code
We need this for dual funding, since the interactive tx construction
protocol requires the full tx to send places. We add it to all PSBTs (if
we have it), here
This adds an environment variable $NO_PYTHON check to Makefiles so that:
- It checks and runs into an defined error instead of some python hickup:
`ModuleNotFoundError: No module named 'mako'`
- makes it possible to manually export this environment variable NO_PYTHON=1
to run the same testcase that travis is running on job 1 which causes
so much pain ;)
Changelog-None
I discovered this accidentally when using the `tests/plugins/dblog.py`
plugin on another testcase: tests/test_connection.py::test_fail_unconfirmed
There the plugin/hook crashes because it can't execute the statement:
```json
{
"jsonrpc": "2.0",
"id": 34,
"error": {
"code": -32600,
"message": "Error while processing db_write: unrecognized token: \"174WHERE\"",
"traceback": "Traceback (most recent call last):\n File \"/home/will/projects/lightning.git/contrib/pyln-client/pyln/client/plugin.py\", line 535, in _dispatch_request\n result = self._exec_func(method.func, request)\n File \"/home/will/projects/lightning.git/contrib/pyln-client/pyln/client/plugin.py\", line 520, in _exec_func\n return func(*ba.args, **ba.kwargs)\n File \"/home/will/projects/lightning.git/tests/plugins/dblog.py\", line 45, in db_write\n plugin.conn.execute(c)\nsqlite3.OperationalError: unrecognized token: \"174WHERE\"\n"
}
}
```
Changelog-Fixed: plugin: Regression with SQL statement expansion that could result in invalid statements being passed to the `db_write` hook.
I got a corrupt file, which looked like multiple concurrent attempts
to build it. So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).
Also move into plugins/ where it logically belongs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We force use of tal_wally_start/tal_wally_end around every wally
allocation, and with "end" make the caller choose where to reparent
everything.
This is particularly powerful where we allocate a tx or a psbt: we
want that tx or psbt to be the parent of the other allocations, so
this way we can reparent the tx or psbt, then reparent everything
else onto it.
Implementing psbt_finalize (which uses a behavior flag antipattern)
was tricky, so I ended up splitting that into 'psbt_finalize' and
'psbt_final_tx', which I think also makes the callers clearer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are simple space-separated key-value pair sets of options instead of the
URI style DSNs, but they are also much more flexible allowing the user to
specify client SSL certificates, server certificates, compression and
encryption levels, and much more (see [1] for more information)
[1]: https://www.postgresql.org/docs/9.1/libpq-connect.html
Changelog-Added: db: Added support for key-value DSNs for postgresql, allowing for a wider variety of configurations and environments.
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This notification will be raised whenever a channel state changes.
The payload includes the channel and peer identifiers and the
old and the new state.
Example payload:
```
{
"channel_state_changed": {
"peer_id": "03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251",
"channel_id": "a2d0851832f0e30a0cf778a826d72f077ca86b69f72677e0267f23f63a0599b4",
"short_channel_id" : "561820x1020x1",
"old_state": "CHANNELD_NORMAL",
"new_state": "AWAITING_UNILATERAL"
}
}
```
Changelog-Added: Plugins: channel_state_changed notification
Greatly simplify the changeset API. Instead of 'diff' we simply generate
the changes.
Also pulls up the 'next message' method, as at some point the
interactive tx protocol will be used for other things as well
(splices/closes etc)
Suggested-By: @rustyrussell
dual funding needs the max-witness-len and utxo fields set for every
input. we should add them when we create a 'fundpsbt', so that every
psbt that c-lightning generates is dual-funding ready
v2 of channel establishment, in the accpeter case, now sends 2 messages
to our peer after saving the information to disk (our commitment
signatures and our funding transaction signatures)
v2 channel open uses a different method to derive the channel_id, so now
we save it to the database so that we dont have to remember how to
derive it for each.
includes a migration for existing channels
fundpsbt / utxopsbt create a (typically) output-less PSBT,
however for elements we require the fees to be encapsulated in an
output.
this patch updates fundpsbt / utxopsbt to add a fee output for elements
transactions. includes test updates.
Fixes#3998
We defer the notification to gossipd till the end of the spends. By itself not
a huge change, but it allows us to later migrate to doing updates blindly and
using the DB as our ground truth. It also allows us to simplify the
`wallet_outpoint_spend` semantics to not return two values in the next commit.
We're going to want this for bolt13 formation as well.
As a result of reworking the logic into "candidate selection" then
"route hint selection", we need to change the way round-robin works.
We use a simple incrementing index now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This removes the reservation cleanup at startup, too, now they're all
using 'reserved_til'.
This changes test_withdraw, since it asserted that outputs were marked
spent as soon as we broadcast a transaction: now they're reserved until
it's mined. Similarly, test_addfunds_from_block assumed we'd see funds
as soon as we broadcast the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` now randomizes input and output order, not BIP69.
Marking spent means if the transaction doesn't confirm for some
reason, the user will need to force a rescan to find the funds. Now
we have timed reservations, reserving for (an additional) 12 hours
should be sufficient.
We also take this opportunity (now we have our own callback path)
to record the tx in the wallet only on success.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some minor phrasing differences cause test changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: txprepare reservations stay across restarts: use fundpsbt/reservepsbt/unreservepsbt
Changelog-Removed: txprepare `destination` `satoshi` argument form removed (deprecated v0.7.3)
If all is set, diff may not be:
==1230518== Use of uninitialised value of size 8
==1230518== at 0x4C5781B: _itoa_word (_itoa.c:179)
==1230518== by 0x4C736F4: __vfprintf_internal (vfprintf-internal.c:1687)
==1230518== by 0x4C88119: __vsnprintf_internal (vsnprintf.c:114)
==1230518== by 0x1D44B6: do_vfmt (str.c:66)
==1230518== by 0x1D45A0: tal_vfmt_ (str.c:92)
==1230518== by 0x1D4401: tal_fmt_ (str.c:44)
==1230518== by 0x15D30F: fmt_amount_sat (amount.c:60)
==1230518== by 0x15D338: fmt_amount_sat_ (amount.c:62)
==1230518== by 0x178C45: type_to_string_ (type_to_string.c:35)
==1230518== by 0x1B8F75: json_fundpsbt (reservation.c:394)
==1230518== by 0x12D0EC: command_exec (jsonrpc.c:602)
==1230518== by 0x12D598: rpc_command_hook_callback (jsonrpc.c:712)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids overwriting the ones in git, and generally makes things neater.
We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Generating these requires python tooling which may not be present, so only
clean them in maintainer mode. Also remove `wallet/statements.po` which is no
longer used or generated at all.
And rename them so they're not cleared by `make clean`. We leave the
old rules in place so old files get cleaned still.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS. Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: build: On some operating systems the postgresql library would not get picked up. `./configure` now uses `pg_config` to locate the headers.
Note that other directories were explicitly depending on the generated
file, instead of relying on their (already existing) dependency on
$(LIGHTNINGD_HSM_CLIENT_OBJS), so we remove that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means some files get renamed, and I took the opportunity to clarify
our naming (the *d* is important!)
1. channeld/channel_wire.csv -> channeld/channeld_wire.csv
2. channeld/gen_channel_wire.h -> channeld/channeld_wiregen.h
3. enum channel_wire_type -> enum channeld_wire
4. WIRE_CHANNEL_FUNDING_DEPTH -> WIRE_CHANNELD_FUNDING_DEPTH.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise valgrind gets upset when we *run* the statements: better
to get a backtrace when we bind, so we can tell which field it is!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We've never hit this, we do check them on insert, and it's slowing
down some operations unnecessarily.
$ time lightning-cli -R --network=regtest --lightning-dir /tmp/ltests-k8jhvtty/test_pay_stress_1/lightning-1/ listpays > /dev/null
Before:
real 0m1.781s
user 0m0.127s
sys 0m0.013s
After:
real 0m1.545s
user 0m0.124s
sys 0m0.024s
Also, the raw listsendpays drops from 0.983s to 0.676s.
(With -O3 -flto, listsendpays is 0.416s, listpays 0.971s).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is an extra safety check for dual funding, where we only want to sign
the inputs we provided!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `signpsbt` takes an optional `signonly` array to limit what inputs to sign.
Required for dual funding where the opener sets it.
Changelog-Added: JSON-RPC: `fundpsbt` takes a new `locktime` parameter
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This more closely mirrors fundpsbt (which will only select unreserved ones)
but you can turn it off if you want to e.g. rbf in future.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: New low-level command `utxopsbt` to create PSBT from existing utxos.
It's *possible* to do this using various RPC calls, but it's
unfriendly:
1. Call getinfo to get the current block height.
2. Call listfunds to map the UTXOs.
3. Create the PSBT and hope you get all the fields correct.
Instead, this presents an interface just like `fundpsbt`, with identical
returns.
I think it's different enough to justify a new command (though it
shares much internally, of course).
In particular, it's now quite simple to create a command which uses
specified utxos, and then adds more to meet any shortfall.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to remember this in the db (it's a P2WSH for option_anchor_outputs),
and we need to set nSequence to 1 to spend it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is best done by passing `struct bitcoin_signature` around instead
of raw signatures. We still save raw sigs to the db, and of course the
wire protocol uses them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
HTLC fees increase (larger weight), and the fee paid by the opener
has to include the anchor outputs (i.e. 660 sats).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the same way we handle option_static_remotekey, which
is also sticky (if negotiated at opening time, it always applies).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You'd need this if you ever wanted to make your own PSBT.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: `listfunds` now has a `redeemscript` field for p2sh-wrapped outputs.
This is what txsend does, only we have a psbt so we have
to change the db interface to take a wally_tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It keeps multiple different variables around the loop, but a simple
"are we done yet?" helper makes this clearer and reduces special
cases or all-vs-target.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Includes:
psbt: Use renamed functions for new wally version
psbt: Set the transaction directly to avoid script workarounds
psbt: Use low-S grinding when computing signatures
tx: Use wally_tx_clone from libwally now that its exported
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
