"multi" means that specifying a parameter twice will append, not override.
Multi args are always given as a JSON array, even if only one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: new "multi" field allows an option to be specified multiple times.
This was fixed in 0.8.2.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: plugins: options to init are no longer given as strings if they are bool or int types (deprecated in 0.8.2).
Still asserts that it's the standard size, but makes it a dynamic
member. For simpliciy, changes the parse_onionpacket API (it must be
a tal object now, so we might as well allocate it here to catch all
the callers).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were masquerading errors when parsing the request by reporting only
a bogus malformed `id` field in the response, when the real issue was
that we were unable to parse the request in the first place (which
caused the null-id error to be returned).
Fixes#4238
Only way to be sure that plugins don't accidentally respond to onion_message
sent via reply path from another message (which would potentially leak our
identity!).
To quote BOLT #7 (Onion Messages) in the offers PR:
```markdown
The reader:
- MUST ignore any message which contains a `blinding` which it did not expect, or does not contain
a `blinding` when one is expected.
...
`blinding` is critical to the use of blinded paths: there are various
means by which a blinded path is passed to a node. The receipt of an
expected `blinding` indicates that blinded path has been used: it is
important that a node not accept unblinded messages when it is expecting
a blinded message, as this implies the sender is probing to detect if
the recipient is the terminus of the blinded path.
Similarly, since blinded paths don't expire, a node could try to use
a blinded path to send an unexpected message hoping for a response.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Required to determine if this msg used expected reply path.
Also remove FIXME (om->enctlv is handled above).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Found in tests/test_connection.py::test_restart_many_payments:
`lightningd: outstanding taken(): lightningd/peer_htlcs.c:532:towire_temporary_channel_failure(((void *)0), ((void *)0))`
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's actually a (very unlikely) race here: we would previously have
crashed with an assertion in invoices_resolve.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We register on it for offers, and without this nobody else can.
Changelog-Changed: plugins: more than one plugin can now register invoice_payment hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Avoids much cut & paste. Some tests don't need any of it, but most
want at least some of this infrastructure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We already do some sanity checks, add this one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: invalid UTF-8 strings now rejected.
This effectively reverts ac93b780d5.
Christian points out that plugins need time before we deprecate
the old options (probably 6 months) as they need to work with
both old and new.
Changelog-Deprecated: **UNDO** plugins: hooks should now be specified using objects, not raw names.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Incase we have been offline while a channel was force closed on us we
now set the 'closer' to 'remote' instead of null because this is by far
the most probable reason.
Changelog-None
This adds a missing plugins_send_getmanifest() call in the rescan function
that lead to a RPC hangup. Not sure though if this is the proper fix.
Changelog-None
We previously registered hooks up in who-replies-to-getmanifest-first
order, but then if any had dependencies it would scatter that order.
This allows users to manually set dependencies developers have
forgotten by specifying the plugins manually in their configuration or
cmdline. This was an excellent consideration by @mschmook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now both python and c libraries are updated, we can officially
deprecate the old form.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: plugins: hooks should now be specified using objects, not raw names.
The next patch will use these to order the hooks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: plugins: hooks can now specify that they must be called 'before' or 'after' other plugins.
This adds a `state_change` 'cause' to a channel.
A 'cause' is some initial 'reason' a channel was created or closed by:
/* Anything other than the reasons below. Should not happen. */
REASON_UNKNOWN,
/* Unconscious internal reasons, e.g. dev fail of a channel. */
REASON_LOCAL,
/* The operator or a plugin opened or closed a channel by intention. */
REASON_USER,
/* The remote closed or funded a channel with us by intention. */
REASON_REMOTE,
/* E.g. We need to close a channel because of bad signatures and such. */
REASON_PROTOCOL,
/* A channel was closed onchain, while we were offline. */
/* Note: This is very likely a conscious remote decision. */
REASON_ONCHAIN
If a 'cause' is known and a subsequent state change is made with
`REASON_UNKNOWN` the preceding cause will be used as reason, since a lot
(all `REASON_UNKNOWN`) state changes are a subsequent consequences of a prior
cause: local, user, remote, protocol or onchain.
Changelog-Added: Plugins: Channel closure resaon/cause to channel_state_changed notification
This will allow nodes (with log-level=debug) to gather how many payments
are made without payment_secrets. We need to know this so we know when
we can make them compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We let the plugin decide what feerate to accept/whether or not to add
funds to the open. To aid this decision, we also send the plugin what we
(c-lightning) currently have as our max and min acceptable feerates.
We also now use these as our default for max/min acceptable feerate
range when sending an openchannel offer to a peer.
In the future, it might be a good idea to make these more easily
changeable, either via a config setting (?) or a command param.
The previous dev-force-features forced you to explicitly declare every
desired feature bit in an array, for each set.
Here, we allow you to also denote adding/subtracing a feature bit
by just passing in the number of the bit to flip and the direction to
turn it. e.g.
'dev-force-features': '+223'
Will turn on opt_dual_fund/odd.
'dev-force-features': '-16'
Will flag off opt_basic_mpp.
"make clean" removes all object files listed in $(ALL_OBJS), which is derived from $(ALL_C_SOURCES).
Source files in $(LIGHTNINGD_SRC_NOHDR) are missing from $(ALL_C_SOURCES) therefore some object files are not removed by "make clean".
This commit fixes this point.
Signed-off-by: YOSHIDA Masanori <masanori.yoshida@gmail.com>
Changelog-None
For compatibility, we only do this if `allow-deprecated-apis` is false
for now. Otherwise scripts parsing should use `grep -v '^# '` or
start using `-N none`.
Changelog-Added: JSON-RPC: `close` now sends notifications for slow closes (if `allow-deprecated-apis`=false)
Changelog-Deprecated: cli: scripts should filter out '^# ' or use `-N none`, as commands will start returning notifications soon
Fixes: #3925
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets callers enable notifications; we won't send any if they don't.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `notifications` command to enable notifications.
And make caller of json_stream_forward_change_id use it, since
we're going to reuse that.
Also call json_out_finished here, so next object doesn't have a ","
prepended.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will allow us to build complex, multi-peer transactions, with
easeTM!
Changelog-Added: EXPERIMENTAL, Plugins: `openchannel_peer_sigs` notification, which contains a peer's signatures for the funding transaction (`opt_dual_fund`)
Prior to this patch update, we expected a client to call
`openchannel_signed` before checking for peer's tx-sigs messages on the
wire.
When moving to a 'multifundchannel' approach, we'll need to be able to
collect sigs from our peers before sending our tx_sigs message. There's
no strict ordering on when tx-sigs messages are sent/received, so this
is fine.
To do this, we go ahead and start up channeld as soon as
commitment_sigs are secured, so that we process incoming tx-sigs from
our peers as soon as we get them.
This is handy/necessary for getting multifundchannel to work, as we need
to know what output to tell all the other peers about.
Changelog-Added: Experimental!! JSON-RPC: openchannel_init returns a field `funding_serial` that indicates the serial_id of the funding output in the provided PSBT
There's a few structs/wire calls that only exist under experimental features.
These were in a common file that was shared/used a bunch of places but
this causes problems. Here we move one of the problematic methods back
into `openingd`, as it's only used locally and then isolate the
references to the `witness_stack` in a new `common/psbt_internal` file.
This lets us remove the iff EXP_FEATURES inclusion switches in most of
the Makefiles.
Rusty pointed out that having an empty channel_id is suboptimal; adding
another call is probably the right idea rather than re-using an existing
one.
Suggested-By: @rustyrussell
...right time.
We re-send the tx_sigs on start/init/reconnect until we've gotten a
funding_locked from our peer. We also build it in channeld now, instead
of in dualopend, and don't pass in a message for them anymore
`openchannel_signed` and `openchannel_update` which allow a user to
continue a openchannel or kick off the completion of a openchannel.
`openchannel_update` should be called until it returns with
`commitments_secured`.
`openchannel_signed` commands hang out across the openingd/channeld
boundary -- we don't return until we've successfully broadcast the
transaction (or timed out waiting for them to send a tx_sigs back).
We need the PSBT to create the finalized tx from once the peer's
tx_signatures are received. Since we're passing the PSBT, we no longer
need the secondary message to be passed, as it was derived from the
PSBT.
Also removes now unused witness serialization code
There are 3 commands for opening a channel with dualfunding.
`openchannel_init` is the first of these.
It initializes the open-channel dialog, and stops once we've run out of
updates (input/outputs) to send to the peer.
Great report from whitslack on this crash at startup:
```
2020-10-07T13:03:21.419Z **BROKEN** lightningd: FATAL SIGNAL 6 (version 0.9.1)
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: common/daemon.c:51 (crashdump) 0x559fb67bcc76
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0x7f61cdca8baf
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: ../sysdeps/unix/sysv/linux/raise.c:50 (__GI_raise) 0x7f61cdca8b31
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/stdlib/abort.c:79 (__GI_abort) 0x7f61cdc92535
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/assert/assert.c:92 (__assert_fail_base) 0x7f61cdc9241e
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: /var/tmp/portage/sys-libs/glibc-2.32-r2/work/glibc-2.32/assert/assert.c:101 (__GI___assert_fail) 0x7f61cdca1241
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: lightningd/subd.c:750 (subd_send_msg) 0x559fb67a1c31
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: lightningd/subd.c:745 (subd_send_msg) 0x559fb67a1c31
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:252 (local_fail_in_htlc) 0x559fb6798f77
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: lightningd/peer_htlcs.c:1441 (onchain_failed_our_htlc) 0x559fb6798f77
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:339 (handle_missing_htlc_output) 0x559fb6786b9d
2020-10-07T13:03:21.419Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:455 (onchain_msg) 0x559fb6786b9d
```
The problem is a channel with an onchaind can be in state FUNDING_STATE_SEEN,
because onchaind has started but not responded to init yet (which it does once it
has analyzed the commitment tx).
Channel B is onchain, and its onchaind fails the HTLC, and we try to send a msg
to channel A's onchaind as if it were channeld.
Explicitly check if it's channeld, rather than trying to see if it's onchaind.
Fixes: #4114
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: crash: assertion fail at restart when source and destination channels of an HTLC are both onchain.
The 100MB log buffer has been the biggest memory footprint for the daemon.
Keeping 10MB for emergency log dumps seems sufficient.
This has been mentioned in the last developer meeting.
Changelog-Changed: In-memory log buffer reduced from 100MB to 10MB
I got a corrupt file, which looked like multiple concurrent attempts
to build it. So instead, build it in one command, but also use
VERBOSE so we print correctly with V=1 (and --quiet).
Also move into plugins/ where it logically belongs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We force use of tal_wally_start/tal_wally_end around every wally
allocation, and with "end" make the caller choose where to reparent
everything.
This is particularly powerful where we allocate a tx or a psbt: we
want that tx or psbt to be the parent of the other allocations, so
this way we can reparent the tx or psbt, then reparent everything
else onto it.
Implementing psbt_finalize (which uses a behavior flag antipattern)
was tricky, so I ended up splitting that into 'psbt_finalize' and
'psbt_final_tx', which I think also makes the callers clearer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
wallet_commit_channel would fill in the old_remote_per_commit and
fee_states, which is weird since the caller doesn't care.
Make the caller set all the channel_info fields, so wallet_commit_channel
is a simple consumer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I mistakenly assumed the block would be freed after processing completed. That
is not true since chaintopology keeps headers and stubs around for reorgs. So
we need to remove the precomputed txids along with the full_txs.
1. Rename memleak_enter_allocations to memleak_find_allocations.
2. Unify scanning for pointers into memleak_remove_region / memleak_remove_pointer.
3. Document the functions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We promised this in 0.8.1!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Plugin: Relative plugin paths are not relative to startup (deprecated v0.7.2.1)
Changelog-Removed: JSON API: The hook `db_write` can no longer return `true` (deprecated in 0.8.1)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is simple, and we now can multifundchannel to every node on testnet
(one simply hangs once we connect).
Changelog-Fixed: Protocol: We now hang up if peer doesn't respond to init message after 60 seconds.
Changelog-Added: JSON-RPC: delpay a new method to delete the payment completed or failed.
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
This notification will be raised whenever a channel state changes.
The payload includes the channel and peer identifiers and the
old and the new state.
Example payload:
```
{
"channel_state_changed": {
"peer_id": "03bc9337c7a28bb784d67742ebedd30a93bacdf7e4ca16436ef3798000242b2251",
"channel_id": "a2d0851832f0e30a0cf778a826d72f077ca86b69f72677e0267f23f63a0599b4",
"short_channel_id" : "561820x1020x1",
"old_state": "CHANNELD_NORMAL",
"new_state": "AWAITING_UNILATERAL"
}
}
```
Changelog-Added: Plugins: channel_state_changed notification
As the cmd gets freed on a received error, the node id in which we iterate in `process_check_funding_broadcast`
may gets freed while we are using it.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Fixing the following error by changing 'enum feerate' to int.
lightningd/bitcoind.c:183:29: error: result of comparison of constant
8 with expression of type 'enum feerate' is always true [-Werror,
-Wtautological-constant-out-of-range-compare]
for (enum feerate f = 0; f < NUM_FEERATES; f++) {
Changelog-Fixed: compile error on macos
Greatly simplify the changeset API. Instead of 'diff' we simply generate
the changes.
Also pulls up the 'next message' method, as at some point the
interactive tx protocol will be used for other things as well
(splices/closes etc)
Suggested-By: @rustyrussell
dual funding needs the max-witness-len and utxo fields set for every
input. we should add them when we create a 'fundpsbt', so that every
psbt that c-lightning generates is dual-funding ready
v2 of channel establishment, in the accpeter case, now sends 2 messages
to our peer after saving the information to disk (our commitment
signatures and our funding transaction signatures)
v2 channel open uses a different method to derive the channel_id, so now
we save it to the database so that we dont have to remember how to
derive it for each.
includes a migration for existing channels
Too trivial a fix to really list in Changelog, but I noticed that we
specified "wumbo" twice. We should really just use the proper name
in listconfigs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: plugin: `bcli` replacements should note that `sendrawtransaction` now has a second required Boolean argument, `allowhighfees`, which if `true`, means ignore any fee limits and just broadcast the transaction. Use `--deprecated-apis` to use older `bcli` replacement plugins that only support a single argument.
We defer the notification to gossipd till the end of the spends. By itself not
a huge change, but it allows us to later migrate to doing updates blindly and
using the DB as our ground truth. It also allows us to simplify the
`wallet_outpoint_spend` semantics to not return two values in the next commit.
We're going to want this for bolt13 formation as well.
As a result of reworking the logic into "candidate selection" then
"route hint selection", we need to change the way round-robin works.
We use a simple incrementing index now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's now only needed by devtools/mkfunding, so include a reduced one
there, and this also means we remove tx_spending_utxos().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This removes the reservation cleanup at startup, too, now they're all
using 'reserved_til'.
This changes test_withdraw, since it asserted that outputs were marked
spent as soon as we broadcast a transaction: now they're reserved until
it's mined. Similarly, test_addfunds_from_block assumed we'd see funds
as soon as we broadcast the tx.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `withdraw` now randomizes input and output order, not BIP69.
This avoids overwriting the ones in git, and generally makes things neater.
We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will make the `openchannel_hook` chainable. Logic is as follows:
- The first plugin that rejects terminates the chain
- If more than one plugin uses the `close_to` parameter, take the first
value and log_unusual for the others.
Changelog-Added: openchannel_hook is now chainable
PR #3957 improved performance considerably, however we still look over the
entire message for the message separator. If instead we just look in the
incrementally read data, we remove the quadratic behavior for large messages.
This is safe since we then loop over the messages which would drain any
message separator from the buffer before we attempt the next read.
Changelog-Fixed: bcli: Significant speedups for block synchronization
