Importantly, the code in jsonrpc.c which actually does the io_break:
```
/* Once the stop_conn conn is drained, we can shut down. */
if (jcon->ld->stop_conn == conn && jcon->ld->state == LD_STATE_RUNNING) {
/* Return us to toplevel lightningd.c */
log_debug(jcon->ld->log, "io_break: %s", __func__);
io_break(jcon->ld);
```
By having the state not set until later, we avoid running this. Of course,
we need to avoid calling the main loop when we get there, if we've already
been told to shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In CI we see crashes in this case:
```
lightningd: lightningd/connect_control.c:734: void connectd_activate(struct lightningd *): Assertion `ret == ld->connectd' failed.
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CI seems to block; Christian suggests the throttler may be to blame somehow?
Since trying to fix it made it worse, let's just remove it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We always call channel_fail_transient() on all channels when a peer
connects, to clean up any previous connections. However, when
we startup, this channel doesn't have an owner yet, resulting in
a fairly weird INFO level message.
Reported-by: Michael Schmook @mschmook
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: `lightningd`: don't log gratuitous "Peer transient failure" message on first connection after restart.
There are cases (difficult to reproduce with a test) where
a payment will fail one time and succeed later.
As far I understand in this case the groupid field of the payment
is the same, and the only thing that change is the status, so
our logic inside the delpay is ambiguous where it is not
possible to delete a payment as described in https://github.com/ElementsProject/lightning/issues/6114
A sequence of commands that explain the problem is
```
$ lc -k listpays payment_hash=H
{
"pays": [
{
"bolt11": "I",
"destination": "redacted",
"payment_hash": "H",
"status": "complete",
"created_at": redacted,
"completed_at": redacted,
"preimage": "P",
"amount_msat": "redacted",
"amount_sent_msat": "redacted"
}
]
}
$ lc delpay H complete
{
"code": 211,
"message": "Payment with hash H has failed status but it should be complete"
}
```
In this case, the delpay is not able to delete a payment because the
listpays is returning only the succeeded one, so by running the
listsendpays we may see the following result where our delpay logic
will be stuck because it works to ensure that all the payments stored
in the database has the status specified by the user
```
➜ VincentSSD clightning --testnet listsendpays -k payment_hash=7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4
{
"payments": [
{
"id": 322,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 1,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 1664,
"created_at": 1679510203,
"completed_at": 1679510205,
"status": "failed",
"bolt11": "lntb1pjpkj4xsp52trda39rfpe7qtqahx8jjplhnj3tatxy8rh6sc6afgvmdz7n0llspp50lr5hmdm0re0xvcp2hv3nf2wwvx0r8q3h3e7jmqz0awdfg6w206qdp0w3jhxarfdenjqargv5sxgetvwpshjgrzw4njqun9wphhyaqxqyjw5qcqp2rzjqtp28uqy77te96ylt7ek703h4ayldljsf8rnlztgf3p8mg7pd0qzwf8a3yqqpdqqqyqqqqt2qqqqqqgqqc9qxpqysgqgeya2lguaj6sflc4hx2d89jvah8mw9uax4j77d8rzkut3rkm0554x37fc7gy92ws9l76yprdva2lalrs7fqjp9lcx40zuty8gca0g5spme3dup"
},
{
"id": 323,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 2,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 3663,
"created_at": 1679510205,
"completed_at": 1679510207,
"status": "failed"
},
{
"id": 324,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 3,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 3663,
"created_at": 1679510207,
"completed_at": 1679510209,
"status": "failed"
},
{
"id": 325,
"payment_hash": "7fc74bedbb78f2f3330155d919a54e730cf19c11bc73e96c027f5cd4a34e53f4",
"groupid": 1,
"partid": 4,
"destination": "030b686a163aa2bba03cebb8bab7778fac251536498141df0a436d688352d426f6",
"amount_msat": 300,
"amount_sent_msat": 4663,
"created_at": 1679510209,
"completed_at": 1679510221,
"status": "complete",
"payment_preimage": "43f746f2d28d4902489cbde9b3b8f3d04db5db7e973f8a55b7229ce774bf33a7"
}
]
}
```
This commit solves the problem by forcing the delete query in the
database to specify status too, and work around this kind of
ambiguous case.
Fixes: f52ff07558 (lightningd: allow delpay to delete a specific payment.)
Reported-by: Antoine Poinsot <darosior@protonmail.com>
Link: https://github.com/ElementsProject/lightning/issues/6114
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Co-Developed-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: delpay be more pedantic about delete logic by allowing
delete payments by status directly on the database.
pubkey_from_hexstr() was failing, which we didn't notice because we
weren't checking the return value. The problem was that we were passing
it a strlen that was half the actual length.
Relevant error:
[libsecp256k1] illegal argument: !secp256k1_fe_is_zero(&ge->x)
==417723== ERROR: libFuzzer: deadly signal
#7 0x7f5deaacc7fb in abort
#8 0x51b0b0 in secp256k1_default_illegal_callback_fn secp256k1.c
#9 0x51bd8e in secp256k1_ec_pubkey_serialize
#10 0x4e235b in pubkey_to_der bitcoin/pubkey.c:29:7
#11 0x4e2941 in pubkey_cmp bitcoin/pubkey.c:89:2
#12 0x4e333d in bitcoin_redeem_2of2 bitcoin/script.c:144:6
#13 0x4f1396 in run tests/fuzz/fuzz-close_tx.c:78:19
Changelog-Added: JSON-RPC: `listclosedchannels` to show old, dead channels we previously had with peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
PSBTv2 support is quite low in the ecosystem, so having a call to convert
log messages and the like should be useful since they'll often be in v2.
Changelog-Added: Added setpsbtversion RPC to aid debugging and compatibility
Libwally update breaks compatibility, so
we do this in one large step.
Changelog-Changed: JSON-RPC: elements network PSET now only supports PSETv2.
Changelog-Added: JSON-RPC: PSBTv2 supported for fundchannel_complete, openchannel_update, reserveinputs, sendpsbt, signpsbt, withdraw and unreserveinputs parameter psbt, openchannel_init and openchannel_bump parameter initialpsbt, openchannel_signed parameter signed_psbt and utxopsbt parameter utxopsbt
The issue is that common_setup() wasn't called by the fuzz target,
leaving secp256k1_ctx as NULL.
UBSan error:
$ UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1" \
./fuzz-channel_id crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1153355603
INFO: Loaded 1 modules (25915 inline 8-bit counters): 25915 [0x563bae7ac3a8, 0x563bae7b28e3),
INFO: Loaded 1 PC tables (25915 PCs): 25915 [0x563bae7b28e8,0x563bae817c98),
./fuzz-channel_id: Running 1 inputs 1 time(s) each.
Running: crash-1575b41ef09e62e4c09c165e6dc037a110b113f2
bitcoin/pubkey.c:22:33: runtime error: null pointer passed as argument 1, which is declared to never be null
external/libwally-core/src/secp256k1/include/secp256k1.h:373:3: note: nonnull attribute specified here
#0 0x563bae41e3db in pubkey_from_der bitcoin/pubkey.c:19:7
#1 0x563bae4205e0 in fromwire_pubkey bitcoin/pubkey.c:111:7
#2 0x563bae46437c in run tests/fuzz/fuzz-channel_id.c:42:3
#3 0x563bae2f6016 in LLVMFuzzerTestOneInput tests/fuzz/libfuzz.c:23:2
#4 0x563bae20a450 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long)
#5 0x563bae1f4c3f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long)
#6 0x563bae1fa6e6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))
#7 0x563bae223052 in main (tests/fuzz/fuzz-channel_id+0x181052) (BuildId: f7f56e14ffc06df54ab732d79ea922e773de1f25)
#8 0x7fa7fa113082 in __libc_start_main
#9 0x563bae1efbdd in _start
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior bitcoin/pubkey.c:22:33 in
While the user trying to fetch an invoice by specifing the quantity we do
not work as expected.
Running the command
```
lightning-cli fetchinvoice -k offer='lno1qgsqvgnwgcg35z6ee2h3yczraddm72xrfua9uve2rlrm9deu7xyfzrcgqffqszsk2p6hycmgv9ek2grpyphxjcm9ypmkjer8v46pyzmhd9jxwet5wvhxxmmdzsqs593pq0ylsvakdua5h976f4g3eautgjt3udvtyga47eaw7339sjrhpwpwz' quantity=2
```
and we answer back with
```json
{
"code": -32602,
"message": "quantity parameter required"
}
```
This is caused because we forget to bind the `quanity` field from the
RPC into the `invrequest`.
Reported-by: @aaronbarnardsound
Link: https://github.com/ElementsProject/lightning/issues/6089
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-EXPERIMENTAL: fetchinvoice: fix: do not ignore the `quantity` field
into the invreq field.
It seems that bitcoind frequently dies on this test. I assume running
the multiple nodes under valgrind with the extra 214 blocks is too
memory-hungry?
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: `checkmessage` now always returns an error when the pubkey is not specified and it is unknown in the network graph (deprecated v0.12.0)
Commit a418615b7f ("rpc: adds
num_channels to listpeers") broke the sql tests. Turns out, no
openchannel v2 tests are run in CI!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It works for the trivial case, where groupid and partid are the same,
but silently deletes nothing in the other cases (or worse, deletes the
wrong entry!).
See: #5835
Changelog-Fixed: `delpay`: actually delete the specified payment (mainly found by `autoclean`).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we had two things to clean, we fired off two requests (eg.
listforwards and listinvoices) and both marked the timer as finished,
triggering an assert.
We already have a refcount for outstanding requests to avoid this
for e.g. outstanding del commands, so use it here too!
```
Jan 19 19:20:00 lightningd[748044]: autoclean: plugins/libplugin.c:445: timer_complete: Assertion `p->in_timer > 0' failed.
Jan 19 19:20:00 lightningd[748044]: autoclean: FATAL SIGNAL 6 (version v22.11.1)
Jan 19 19:20:00 lightningd[748044]: 0x562c388136e4 send_backtrace
Jan 19 19:20:00 lightningd[748044]: common/daemon.c:33
Jan 19 19:20:00 lightningd[748044]: 0x562c3881376c crashdump
Jan 19 19:20:00 lightningd[748044]: common/daemon.c:46
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0898d5f ???
Jan 19 19:20:00 lightningd[748044]: ./signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0898ce1 __GI_raise
Jan 19 19:20:00 lightningd[748044]: ../sysdeps/unix/sysv/linux/raise.c:51
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0882536 __GI_abort
Jan 19 19:20:00 lightningd[748044]: ./stdlib/abort.c:79
Jan 19 19:20:00 lightningd[748044]: 0x7f26d088240e __assert_fail_base
Jan 19 19:20:00 lightningd[748044]: ./assert/assert.c:92
Jan 19 19:20:00 lightningd[748044]: 0x7f26d0891661 __GI___assert_fail
Jan 19 19:20:00 lightningd[748044]: ./assert/assert.c:101
Jan 19 19:20:00 lightningd[748044]: 0x562c3880355d timer_complete
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:445
Jan 19 19:20:00 lightningd[748044]: 0x562c38800b54 clean_finished
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:122
Jan 19 19:20:00 lightningd[748044]: 0x562c388010ed clean_finished_one
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:132
Jan 19 19:20:00 lightningd[748044]: 0x562c388011b6 del_done
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:149
Jan 19 19:20:00 lightningd[748044]: 0x562c388058b5 handle_rpc_reply
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:768
Jan 19 19:20:00 lightningd[748044]: 0x562c38805a39 rpc_read_response_one
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:944
Jan 19 19:20:00 lightningd[748044]: 0x562c38805ad7 rpc_conn_read_response
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:968
Jan 19 19:20:00 lightningd[748044]: 0x562c38876b60 next_plan
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/io.c:59
Jan 19 19:20:00 lightningd[748044]: 0x562c38876fe7 do_plan
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/io.c:407
Jan 19 19:20:00 lightningd[748044]: 0x562c38877080 io_ready
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/io.c:417
Jan 19 19:20:00 lightningd[748044]: 0x562c3887823c io_loop
Jan 19 19:20:00 lightningd[748044]: ccan/ccan/io/poll.c:453
Jan 19 19:20:00 lightningd[748044]: 0x562c38805d11 plugin_main
Jan 19 19:20:00 lightningd[748044]: plugins/libplugin.c:1801
Jan 19 19:20:00 lightningd[748044]: 0x562c38801c7a main
Jan 19 19:20:00 lightningd[748044]: plugins/autoclean.c:613
```
Fixes: #5912
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When doing some plugin related work, I discovered that the datastore API
has two issues:
- Error messages on startup of plugins init method when the datastore is
still completely empty: "Parsing '{datastore:[0:': token has no index 0: []"
- Data is escaped but not unwrapped again when sending and getting from
the API.
[ Removed xfail, it now passes! --RR ]
Closes: #5990
Since it's not spec-final yet (hell, it's not even properly specified
yet!) we need to put it behind an experimental flag.
Unfortunately, we don't have support for doing this in a plugin; a
plugin must present features before parsing options. So we need to do
it in core.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Though there's already a `createinvoice` command, there are usecases where a
user may want to sign an invoice that they don't yet have the preimage to. For
example, they may have an htlc_accepted plugin that pays to obtain the preimage
from someone else and returns a `{ "result": "resolve", ... }`.
This RPC command addresses this usecase without overly complicating the
semantics of the existing `createinvoice` command.
Changelog-Added: JSON-RPC: `signinvoice` new command to sign BOLT11
invoices.
We might be disconnected, but the subd isn't dead yet:
```
> assert out[0] == '# peer is offline, will negotiate once they reconnect (5 seconds before unilateral close).'
E AssertionError: assert '# Timed out, forcing close.' == ('# peer is offline, will negotiate once they reconnect (5 seconds before '\n 'unilateral close).')
E - # peer is offline, will negotiate once they reconnect (5 seconds before unilateral close).
E + # Timed out, forcing close.
tests/test_closing.py:164: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
make the number of blocks mined father away from the cltv timeout
from borked/flakey test run:
lightningd-3 2023-01-26T21:45:19.261Z DEBUG 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-channeld-chan#2: billboard perm: Received error channel 27a4a4dd880e86
1e390517de3e786a237c5ad1f00faab277382664e76b5c3870: Fulfilled HTLC 0 SENT_REMOVE_COMMIT cltv 116 hit deadline
When a channel open fails, we use tx-abort instead of warning/error.
This means that the peer won't disconnect! And instead when a new
message arrives, we'll need to rebuild the dualopend subd (if missing).
Makes opens a bit easer to retry (no reconnect needed), as well as keeps
the connection alive for other channels we may have with that peer.
Changelog-Changed: Experimental-Dual-Fund: open failures don't disconnect, but instead fail the opening process
Dropping the connection is bad behavior on an openchannel failure,
especially given that there might be other channels currently connected.
We should maintain the connection but close out the dualopend
daemon for that attempt.
This test partially documents the behaivor, but fails
Changelog-None
v2 opens require you to use native segwit inputs
Changelog-Added: JSONRPC: `upgradewallet` command, sweeps all p2sh-wrapped outputs to a native segwit output
1. Allow 'any' as an option to zeroconf-selective.py plugin so we can use
it in line_graph where we don't know ids yet.
2. Use modern helpers like line_graph and remove debugging statement.
3. Don't use listchannels(): it's true that it shows local channels as well,
but that's a quirk I'd like to remove.
4. Make flake8 happy.
5. Rename to be more specific now it's a more narrow test.
I manually tested that the test still failed with the fixes removed, too,
so it is still the same test!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we only specify the node_id, we get the "first" channel.
Closes: #5803
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` uses the correct local channel for payments when there are multiple available (not just always the first!)
We fixed most of them. Now hone in to the case which fails: `pay`
when it needs to use the direct zero-conf channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Modifications from issue #5803 to work here:
1. import json
2. Add xfail
3. Increase channel sizes by 10x so we can open them
4. Fix plugin path
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can actually catch l2 with HTLCs still closing and mine blocks,
then it force closes due to HTLC timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that our subtables were not being cleaned, despite being "ON
DELETE CASCADE". This is because foreign keys were not enabled, but
then I got foreign key errors: rowid cannot be a foreign key anyway!
So create a real "rowid" column. We want "ON DELETE CASCADE" for
nodes and channels (and other tables in future) where we update
partially.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to check if the key parameter is an empty array in
`listdatastore` as we do assume an array of at least length 1 in
`wallet.c:5306`.
Signed-off-by: Peter Neuroth <pet.v.ne@gmail.com>
This *would* be a 1-line change (add it to Makefile) except that we
previously assumed a "list" prefix on commands.
These use the default refreshing, but they could be done better using
the time-range parameters.
Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For now, we ignore every deprecated field, but put in the logic so
that future deprecations will work as expected.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This requires us to rename "index" fields, rename fields if we have a
sub-object, and create sub-tables if we have an array, and handle the
fact that some listX commands don't contain array X (listsendpays
contains "payments").
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a core concept in the spec which isn't directly exposed.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
This adds the option to explicitly enable ip-discovery, which maybe
helpful for example when a user wants TOR announced along with
discovered IPs to improve connectivity and have TOR just as a fallback.
Changelog-Added: Adds config switch 'announce-addr-discovered': on/off/auto
The close call can fail, since we already unilaterally closed since we mined blocks
too fast:
```
2023-01-14T01:00:10.2502199Z E pyln.client.lightning.RpcError: RPC call failed: method: close, payload: ['107x1x1', None, 'bcrt1qeyyk6sl5pr49ycpqyckvmttus5ttj25pd0zpvg'], error: {'code': -32602, 'message': "Short channel ID not active: '107x1x1'"}
...
2023-01-14T01:00:10.5288050Z lightningd-4 2023-01-14T00:59:59.650Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Peer permanent failure in CHANNELD_NORMAL: Fulfilled HTLC 0 SENT_REMOVE_COMMIT cltv 113 hit deadline
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to create some p2sh-segwit addresses just to mix things up. This
streamlines back to just bech32.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
With the next change (which, as a side-effect, speeds up listpeers),
we seem to hit a race in this test. The bookkeeper doesn't get to
process the final payment before the node is shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's no guarantee as to iteration order for accounts/channels, but
this test was relying on them.
Adding account attribution and comparing by account_ids fixes
Fixes: #5869
Reported-By: @rustyrussell
1. When we receive a commando command from a remote using the `filter`
field, use it.
2. Add a `filter` parameter to `commando` to send it: this is usually
more efficient than using filtering locally.
Of course, older remote nodes will ignore the filter, but that's
harmless.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `commando` now supports `filter` as a parameter (for send and receive).
This was reported a while ago: now do it properly.
Fixes: #5637
Changelog-Fixed: Plugins: `commando` now responds to remote JSON calls with the correct JSON `id` field.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We change the libplugin API so commando can provide its own ID base.
This id chaining enables much nicer diagnostics!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We don't do this yet, so we add deprecated to those test (until next
patch!).
Changelog-Deprecated: plugins: `commando` JSON commands without an `id` (see doc/lightningd-rpc.7.md for how to construct a good id field).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The top of the file indicates the following errors:
#define NO_ERROR 0
#define ERROR_FROM_LIGHTNINGD 1
#define ERROR_TALKING_TO_LIGHTNINGD 2
#define ERROR_USAGE 3
But we didn't use the right one for opt_parse failure, and didn't use the
correct constants everywhere.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids any confusion between primitive and string ids, and in
particular stops an issue with commando once it starts chaining ids,
that weird ids can be double-escaped and commando will not recognize
the response, leaving the client hanging. It's the client's fault for
using a weird id, but it's still rude (and triggered by our tests!).
It also makes substituting the id in passthrough simpler, FTW.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Issue #5363 documented an earlier bug in mkfunding. These tests validate that fix as well as checking that basic usage produces the proper response and exit status.
Changelog-None
This broke BTCPayServer, so revert. I originally (accidentally!)
implemented this such that it broadcast both DNS and IP entries, but
Michael reported earlier that they still don't propagage well, so
simply suppress them.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5795
Changelog-Changeed: Config: `announce-addr-dns` needs to be set to *true* to put DNS names into node announcements, otherwise they are suppressed.
Changelog-Deprecated: Config: `announce-addr-dns` (currently defaults to `false`). This will default to `true` once enough of the network has upgraded to understand DNS entries.
If we both support large channels, we can actually send giant HTLCs.
This, in turn, fixes pay (which relies on this field!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `pay` now knows it can use locally-connected wumbo channels for large payments.
Fixes: #5250Fixes: #5417
We have a primary key that is spanning the `in_channel_id` and the
`in_htcl_id`. The latter gets set to NULL when the HTLC and channel
gets deleted, so we coalesce with a random large number that is
unlikely to collide for the primary key.
This is a minimal fix: we wait until all plugins reply from init before
continuing. Really large or busy nodes can have other things monopolize
lightningd, then the timer goes off and we blame the plugin (which has
responded, we just haven't read it yet!).
The real answer is to have some timeouts only advance when we're idle,
or have them low-priority so we only activate them when we're idle (this
doesn't apply to all timers: some are probably important!). But
this is a minimal fix for -rc3.
Fixes: https://github.com/ElementsProject/lightning/issues/5736
Changelog-Fixed: plugins: on large/slow nodes we could blame plugins for failing to answer init in time, when we were just slow.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
On testnet I noticed if we can't reach bitcoind for some reason, we'll
keep RBFing our penalty tx ("it didn't go in, RBF harder!!"). Makes
no sense to grossly exceed the amount needed for next block, so simply
cap penalty at 2x "estimatesmartfee 2 CONSERVATIVE".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were missing the OP_PUSH for the pubkeys, and the spec mentions we
should be using 73 bytes to estimate the witness weight. Effectively
this adds 4 bytes which really just matters in case fees hit the
floor, and computing the weight becomes important.
Changelog-Fixed: onchaind: Witness weight estimations could be slightly lower than the VLS signer
This is how we put new invoice_requests into the db; this will be used
by a new "invoicerequest" command which replaces "offerout".
The API is now the same as the offers api.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We no longer use offers for "I want to send you money", but we'll use
invoice_requests directly. Create a new table for them, and
associated functions.
The "localofferid" for "pay" and "sendpay" is now "localinvreqid".
This is an experimental-only option, so document the change under
experimental only.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: JSON-RPC: `pay` and `sendpay` `localofferid` is now `localinvreqid`.
I know this is an unforgivably large diff, but the spec has changed so
much that most of this amounts to a rewrite.
Some points:
* We no longer have "offer_id" fields, we generate that locally, as all
offer fields are mirrored into invoice_request and then invoice.
* Because of that mirroring, field names all have explicit offer/invreq/invoice
prefixes.
* The `refund_for` fields have been removed from spec: will re-add locally later.
* quantity_min was removed, max == 0 now mean "must specify a quantity".
* I have put recurrence fields back in locally.
This brings us to 655df03d8729c0918bdacac99eb13fdb0ee93345 ("BOLT 12:
add explicit invoice_node_id.")
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The "path" is just a message to ourselves. This meets the minimal
requirement for bolt12 invoices: that there be a blinded path (at
least so we can use the path_id inside in place of "payment_secret").
We expose the method to make this path_id to a common routine: offers
will need this for generating more sophisticated paths.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `listfunds` now lists coinbase outputs as 'immature' until they're spendable
Changelog-Changed: JSON-RPC: UTXOs aren't spendable while immature
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: LightningRpc has new `reply_filter` context manager for reducing output of RPC commands.
We suppress schema reply checking when filter is set: we could just
remove all the `required` fields in the JSON schema.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No tests currently use it, and if they do we'll want to do some
per-test objects. Otherwise, we are about it introduce a dependency
on common/json_filter.o, which is a can of worms.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Remove the very concept of ONION_REPLY_SIZE, instead make it a
local variable in create_onionreply().
2. Use the proper fromwire_ primitives in unwrap_onionreply() so we
don't have to do explicit length checks.
3. Make fromwire_tal_arrn() return NULL if it fails to pull, instead of
a zero-length allocation.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: we now correctly decrypt non-256-length onion errors (we always forwarded them fine, now we actually can parse them).
This is because JSON technically does not allow numeric keys in maps.
Changelog-Added: JSON-RPC: The `extratlvs` argument for `keysend` now allows quoting the type numbers in string
It's 2b7ad577d7a790b302bd1aa044b22c809c76e49d, which reverts the
point32 changes.
It also restores send_invoice in `invoice`, which we had removed
from spec and put into the recurrence patch.
I originally had implemented compatibility, but other changes
which followed this are far too widespread.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: offers: complete rework of spec from other teams (yay!) breaks previous compatibility (boo!)
It was very tied to x-only keys; we could support it in a backwards
compatibility mode for a while, but getting refunds or proving old
pre-finalization invoices is not worth spending time on.
Changelog-EXPERIMENTAL: offers: old `payer_key` proofs won't work.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use the saved previous outputs (plus maybe some new ones?) to build a
psbt for an RBF request.
RBFs utxo reuse is now working so we can unfail the test (and update
it to reflect that the lease sticks around through an RBF cycle).
Changelog-Fixed: Plugins: `funder` now honors lease requests across RBFs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: `keysend` now removes unknown even (technically illegal!) fields, to try to accept more payments.
The goal here is to test the node validation, not whether we can
trigger the schema validation with bogus values. So we bypass the
verifying RPC wrapper.
"sphinx_add_hop" takes a literal hop to include,
"sphinx_add_modern_hop" prepends the length. Now we always prepend a
length, make it clear that the literal version is a shortcut:
* sphinx_add_hop -> sphinx_add_hop_has_length
* sphinx_add_modern_hop -> sphinx_add_hop
In addition, we check that length is actually correct! This means
`createonion` can no longer create legacy or otherwise-invalid onions:
fix tests and update man page to remove legacy usage.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: `createonion` no longer allows non-TLV-style payloads.
We still have an "enum forward_style" for the database, where old-style
forwards can still exist.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer forward HTLCs with legacy onions.
We used to ensure the l3<->l4 channel was private by
simply not mining enough blocks to announce. Then
we started mining 13 blocks to close the channel, and
it will get announced, causing a failure:
```
assert non_public(l2) == []
> wait_for(lambda: non_public(l3) == [scid34, scid34])
...
> raise ValueError("Timeout while waiting for {}", success)
E ValueError: ('Timeout while waiting for {}', <function test_gossip_persistence.<locals>.<lambda> at 0x7f6cc69b4170>)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Second pay can fail if first is not completely settled:
```
def test_zeroconf_forward(node_factory, bitcoind):
"""Ensure that we can use zeroconf channels in forwards.
...
# Make sure (esp in non-dev-mode) blockheights agree so we don't WIRE_EXPIRY_TOO_SOON...
sync_blockheight(bitcoind, [l1, l2, l3])
inv = l3.rpc.invoice(42 * 10**6, 'inv1', 'desc')['bolt11']
l1.rpc.pay(inv)
# And now try the other way around: zeroconf channel first
# followed by a public one.
wait_for(lambda: len(l3.rpc.listchannels()['channels']) == 4)
inv = l1.rpc.invoice(42, 'back1', 'desc')['bolt11']
> l3.rpc.pay(inv)
...
> raise RpcError(method, payload, resp['error'])
E pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lnbcrt420p1p3junrssp588gtjzmlrr4pfj7ssmdlulzhhushrpq3rdqxjuz2m33scsvzdjlspp5fk5yhu6netc0d0sgp8es52vjk6akhd3uayr08u8max4d8rwzpjuqdq8v3jhxccxqyjw5qcqp99qyysgqcpyyugejv4nya97v6gw8fhtr0ru3vq87jjlltav99wlat436a95n0z8yzdp699p9md0zz9tmnsjpvfj622n9g9fh7r6ldhpgh9wmr4qpcru3rk'}, error: {'code': 210, 'message': 'Destination 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518 is not reachable directly and all routehints were unusable.', 'attempts': [{'status': 'failed', 'failreason': 'Destination 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518 is not reachable directly and all routehints were unusable.', 'partid': 0, 'amount_msat': 42msat}]}
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have to allow them (as otherwise `fees_collected_msat` in getinfo breaks),
but it means that actually, in_htlc_id might be missing in listforwards
(also, out_htlc_id might be missing, which we didn't catch before).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5628
Otherwise what the hook sees is actually a lie, and if it sets it
we might override it.
The side effect is that we add an explicit "forward_to" field, and
allow hooks to override it. This lets a *hook* control channel
choice explicitly.
Changelod-Added: Plugins: `htlc_accepted_hook` return can specify what channel to forward htlc to.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the minimal change to meet the desired outcome of https://github.com/lightning/bolts/issues/934
which wants to give obsolete-db nodes a chance to fix things up, before we
close the channel.
We need to dance around a bit here, since we *will* close the channel if
we receive an ERROR, so we suppress that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds a new "chan_dying" message to the gossip_store, but since we
already changed the minor version in this PR, we don't bump it again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: We now delay forgetting funding-spent channels for 12 blocks (as per latest BOLTs, to support splicing in future).
We will now simply reject old-style ones as invalid. Turns out the
only trace we could find is a channel between two nodes unconnected to
the rest of the network.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now require all channel_update messages include htlc_maximum_msat (as per latest BOLTs)
If they really upgrade directly from 0.9.2, it will simply delete the
store and re-fetch it.
We still update from v9 (which could be v0.11), since it's a noop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If channeld hasn't exited yet, it's possible we'll send the message (we would fail later, in
waitsendpay, but just not immediately). So wait for that explicitly.
```
2022-09-22T22:49:59.6737296Z with pytest.raises(RpcError, match=r'failed: WIRE_TEMPORARY_CHANNEL_FAILURE \(First peer not ready\)'):
2022-09-22T22:49:59.6737566Z > l1.rpc.sendpay(route, rhash, payment_secret=inv['payment_secret'])
2022-09-22T22:49:59.6737865Z E Failed: DID NOT RAISE <class 'pyln.client.lightning.RpcError'>
2022-09-22T22:49:59.6737873Z
```
And from the listpeers output, ou can see "connected" false, but owner channeld:
```
2022-09-22T22:49:59.7493163Z DEBUG:root:{
2022-09-22T22:49:59.7493320Z "id": "-c:listpeers#26",
2022-09-22T22:49:59.7493397Z "result": {
2022-09-22T22:49:59.7493477Z "peers": [
2022-09-22T22:49:59.7493548Z {
2022-09-22T22:49:59.7493709Z "id": "022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59",
2022-09-22T22:49:59.7493801Z "connected": false,
2022-09-22T22:49:59.7493884Z "channels": [
2022-09-22T22:49:59.7493955Z {
2022-09-22T22:49:59.7494058Z "state": "CHANNELD_NORMAL",
2022-09-22T22:49:59.7494250Z "scratch_txid": "4b95a3b1b5e1a970401a169a3697f3a9bfbfbcb59d3d21434aa1f3fb2980db8d",
2022-09-22T22:49:59.7494365Z "last_tx_fee_msat": "7965000msat",
2022-09-22T22:49:59.7494437Z "feerate": {
2022-09-22T22:49:59.7494529Z "perkw": 11000,
2022-09-22T22:49:59.7494618Z "perkb": 44000
2022-09-22T22:49:59.7494690Z },
2022-09-22T22:49:59.7494785Z "owner": "channeld",
2022-09-22T22:49:59.7494894Z "short_channel_id": "103x1x0",
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
CI is really slow: it sees all three expire at once. But making the
timeouts too long is painful in non-VALGRIND, so I ended up making it
conditional.
```
# First it expires.
wait_for(lambda: only_one(l3.rpc.listinvoices('inv1')['invoices'])['status'] == 'expired')
# Now will get autocleaned
wait_for(lambda: l3.rpc.listinvoices('inv1')['invoices'] == [])
> assert l3.rpc.autoclean_status()['autoclean']['expiredinvoices']['cleaned'] == 1
E assert 3 == 1
tests/test_plugin.py:2975: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is actually what the autoclean plugin wants, especially since
you can't otherwise delete a payment which has failed then succeeded.
But insist on neither or both being specified, at least for now.
Changelog-Added: JSON-RPC: `delpay` takes optional `groupid` and `partid` parameters to specify exactly what payment to delete.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's more natural: we will eventually support dynamic config variables,
so this will be quite nice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Plugins: `autoclean` can now delete old forwards, payments, and invoices automatically.
And take the opportunity to rename l0 and l1 in the tests to the
more natural l1 l2 (since we add l3).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using `listfowards` for this wrong; expose this directly if people
care (and unlike listforwards, which could be deleted, we have to
remember these while the channel is still open!).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listhtlcs` new command to list all known HTLCS.
And document that we never know payment_hash.
Changelog-Added: JSON-RPC: `listforwards` now shows `in_htlc_id` and `out_htlc_id`
Changelog-Changed: JSON-RPC: `listforwards` now never shows `payment_hash`; use `listhtlcs`.
Normally, we'd use the delete_columns function to remove the old
`short_channel_id` string field, *but* we can't do that for sqlite, as
there are other tables with references to it. So add a FIXME to do
it once everyone has upgraded to an sqlite3 which has native support
for column deletion.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This one directly contains the scids of the channels involved, not
references, so can outlive the channels. As a side-effect, however,
it now never lists `payment_hash`. Having it listed (via join) is not
possible as it is a *string* in the channels table, and difficult
anyway because of channel aliases.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The old `experimental-accept-extra-tlv-types` is now `accept-htlc-tlv-types`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Config: `accept-htlc-tlv-types` lets us accept unknown even HTLC TLV fields we would normally reject on parsing (was EXPERIMENTAL-only `experimental-accept-extra-tlv-types`).
"Who needs specs?" FFS...
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `keysend` will now attach the longest valid text field in the onion to the invoice (so you can have Sphinx.chat users spam you!)
We assume that because we've told l3 to shut down, l2 already sees it
as disconnected. But CI is ...slow... today!
```
# `l3` is disconnected and we can't send messages to it
> assert(not l2.rpc.listpeers(l3.info['id'])['peers'][0]['connected'])
E assert not True
tests/test_misc.py:2218: AssertionError
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Prior to this we might end up with a commitment transaction without
any outputs, if combined with `--dev-allowdustreserve`. Otherwise the
reserve being larger than dust means the funder could not drop its
direct output to be below dust.
Reported-by: Rusty Russell <@rustyrussell>
Technically this is a non-conformance with the spec, hence the `dev`
flag to opt-in, however I'm being told that it is also implemented in
other implementations. I'll follow this up with a proposal to the spec
to remove the checks we now bypass.
They can set their name explicitly, but if they don't we extract it from argv[0].
We also set it around callbacks, so it will be expanded by default.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Build them from the command which caused them, and take plugin name
as basename with extension stripped.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
First, merge the _ahf_ and non-ahf interfaces.
Second, remove the always-NULL txs->cmd field.
Then, add optional id_prefix for bitcoind_sendrawx, so if it's
triggered by a command (e.g. "withdraw") it's shown correctly in logs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
