Commit Graph

14947 Commits

Author SHA1 Message Date
ShahanaFarooqui
7baa1e93ca docs: Fixes typo in library name flask-restx
clnrest installation instruction on the markdown tries to install `flask_restx` not `flask-restx`.

Reference: clnrest plugin complains of flask_restx missing (#7383)

Changelog-None.
2024-06-26 21:31:49 -07:00
ShahanaFarooqui
cd07ce83f7 docs: Moved python plugins after CLN instructions
Changelog-None.
2024-06-26 21:31:49 -07:00
Epic Curious
6c338d6173 fix(deps): add pytest dependency for make check 2024-06-26 21:31:49 -07:00
Epic Curious
41c766882b docs: chore: bump lightning tag in installation.md 2024-06-26 21:31:49 -07:00
Epic Curious
5c43d44800 fix(deps): add python3-pip to clnrest dependencies 2024-06-26 21:31:49 -07:00
Epic Curious
1cb749ecbb fix(deps): add libsecp256k1-dev to clnrest plugin instructions 2024-06-26 21:31:49 -07:00
Epic Curious
02262586ec fix(deps): add libffi-dev build dependency
Fix the fatal error  `ffi.h: No such file or directory` during pip3 install requirements.txt
2024-06-26 21:31:49 -07:00
ShahanaFarooqui
51e46339a7 docs: notifications schema update
Changelog-None.
2024-06-26 21:06:16 -07:00
ShahanaFarooqui
72079cc920 script: Adding Delete action for readme docs
Documents which are deleted from lightning schema were still listed on readme server.

This script was adding or updating the list but delete action was missing.

Changelog-None.
2024-06-25 12:34:07 -07:00
Rusty Russell
c30502779e CI: set elements and bitcoind versions in one place.
We have them spread everywhere: unify them, and make setup.sh use install-bitcoind.sh

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 09:36:00 +09:30
Rusty Russell
cb4dd7489c lightningd: check command should return as much detail as possible.
If they're explicitly calling "check", don't obfuscate the result!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
9fd29d35dd common: move now-enlarged command_fail_badparam into its own source file.
It's getting a bit awkward to inline now: it's non-trivial.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
f33c5188ef common: don't mention the contents of invalid parameters when in non-dev mode.
Shahana decided this was the optimal UX path, though I insisted that we still
report the actual problem directly when in dev mode, as a compromise.

Suggested-by: https://github.com/Amperstrand
Changelog-Changed: JSON-RPC: Do not return the contents of invalid parameters in error messages, refer to logs (use 'check' to get full error messages)
Fixes: https://github.com/ElementsProject/lightning/issues/7338
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
e0e38c2cd5 common: new function command_log to log something about a specific command.
Needs implementations for lightningd and libplugin, since they both use this
infrastructure.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
3531414d18 lightningd: rename command_log() to command_logger()
It doesn't actually log, just gets the `struct logger`, so this name is better.

We're also going to implement command_log() as an actual logging
function in next commit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
8288603bab pytest: fix flake in test_emergencyrecover
```
2024-06-24T05:09:32.5233603Z         assert l1.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234187Z >       assert l2.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234917Z E       AssertionError: assert 'FUNDING_SPEND_SEEN' == 'ONCHAIN'
2024-06-24T05:09:32.5235464Z E         - ONCHAIN
2024-06-24T05:09:32.5235773Z E         + FUNDING_SPEND_SEEN
2024-06-24T05:09:32.5236096Z 
2024-06-24T05:09:32.5236242Z tests/test_misc.py:2907: AssertionError
```

It's possible that l2 hasn't seen the onchain tx yet.  What we really want to do is
wait for it, then test that l1 really can spend the output it has recovered.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
f642e4e551 pytest: fix test_feerate_stress race
```
2024-06-24T05:14:14.9939033Z >       l1.rpc.call('dev-feerate', [l2.info['id'], rate - 5])
2024-06-24T05:14:14.9939354Z 
2024-06-24T05:14:14.9939466Z tests/test_connection.py:3439: 
...
2024-06-24T05:14:14.9967617Z >           raise RpcError(method, payload, resp['error'])
2024-06-24T05:14:14.9968833Z E           pyln.client.lightning.RpcError: RPC call failed: method: dev-feerate, payload: ['022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 2290], error: {'code': -1, 'message': 'Peer bad state'}
```

The disconnect can actually take a while: wait for both sides to
believe they're reconnected.  Also wait a little to make sure the
feerate change doesn't create a bad signature.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
bb64fc8ddc pytest: fix flake in test_feerate_stress.
If randomly l1done or l2done are zero, we will create invalid hex:

```
with pytest.raises(RpcError, match='WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS|WIRE_TEMPORARY_CHANNEL_FAILURE'):
>           l2.rpc.waitsendpay("{:064x}".format(l2done - 1), timeout=TIMEOUT)

tests/test_connection.py:3438: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
contrib/pyln-client/pyln/client/lightning.py:1361: in waitsendpay
    return self.call("waitsendpay", payload)
contrib/pyln-testing/pyln/testing/utils.py:715: in call
    schemas[0].validate(testpayload)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Validator(schema={'pairedWith': [['partid', 'groupid']], 'properties': {'groupid': {'description': ['Grouping key...pa... {'description': ['A timeout in... the payment.'], 'type': 'u32'}}, 'required': ['payment_hash']}, format_checker=None)
args = ({'payment_hash': '-000000000000000000000000000000000000000000000000000000000000001', 'timeout': 180},)
kwargs = {}
error = <ValidationError: "'-000000000000000000000000000000000000000000000000000000000000001' is not of type 'hash'">

    def validate(self, *args, **kwargs):
        for error in self.iter_errors(*args, **kwargs):
>           raise error
E           jsonschema.exceptions.ValidationError: '-000000000000000000000000000000000000000000000000000000000000001' is not of type 'hash'
E           
E           Failed validating 'type' in schema['properties']['payment_hash']:
E               {'description': ['The hash of the *payment_preimage*.'], 'type': 'hash'}
E           
E           On instance['payment_hash']:
E               '-000000000000000000000000000000000000000000000000000000000000001'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
fd9975c288 reckless: don't use time-based directories.
This always fails on my test machine, which runs many tests in parallel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
b748c1845c pytest: fix flake in test_closing.py::test_penalty_htlc_tx_timeout
lightningd-3 penalizes lightningd-2 but then it can see the preimage for the HTLC which
has already been timed out:

```
2024-06-24T02:41:29.4633900Z lightningd-3 2024-06-24T02:33:54.073Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-onchaind-chan#1: HTLC already resolved by THEIR_HTLC_TIMEOUT_TO_THEM when we found preimage
```

This is fair: the test deliberately takes l3 offline for long enough
that the HTLC can get timed out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
80cde517c9 lightningd: fix race in initialization.
Found by very slow CI: the two io_breaks() can combine into one, so we
block waiting for the second initialization which never happens.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
9da66100c1 chaintopology: free outstanding requests so we don't get responses while shutting down.
I've never seen this, but the race looks possible so we should close it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
928b7c83fc lightningd: pass explicit context for bitcoind request functions.
That way if you free the context, you free the call (i.e. the callback
will not be called).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
492ed84dbf lightningd: move checkchain_timer out of bitcoind, remove stopped flag.
checkchain_timer is run by chaintopology, so why have the pointer in
bitcoind?

And since we free the timers, we don't need them to self-disable by
checking the stopped flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
f2b033d7e1 lightningd: move all chainheight logic into setup_topology.
Handling half in main() and half here was a mess.  And the name
"max_blockheight" was poor: it was the max in the db, or UINT32_MAX,
but then we changed it depending on what height we wanted to start at.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
ff15b943fe lightningd: remove unused "min_blockheight" from chain_topology struct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
f643fb88f5 lightningd: wait for bitcoind at startup if it's fallen behind.
Fixes: https://github.com/ElementsProject/lightning/issues/6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
59afbac7cf lightningd: remove first_call param for getchaininfo.
It was a weird arbitrary bool passed through from the caller, which we no longer need.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
eb70476fb3 lightningd: split chaintopology init from normal loops.
The current code is confusing: there are polling loops, but we wait for
them to run once. 

Be explicit: make the calls once, then start the loops in begin_topology.

This removes various chain_topology struct members which only exist for
startup.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
f318d08e94 lightningd: add arg to bitcoind_estimate_fees callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
436bf6c9e1 lightningd: pass height back from getrawblockbyheight.
Several callers stash this, but we have it, so make it explicit:
here's the block you asked for.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
ShahanaFarooqui
b803609247 docs: Removed example usage
Example usage has been either merged with examples or usage keys. This heading was not required separately.

This PR edits & rearranges Examples, Example Notifications and Example Usage data.

Changelog-None.
2024-06-22 17:02:33 +02:00
ShahanaFarooqui
e94b793f05 docs: Example notifications heading
- Added missing Example Notifications Heading
- Renamed `example_json_notifications` to `example_notifications`
- Moved `example_notifications` to the end of the page
2024-06-22 17:02:33 +02:00
ShahanaFarooqui
976c0c90bc docs: Changes examples format
- Changed ALL `doc/schemas/lightning-*.json` file's `json_example` to `examples`
- Change the heading from example to examples
- Bring shell command before the json command
- Move Example to the end of the page
- Remove horizontal line from Example
2024-06-22 17:02:33 +02:00
Christian Decker
1347e86671 wallet: Log to chan->log rather than wallet->log
Suggested-by: Rusty Russell <@rustyrussell>
2024-06-22 14:50:21 +02:00
Christian Decker
001b190c61 feat(main): Print channel status when loading it from DB
I was trying to debug a node with several multiplexed channels, and
was finding it a bit difficult to determine which channel index
matches which result of `listpeerchannels` as well as figuring out
what their status was. This just prints the status in string format
when loading the channel from the DB.

Changelog-Changed: wallet: The channel status is printed when loading it from the DB
2024-06-22 14:50:21 +02:00
Rusty Russell
e21b70cad0 pytest: fix default/description order in zeroconf test plugin.
Noticed as I was debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 17:20:06 +09:30
Christian Decker
31cf9225f4 pyln: Turn the plugin options into a real Type (uppercase T)
[ Fix not to include 'value' and 'default' (if None) in getmanifest response --RR ]
[ Fix to support [] operator for existing plugins (including our test ones!) --RR ]
2024-06-20 17:20:06 +09:30
Rusty Russell
1eb1db0e24 pyln-client: allow dynamic option setter to throw exceptions
And don't set the value unless it passes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 17:20:06 +09:30
Christian Decker
457d25c1ce pyln: Drive-by type fixed 2024-06-20 17:20:06 +09:30
Christian Decker
2737f2e97e pyln: Add and test a callback based setconfig listener
I was looking into using the `threading.Condition` but since we're
already rather heavily using callbacks, this allows us to stay
single-threaded, and not having to completely hook the `setconfig`
function.

Changelog-Added: pyln-client: Added a notification mechanism for config changes
2024-06-20 17:20:06 +09:30
Rusty Russell
572ccac0e5 pyln-client: allow dynamic=True add_option to actually allow updates.
We didn't actually *change* the value you'd see, when we got a setconfig call!

Changelog-Added: pyln-client: implement setconfig hook for plugins so you can see changes in `dynamic` options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 17:20:06 +09:30
Rusty Russell
ead211e5e4 autoclean: call list in easy stages.
listforwards on a large node can easily run out of memory.  Sip, don't
gulp!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
c62e1f432f autoclean: use filter to reduce processing.
We can filter down to only the list* fields we need.  In the case of a
node with 1M forwards, this reduces listforwards time from 5 seconds
to 4 seconds.  It will also reduce memory consumption.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
c7ce9c55df libplugin: add plugin_option_dev_dynamic (our first dynamic dev option coming).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
1f9ddd24b2 libplugin: support filtering of outgoing commands.
This is crude, handing a raw JSON string, but it works.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
d2465d4498 autoclean: rework logic to clarify subsystems.
There are really three subsystems: invoices, forwards and sendpays,
each of which has two variants we care about (successes and failures).
If we split the code that way, we can extract the core differences in
each of these cases and share most of the logic.

It's a bit awkward to iterate over each "subsystem" in the JSON
parameter sense, so we have some iteration code to do that where we
need to.

The result is going to be much easier to paginate!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
83eed5ce8e autoclean: use a pointer for our global clean_info.
We're about to make this a more complex struct, so introducing a
new_clean_info() function unifies the code paths.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
84b6601bb3 autoclean: remove autocleaninvoice command.
Changelog-Removed: JSON-RPC: `autocleaninvoice` command (deprecated v22.11, EOL v24.02)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
b525e26fa5 lightningd: don't load all forwardings into memory at once.
Use modern-style iterators: this can be huge and we will run out of
memory!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30