Commit graph

14939 commits

Author SHA1 Message Date
ShahanaFarooqui
72079cc920 script: Adding Delete action for readme docs
Documents which are deleted from lightning schema were still listed on readme server.

This script was adding or updating the list but delete action was missing.

Changelog-None.
2024-06-25 12:34:07 -07:00
Rusty Russell
c30502779e CI: set elements and bitcoind versions in one place.
We have them spread everywhere: unify them, and make setup.sh use install-bitcoind.sh

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 09:36:00 +09:30
Rusty Russell
cb4dd7489c lightningd: check command should return as much detail as possible.
If they're explicitly calling "check", don't obfuscate the result!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
9fd29d35dd common: move now-enlarged command_fail_badparam into its own source file.
It's getting a bit awkward to inline now: it's non-trivial.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
f33c5188ef common: don't mention the contents of invalid parameters when in non-dev mode.
Shahana decided this was the optimal UX path, though I insisted that we still
report the actual problem directly when in dev mode, as a compromise.

Suggested-by: https://github.com/Amperstrand
Changelog-Changed: JSON-RPC: Do not return the contents of invalid parameters in error messages, refer to logs (use 'check' to get full error messages)
Fixes: https://github.com/ElementsProject/lightning/issues/7338
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
e0e38c2cd5 common: new function command_log to log something about a specific command.
Needs implementations for lightningd and libplugin, since they both use this
infrastructure.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
3531414d18 lightningd: rename command_log() to command_logger()
It doesn't actually log, just gets the `struct logger`, so this name is better.

We're also going to implement command_log() as an actual logging
function in next commit.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-25 08:47:50 +09:30
Rusty Russell
8288603bab pytest: fix flake in test_emergencyrecover
```
2024-06-24T05:09:32.5233603Z         assert l1.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234187Z >       assert l2.rpc.listfunds()["channels"][0]["state"] == "ONCHAIN"
2024-06-24T05:09:32.5234917Z E       AssertionError: assert 'FUNDING_SPEND_SEEN' == 'ONCHAIN'
2024-06-24T05:09:32.5235464Z E         - ONCHAIN
2024-06-24T05:09:32.5235773Z E         + FUNDING_SPEND_SEEN
2024-06-24T05:09:32.5236096Z 
2024-06-24T05:09:32.5236242Z tests/test_misc.py:2907: AssertionError
```

It's possible that l2 hasn't seen the onchain tx yet.  What we really want to do is
wait for it, then test that l1 really can spend the output it has recovered.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
f642e4e551 pytest: fix test_feerate_stress race
```
2024-06-24T05:14:14.9939033Z >       l1.rpc.call('dev-feerate', [l2.info['id'], rate - 5])
2024-06-24T05:14:14.9939354Z 
2024-06-24T05:14:14.9939466Z tests/test_connection.py:3439: 
...
2024-06-24T05:14:14.9967617Z >           raise RpcError(method, payload, resp['error'])
2024-06-24T05:14:14.9968833Z E           pyln.client.lightning.RpcError: RPC call failed: method: dev-feerate, payload: ['022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59', 2290], error: {'code': -1, 'message': 'Peer bad state'}
```

The disconnect can actually take a while: wait for both sides to
believe they're reconnected.  Also wait a little to make sure the
feerate change doesn't create a bad signature.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
bb64fc8ddc pytest: fix flake in test_feerate_stress.
If randomly l1done or l2done are zero, we will create invalid hex:

```
with pytest.raises(RpcError, match='WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS|WIRE_TEMPORARY_CHANNEL_FAILURE'):
>           l2.rpc.waitsendpay("{:064x}".format(l2done - 1), timeout=TIMEOUT)

tests/test_connection.py:3438: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
contrib/pyln-client/pyln/client/lightning.py:1361: in waitsendpay
    return self.call("waitsendpay", payload)
contrib/pyln-testing/pyln/testing/utils.py:715: in call
    schemas[0].validate(testpayload)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = Validator(schema={'pairedWith': [['partid', 'groupid']], 'properties': {'groupid': {'description': ['Grouping key...pa... {'description': ['A timeout in... the payment.'], 'type': 'u32'}}, 'required': ['payment_hash']}, format_checker=None)
args = ({'payment_hash': '-000000000000000000000000000000000000000000000000000000000000001', 'timeout': 180},)
kwargs = {}
error = <ValidationError: "'-000000000000000000000000000000000000000000000000000000000000001' is not of type 'hash'">

    def validate(self, *args, **kwargs):
        for error in self.iter_errors(*args, **kwargs):
>           raise error
E           jsonschema.exceptions.ValidationError: '-000000000000000000000000000000000000000000000000000000000000001' is not of type 'hash'
E           
E           Failed validating 'type' in schema['properties']['payment_hash']:
E               {'description': ['The hash of the *payment_preimage*.'], 'type': 'hash'}
E           
E           On instance['payment_hash']:
E               '-000000000000000000000000000000000000000000000000000000000000001'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
fd9975c288 reckless: don't use time-based directories.
This always fails on my test machine, which runs many tests in parallel.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
b748c1845c pytest: fix flake in test_closing.py::test_penalty_htlc_tx_timeout
lightningd-3 penalizes lightningd-2 but then it can see the preimage for the HTLC which
has already been timed out:

```
2024-06-24T02:41:29.4633900Z lightningd-3 2024-06-24T02:33:54.073Z **BROKEN** 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-onchaind-chan#1: HTLC already resolved by THEIR_HTLC_TIMEOUT_TO_THEM when we found preimage
```

This is fair: the test deliberately takes l3 offline for long enough
that the HTLC can get timed out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 21:12:43 +09:30
Rusty Russell
80cde517c9 lightningd: fix race in initialization.
Found by very slow CI: the two io_breaks() can combine into one, so we
block waiting for the second initialization which never happens.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
9da66100c1 chaintopology: free outstanding requests so we don't get responses while shutting down.
I've never seen this, but the race looks possible so we should close it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
928b7c83fc lightningd: pass explicit context for bitcoind request functions.
That way if you free the context, you free the call (i.e. the callback
will not be called).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
492ed84dbf lightningd: move checkchain_timer out of bitcoind, remove stopped flag.
checkchain_timer is run by chaintopology, so why have the pointer in
bitcoind?

And since we free the timers, we don't need them to self-disable by
checking the stopped flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
f2b033d7e1 lightningd: move all chainheight logic into setup_topology.
Handling half in main() and half here was a mess.  And the name
"max_blockheight" was poor: it was the max in the db, or UINT32_MAX,
but then we changed it depending on what height we wanted to start at.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
ff15b943fe lightningd: remove unused "min_blockheight" from chain_topology struct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
f643fb88f5 lightningd: wait for bitcoind at startup if it's fallen behind.
Fixes: https://github.com/ElementsProject/lightning/issues/6924
Changelog-Changed: lightningd: we wait for bitcoind if it has somehow gone backwards (as long as header height is still ok).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
59afbac7cf lightningd: remove first_call param for getchaininfo.
It was a weird arbitrary bool passed through from the caller, which we no longer need.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
eb70476fb3 lightningd: split chaintopology init from normal loops.
The current code is confusing: there are polling loops, but we wait for
them to run once. 

Be explicit: make the calls once, then start the loops in begin_topology.

This removes various chain_topology struct members which only exist for
startup.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
f318d08e94 lightningd: add arg to bitcoind_estimate_fees callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
Rusty Russell
436bf6c9e1 lightningd: pass height back from getrawblockbyheight.
Several callers stash this, but we have it, so make it explicit:
here's the block you asked for.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-24 17:26:11 +09:30
ShahanaFarooqui
b803609247 docs: Removed example usage
Example usage has been either merged with examples or usage keys. This heading was not required separately.

This PR edits & rearranges Examples, Example Notifications and Example Usage data.

Changelog-None.
2024-06-22 17:02:33 +02:00
ShahanaFarooqui
e94b793f05 docs: Example notifications heading
- Added missing Example Notifications Heading
- Renamed `example_json_notifications` to `example_notifications`
- Moved `example_notifications` to the end of the page
2024-06-22 17:02:33 +02:00
ShahanaFarooqui
976c0c90bc docs: Changes examples format
- Changed ALL `doc/schemas/lightning-*.json` file's `json_example` to `examples`
- Change the heading from example to examples
- Bring shell command before the json command
- Move Example to the end of the page
- Remove horizontal line from Example
2024-06-22 17:02:33 +02:00
Christian Decker
1347e86671 wallet: Log to chan->log rather than wallet->log
Suggested-by: Rusty Russell <@rustyrussell>
2024-06-22 14:50:21 +02:00
Christian Decker
001b190c61 feat(main): Print channel status when loading it from DB
I was trying to debug a node with several multiplexed channels, and
was finding it a bit difficult to determine which channel index
matches which result of `listpeerchannels` as well as figuring out
what their status was. This just prints the status in string format
when loading the channel from the DB.

Changelog-Changed: wallet: The channel status is printed when loading it from the DB
2024-06-22 14:50:21 +02:00
Rusty Russell
e21b70cad0 pytest: fix default/description order in zeroconf test plugin.
Noticed as I was debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 17:20:06 +09:30
Christian Decker
31cf9225f4 pyln: Turn the plugin options into a real Type (uppercase T)
[ Fix not to include 'value' and 'default' (if None) in getmanifest response --RR ]
[ Fix to support [] operator for existing plugins (including our test ones!) --RR ]
2024-06-20 17:20:06 +09:30
Rusty Russell
1eb1db0e24 pyln-client: allow dynamic option setter to throw exceptions
And don't set the value unless it passes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 17:20:06 +09:30
Christian Decker
457d25c1ce pyln: Drive-by type fixed 2024-06-20 17:20:06 +09:30
Christian Decker
2737f2e97e pyln: Add and test a callback based setconfig listener
I was looking into using the `threading.Condition` but since we're
already rather heavily using callbacks, this allows us to stay
single-threaded, and not having to completely hook the `setconfig`
function.

Changelog-Added: pyln-client: Added a notification mechanism for config changes
2024-06-20 17:20:06 +09:30
Rusty Russell
572ccac0e5 pyln-client: allow dynamic=True add_option to actually allow updates.
We didn't actually *change* the value you'd see, when we got a setconfig call!

Changelog-Added: pyln-client: implement setconfig hook for plugins so you can see changes in `dynamic` options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 17:20:06 +09:30
Rusty Russell
ead211e5e4 autoclean: call list in easy stages.
listforwards on a large node can easily run out of memory.  Sip, don't
gulp!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
c62e1f432f autoclean: use filter to reduce processing.
We can filter down to only the list* fields we need.  In the case of a
node with 1M forwards, this reduces listforwards time from 5 seconds
to 4 seconds.  It will also reduce memory consumption.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
c7ce9c55df libplugin: add plugin_option_dev_dynamic (our first dynamic dev option coming).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
1f9ddd24b2 libplugin: support filtering of outgoing commands.
This is crude, handing a raw JSON string, but it works.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
d2465d4498 autoclean: rework logic to clarify subsystems.
There are really three subsystems: invoices, forwards and sendpays,
each of which has two variants we care about (successes and failures).
If we split the code that way, we can extract the core differences in
each of these cases and share most of the logic.

It's a bit awkward to iterate over each "subsystem" in the JSON
parameter sense, so we have some iteration code to do that where we
need to.

The result is going to be much easier to paginate!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
83eed5ce8e autoclean: use a pointer for our global clean_info.
We're about to make this a more complex struct, so introducing a
new_clean_info() function unifies the code paths.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
84b6601bb3 autoclean: remove autocleaninvoice command.
Changelog-Removed: JSON-RPC: `autocleaninvoice` command (deprecated v22.11, EOL v24.02)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
Rusty Russell
b525e26fa5 lightningd: don't load all forwardings into memory at once.
Use modern-style iterators: this can be huge and we will run out of
memory!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-20 16:14:38 +09:30
daywalker90
6ad8cfdffa CI: upgrade CI actions to latest releases
Changelog-None
2024-06-19 21:21:00 -07:00
fanquake
2614dfc104 ci: use Bitcoin Core 27.1
Changelog-None
2024-06-20 11:28:21 +09:30
Niklas Gögge
259015720e fuzz: Add input for from_bech32_charset off-by-one bug 2024-06-20 10:53:50 +09:30
Rusty Russell
1a1696f2f9 lightningd: don't try to "save" incoming channel by closing early if incoming already onchain.
From test_penalty_htlc_tx_timeout[False] flake (with keys replaced by l2 and l5, for clarity)

```
lightningd-3 2024-06-04T04:34:00.942Z UNUSUAL l2-chan#1: Peer permanent failure in CHANNELD_NORMAL: Funding transaction spent
lightningd-3 2024-06-04T04:34:01.570Z UNUSUAL l5-chan#3: Peer permanent failure in CHANNELD_NORMAL: Funding transaction spent
lightningd-3 2024-06-04T04:34:01.655Z UNUSUAL l5-chan#3: Abandoning unresolved onchain HTLC at block 132 (expired at 125) to avoid peer closing incoming HTLC at block 131
lightningd-3 2024-06-04T04:34:02.802Z **BROKEN** l5-chan#3: FUNDS LOSS of 50000000msat: peer took funds onchain before we could time out the HTLC, but we abandoned incoming HTLC to save the incoming channel
```

So, we were already closing l2, no reason to abandon it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-19 23:58:50 +02:00
Rusty Russell
db67df3364 lightningd: log BROKEN if we lost funds, only unusual if we closed abandoned HTLC.
Started seeing these in my logs, and it's the wrong way around!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-19 23:58:50 +02:00
Rusty Russell
0afaf2f3ca msggen: don't spam every time we rebuild.
2923 lines of output?  Even with warnings-only, it's 59 lines, so only
enable that with `make V=1`.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-19 12:34:41 +02:00
Rusty Russell
7c3aa9477f pyproject.toml: Update ancient flake8
v4.0 no longer works for me (see below, and widely reported elsewhere).
v5.0 doesn't understand f strings, and creates a flood of complaints.
v6.0 requires python >= 3.8.1, so we need to update that.
v7.0 is the latest, but why push it.

```
make check-python-flake8
Traceback (most recent call last):
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/bin/flake8", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 375, in run
    self._run(argv)
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 363, in _run
    self.initialize(argv)
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 343, in initialize
    self.find_plugins(config_finder)
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/main/application.py", line 157, in find_plugins
    self.check_plugins = plugin_manager.Checkers(local_plugins.extension)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/plugins/manager.py", line 363, in __init__
    self.manager = PluginManager(
                   ^^^^^^^^^^^^^^
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/plugins/manager.py", line 243, in __init__
    self._load_entrypoint_plugins()
  File "/home/rusty/.cache/pypoetry/virtualenvs/cln-meta-project-BgKQHyxC-py3.12/lib/python3.12/site-packages/flake8/plugins/manager.py", line 261, in _load_entrypoint_plugins
    eps = importlib_metadata.entry_points().get(self.namespace, ())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'EntryPoints' object has no attribute 'get'
make: *** [Makefile:535: check-python-flake8] Error 1
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2024-06-19 12:34:41 +02:00
ShahanaFarooqui
3d841d5b8a doc: Added Miscellaneous section for Docker users
Added CLN image execution and testing hints for Docker users.

Changelog-None.
2024-06-19 18:41:53 +09:30