Commit Graph

12548 Commits

Author SHA1 Message Date
Michael Schmoock
1fb1e0eec4 pytest: test ip discovery for custom port 2023-01-25 17:57:04 +01:00
Michael Schmoock
a62c74be7b doc: usage of --announce-addr-discovered-port option 2023-01-25 17:57:04 +01:00
Michael Schmoock
ca9e3e4cc1 opts: adds --announce-addr-discovered-port config option
This will give the user an option to set a custom port when using
discovered IPs for node_announcents. Without this, only the selected
networks default port can used.

Changelog-Added: Adds --announce-addr-discovered-port config option to set custom port for IP discovery.
2023-01-25 17:57:04 +01:00
Michael Schmoock
34cfc93939 cli: getinfo output to regard --ip-discovery 2023-01-25 14:37:56 +01:00
Michael Schmoock
30dea0a431 opts: deprecate --disable-ip-discovery switch
This switch was not doing anything useful anymore.
We deprecate it anyways to notify the user about the new switch.

Changelog-Deprecated: The old --disable-ip-discovery config switch
2023-01-25 14:37:56 +01:00
Michael Schmoock
a2b94f16f8 pytest: fix and adapt test_remote_addr_disabled 2023-01-25 14:37:56 +01:00
Michael Schmoock
ee046662a7 doc: announce-addr-discovered config switch 2023-01-25 14:37:56 +01:00
Michael Schmoock
3babbc5201 opts: announce-addr-discovered on/off/auto switch
This adds the option to explicitly enable ip-discovery, which maybe
helpful for example when a user wants TOR announced along with
discovered IPs to improve connectivity and have TOR just as a fallback.

Changelog-Added: Adds config switch 'announce-addr-discovered': on/off/auto
2023-01-25 14:37:56 +01:00
Michael Schmoock
bd75f8ea6c opts: adds the autobool on/off/auto feature 2023-01-25 14:37:56 +01:00
Michael Schmoock
e932f05bc8 ci: adds git fetch before doing schema checks
This fixes the CI errors when doing `make check-source`
steps 'schema-added-check' and 'schema-removed-check'.
These errors prevented CI from performing these steps correctly:

```
fatal: ambiguous argument 'main': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
```

I changed it so that CI does a `git fetch origin` at first and do
the `git diff` against 'origin/master' (which then exist).
Also fixed a bug in the script that was missing $$master in the same line.

Also I added that the script shows the actual diff before failing,
so the user quickly sees whats wrong.
2023-01-25 14:37:56 +01:00
Rusty Russell
0e25d56329 lightningd: use a hash table for peer->dbid.
Otherwise, loading up when we have 100k peers is *painful*!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell
cfa632b0e9 lightningd: use hash map for peers instead of linked list.
After connecting 100,000 peers with one channel each (not all at
once!), we see various places where we exhibit O(N^2) behaviour.

Fix these by keeping a map of id->peer instead of a simple
linked-list.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell
17aa047b17 pytest: fix output order assumption in test_setchannel_all
Hash table coming up makes it random!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell
6a95d3a25e common: expose node_id_hash functions.
They're used in several places, and we're about to add more.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Rusty Russell
0faa8397c3 wallet: add dependency on lightningd/ headers.
wallet should really be a subdir of lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-21 08:05:31 -06:00
Matt Whitlock
acfb63e4bf channeld: remove dead HTLCs from htable and free them (eventually)
The channel->htlcs map was exhibiting unbounded growth, as elements were
never removed from it. This was causing lightning_channeld processes to
consume ever-increasing amounts of memory, and iterating over the map
was causing ever-increasing CPU utilization. There were FIXME comments
suggesting that the intention was to remove HTLCs from the map upon
their deaths. This commit implements that intention.

Changelog-Fixed: channeld no longer retains dead HTLCs in memory.
2023-01-20 14:04:08 -06:00
daywalker90
82c94330a5 add PartialEq to ShortChannelId
PartialEq makes it possible to compare 2 ShortChannelIds for equality which i found often helpful
2023-01-17 14:58:07 +01:00
Vincenzo Palazzo
8f94e8b943 comm: make sure that our version check is reliable
Rework the logic of the version check used in the
database migration, and make sure
that it is full functional to avoid confusion
at release time.

Changelog-Fixed: database: Correctly identity official release versions for database upgrade.

Reported-by: @urza
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2023-01-17 14:15:24 +10:30
Rusty Russell
8d825ef0b7 lightningd: fix valgrind reported leak when we exit early.
```
E           Global errors:
E            - Node /tmp/ltests-adkwu44c/test_logging_1/lightning-2/ has memory leaks: [
E               {
E                   "backtrace": [
E                       "ccan/ccan/tal/tal.c:442 (tal_alloc_)",
E                       "lightningd/peer_control.c:2203 (load_channels_from_wallet)",
E                       "lightningd/lightningd.c:1105 (main)"
E                   ],
E                   "label": "lightningd/peer_control.c:2203:struct htlc_in_map",
E                   "parents": [
E                       "lightningd/lightningd.c:107:struct lightningd"
E                   ],
E                   "value": "0x55d920a345e8"
E               }
E           ]
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-17 14:13:45 +10:30
Rusty Russell
288f5df8d1 ccan: update to fix recent gcc "comparison will always evaluate as 'false'" warning
```
lightningd/jsonrpc.c: In function ‘destroy_json_command’:
lightningd/jsonrpc.c:1180:63: error: the comparison will always evaluate as ‘false’ for the address of ‘canary’ will never be NULL [-Werror=address]
lightningd/jsonrpc.c:108:53: note: ‘canary’ declared here
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-17 14:13:45 +10:30
Christian Decker
1f21796454 gha: Temporarily disable test_notify
It is failing due to an error somewhere:

```
*** buffer overflow detected ***: terminated
```
2023-01-16 13:53:48 +10:30
Christian Decker
e3a9bda301 ci: Downgrade the upload-artifact action to v2.2.4
There seem to be a couple of issues with Bad Requests with later
versions, trying this :-)
2023-01-16 13:53:48 +10:30
Christian Decker
b0e3d483e6 gci: Add a test for the postgres backend 2023-01-16 13:53:48 +10:30
Christian Decker
383fca7d0e gci: Add an explicit name to the matrix for display
This is going to be useful to discern what goal the job has.
2023-01-16 13:53:48 +10:30
Christian Decker
e17611c570 gci: Split out installation of elements and bitcoin into a script
It was crowding the github workflow file
2023-01-16 13:53:48 +10:30
Christian Decker
71b581da4d make: Clean up duplicate cargo examples build rules 2023-01-16 13:53:48 +10:30
Christian Decker
e76618e2a6 ci: Use bzip2 and release to reduce artifact size 2023-01-16 13:53:48 +10:30
Christian Decker
b40fd3efbd ci: Add gather step
We can require a status to be successful in PRs, but they are
referenced by name, and so we'd have to add each matrix job as
required. That's rather cumbersome, so have this artificial gather
step at the end which signals success on the entire run.
2023-01-16 13:53:48 +10:30
Christian Decker
ca3053707f ci: Build and test in ubuntu:22.04 and install lowdown 2023-01-16 13:53:48 +10:30
Christian Decker
bcc75b6e98 tests: Fix a small memory leak in the onion test vector tester
Changelog-None
2023-01-16 13:53:48 +10:30
Christian Decker
6fe6395678 ci: Add bitcoind to integration test job 2023-01-16 13:53:48 +10:30
Christian Decker
34f0954074 ci: Automatically cancel CI runs if we push a new version to the PR
We often have a number of changes in flight, and we amend PRs before the previous run completes. This allows us to cancel that prior run, preserving our precious runners.
2023-01-16 13:53:48 +10:30
Christian Decker
e7ee40e951 ci: Split out the unit tests
They are run in parallel with the integration tests, not on the
critical path, but can be a bit lengthy since they build a log of C
code, and run in valgrind.
2023-01-16 13:53:48 +10:30
Christian Decker
a650dcb26d ci: Split the pre-flight checks into separate steps
One of these escapes, and actually builds the entire thing. This
should not actually build the CLN binaries, just check it works.
2023-01-16 13:53:48 +10:30
Christian Decker
a20540eb15 ci: Add a testpack.tar target to tranfer artifacts between CI jobs
The tester CI job uses absolute paths to ensure it is testing the
correct binaries. That clashes with the transfer between builder and
tester job using the `install` target because that switches things
around. This commit introduces a new target that just collects
artifacts in place, and tars them. Then we can use `tar` to unpack
them on the tester jobs again.
2023-01-16 13:53:48 +10:30
Christian Decker
69e37a8865 gci: Re-add tests of pre-compiled binaries 2023-01-16 13:53:48 +10:30
Christian Decker
10abb620a8 gci: Clone BOLTs repository in pre-build checks 2023-01-16 13:53:48 +10:30
Christian Decker
6d67eb934d py: Ignore missing whitespace after keyword for now 2023-01-16 13:53:48 +10:30
Christian Decker
4d668e76a0 gci: Split out the stages better
We want to compile with one set of dependencies, and run the tests
with another. This also helps us cut down on the times we compile CLN
itself, by sharing them among stages, and simplifies the logic of each
stage to have one specific goal.
2023-01-16 13:53:48 +10:30
Christian Decker
a00190dce5 gci: Update Github Actions steps to their latest versions
They were causing warnings.
2023-01-16 13:53:48 +10:30
Christian Decker
8c075c4cda gci: Stop uploading unit test results
We never used them, and they were creating distracting warnings.
2023-01-16 13:53:48 +10:30
Christian Decker
1fbf774e04 gci: Split out the lnprototest from the larger CI run
The lnprototests are often blocking PRs, due to them failing, but we
can't restart them when valgrind runs are still ongoing, since they'd
also restart. Splitting allows us to rerun them selectively and waste
less time.

Ideally we'd just fix them, but I am by no means knowledgeable enough
to fix them now.
2023-01-16 13:53:48 +10:30
Rusty Russell
9d5841a0e6 pytest: fix flake in tests/test_closing.py::test_closing_specified_destination
The close call can fail, since we already unilaterally closed since we mined blocks
too fast:

```
2023-01-14T01:00:10.2502199Z E           pyln.client.lightning.RpcError: RPC call failed: method: close, payload: ['107x1x1', None, 'bcrt1qeyyk6sl5pr49ycpqyckvmttus5ttj25pd0zpvg'], error: {'code': -32602, 'message': "Short channel ID not active: '107x1x1'"}
...
2023-01-14T01:00:10.5288050Z lightningd-4 2023-01-14T00:59:59.650Z UNUSUAL 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: Peer permanent failure in CHANNELD_NORMAL: Fulfilled HTLC 0 SENT_REMOVE_COMMIT cltv 113 hit deadline
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-14 21:17:45 +10:30
Rusty Russell
932ca9e91f lightningd: deprecate p2sh-segwit addresses for newaddr addresstype
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `newaddr`: `addresstype` `p2sh-segwit` (use default, or `bech32`).
2023-01-14 21:17:45 +10:30
Rusty Russell
2f36c03307 pytest: use bech32 addresses everywhere.
We used to create some p2sh-segwit addresses just to mix things up.  This
streamlines back to just bech32.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-14 21:17:45 +10:30
Rusty Russell
85fc46f76c pyln-testing: don't default openchannel and fundwallet to p2sh-segwit, use bech32.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-14 21:17:45 +10:30
Christian Decker
e5d384a427 gossip: Do not send warnings if we fail to parse a channel_update
We removed a warning about the channel_update being malformed since
the warning could cause lnd to disconnect (seems they treat
channel-unrelated warnings as fatal?). This was caused by lnd not
enforcing the `htlc_maximum`, thus the parsing would fail. We can
re-add the warning once our assumption that `htlc_maximum` being set
is valid.

Changelog-Fixed: gossip: We no longer send warning that lnd would not understand if we get outdated gossip
2023-01-14 12:39:10 +10:30
Rusty Russell
2b5f4d14d1 CI: fix schema diff check.
GitHub's master branch is called "main" apparently.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-14 10:13:28 +10:30
Riccardo Casatta
6b977f0292 cln-grpc: update listpeers json fixing tests 2023-01-13 21:42:27 +01:00
Rusty Russell
f1373fd98c doc: remove manual field descriptions from listpeerchannels(7).
Some are best copied into the schema, but some are already
out-of-date, so cleanest to remove them and rely on the generated (and
thus, checked!) fields.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-01-13 10:42:42 +10:30