We were accidentally testing against `bitcoind` rather than
`elementsd` which meant we believed everything was find, when in
reality some tests broke. By only installing the required daemon we
ensure that such a mixup causes a CI failure.
If you previously configured with `--enable-developer` we turn that into `--enable-debugbuild`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: build: `--enable-developer` arg to configure (and DEVELOPER variables): use `./configure --enable-debugbuild` and `developer` setting at runtime.
We use this file as a proxy for breaking changes in the signer
protocol. It may not catch all the breaking changes, but it's a
good first approximation.
Update the lightningd <-> channeld interface with lots of new commands to needed to facilitate spicing.
Implement the channeld splicing protocol leveraging the interactivetx protocol.
Implement lightningd’s channel_control to support channeld in its splicing efforts.
Changelog-Added: Added the features to enable splicing & resizing of active channels.
GH was complaining about Node v12 being used in v2.2.4. We used to
hold back from this upgrade because v3 was breaking quotas, but we
should try again.
The lnprototest occasionally fails on our CI due to the
timeout duration we set while waiting for cln to start up.
This timeout is insufficient for machines like the ones
used in the GitHub CI environment.
To address this issue, this commit introduces the use of
environment variables to increase the lnprototest timeout,
ensuring sufficient time for cln to initialize.
Changelog-None
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
With the exception of fuzzing, make all builds in the `compile` job,
and simply download them in the other steps.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We used to have to use environment variables to pass through to
github/scripts/build.sh, but now we run ./configure directly it's
clearer to use explicit flags (though some matrixes still use env vars
for simplicity).
We also don't need to set COMPAT, as it's the default (MacOS tests
that we build without it, but otherwise we assume it's on).
And we make `gather` actually depend on all the other steps!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This controls debug flags for the build, rather than --developer,
which is going away.
I thought about making this flag control the RUST_PROFILE too, but
it seems that we want that set to "release" for CI, whereas for the
C code we want --enable-debugbuild.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It seems to reliably be getting a SIGTERM after 17-18 minutes:
```
[gw1] [ 91%] PASSED tests/test_plugin.py::test_forward_event_notification
Error: Process completed with exit code 143.
```
Perhaps this is out of mem? So, try -n2. We also make the configure
line explicit, rather than relying on environment vars (we should probably
do this for the other cases, too)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
build.sh did this, but we didn't carry it across to the new test
matrix, leading to spurious CI failures:
```
lightningd-1 2023-06-06T16:15:34.931Z DEBUG plugin-bookkeeper: Setting up database at ***localhost:39061/accounts_0_7ccmg745
lightningd-1 2023-06-06T16:15:34.931Z INFO plugin-bookkeeper: Creating database
lightningd-1 2023-06-06T16:15:34.931Z **BROKEN** plugin-bookkeeper: Error vacuuming db: VACUUM command failed: ERROR: deadlock detected\nDETAIL: Process 77248 waits for AccessShareLock on relation 1260 of database 0; blocked by process 77414.\nProcess 77414 waits for RowExclusiveLock on relation 1214 of database 0; blocked by process 77248.\nHINT: See server log for query details.\n
lightningd-1 2023-06-06T16:15:34.931Z INFO plugin-bookkeeper: Killing plugin: exited before replying to init
lightningd-1 2023-06-06T16:15:34.931Z **BROKEN** plugin-bookkeeper: Plugin marked as important, shutting down lightningd!
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This PR:
- adds all the guides (in markdown format) that is published at https://docs.corelightning.org/docs
- adds a github workflow to sync any future changes made to files inside the guides folder
- does not include API reference (json-rpc commands). Those will be handled in a separate PR since they're used as manpages and will require a different github workflow
Note that the guides do not exactly map to their related files in doc/, since we reorganized the overall documentation structure on readme for better readability and developer experience. For example, doc/FUZZING.md and doc/HACKING.md#Testing are merged into testing.md in the new docs. As on the creation date of this PR, content from each of the legacy documents has been synced with the new docs. Until this PR gets merged, I will continue to push any updates made to the legacy documents into the new docs.
If this looks reasonable, I will add a separate PR to clean up the legacy documents from doc/ (or mark them deprecated) to avoid redundant upkeep and maintenance.
Changelog-None
This reflects what we actually do when we apply the commit, and also
means we can easily iterate the commits.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Build: all experimental features are now runtime-enabled; no more ./configure --enable-experimental-features
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Without this, a stuck test (such as before the previous commit, where a plugin crashed
when running a command) simply gets timed out by the full CI timeout.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There are several benefits of doing this:
- prevent fuzz target bit rot
- more test coverage in CI
- greater visibility of fuzz tests, encouraging contributions to the
seed corpus and tests themselves
We can require a status to be successful in PRs, but they are
referenced by name, and so we'd have to add each matrix job as
required. That's rather cumbersome, so have this artificial gather
step at the end which signals success on the entire run.
We often have a number of changes in flight, and we amend PRs before the previous run completes. This allows us to cancel that prior run, preserving our precious runners.
They are run in parallel with the integration tests, not on the
critical path, but can be a bit lengthy since they build a log of C
code, and run in valgrind.
The tester CI job uses absolute paths to ensure it is testing the
correct binaries. That clashes with the transfer between builder and
tester job using the `install` target because that switches things
around. This commit introduces a new target that just collects
artifacts in place, and tars them. Then we can use `tar` to unpack
them on the tester jobs again.
We want to compile with one set of dependencies, and run the tests
with another. This also helps us cut down on the times we compile CLN
itself, by sharing them among stages, and simplifies the logic of each
stage to have one specific goal.
The lnprototests are often blocking PRs, due to them failing, but we
can't restart them when valgrind runs are still ongoing, since they'd
also restart. Splitting allows us to rerun them selectively and waste
less time.
Ideally we'd just fix them, but I am by no means knowledgeable enough
to fix them now.
We used to install it only when Rust was configured, but for some
reason all builds now seem to be Rust builds. That's ok, so just
provide the necessary dependency.
The runner version was recently bumped causing our tests to fail
because it couldn't find the `poetry` version it just installed. We
instead install python3.10, and force its use, since otherwise we end
up compiling grpcio and fail to do so.
Changelog-None
I was wondering why the workflow never published, turns out we were triggering on
branch pushes only, not tags. So the branch would get pushed, but the tag is pushed
afterwards, and thus not triggering.
Requires us to update to latest lnproto which is now using the most up
to date python-bitcoinlib, as well as updating our python lock files
(which pin the grpcio deps, because of locking problems h/t @cdecker)
If compilation failed, we didn't stop (though except for fuzzing, we
would fail when we try to run the tests).
Also use make -s instead of redirecting make ooutput.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Over time, it has cost us more developer cycles than it has gained.
It has hidden intermittant bugs, and allowed cruft to accumulate:
when we eventually tried to figure out what was going wrong, the
actual change which caused it was now stale and forgotten.
This was a particular bane during the connectd rewrite, and I
worked through some issues which had occurred before, but were not
more likely.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
For some reason Github complains that @wythe doesn't have access to
the public repository, removing in order to silence that warning. Feel
free to contact me to investigate what's happening and re-enable :-)
Changelog-None
I have no idea why someone else owns the directory suddenly, but all git
commands fail. Workaround as suggested by the error message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
