bitcoin/ci
Ava Chow d97ddbe797
Merge bitcoin/bitcoin#30193: ci: move ASan job to GitHub Actions from Cirrus CI
9eea51d905 ci: move Asan / LSan / USDT job to Github Actions (Max Edwards)
4b527fa93b ci: add IPV6 network to ci container (Max Edwards)

Pull request description:

  PR for moving the ASAN + LSAN + USDT + friends job to github actions from Cirrus.

  The motivation for this PR is that this task needs a full VM (or bare metal) to function, because of the tracepoints. It can not run in a container on an arbitrary Linux, because the outside machine must exactly match the specification of the distro used in the CI task config. This requires more maintenance for the persistent worker, and I think moving to GHA will reduce the maintenance burden, or at least make it possible for anyone to work on.

  Also, it makes it easier to run the task on forks (bitcoin-inquisition, bitcoin-knots, devel forks, ...) without having to set-up a real machine.

ACKs for top commit:
  maflcko:
    review ACK 9eea51d905
  achow101:
    ACK 9eea51d905
  hebasto:
    ACK 9eea51d905.

Tree-SHA512: 1111c1c9e3a11e725dff1344643fff3c91fb9b4d7c1cc9a7d507a8f146f5223316a00272030b41ae37ecb59d044f2e90e1cd907450049b25f094f0b60643d4c7
2024-06-17 15:49:43 -04:00
..
lint lint: add markdown hyperlink checker 2024-05-24 12:11:50 +01:00
retry
test Merge bitcoin/bitcoin#30193: ci: move ASan job to GitHub Actions from Cirrus CI 2024-06-17 15:49:43 -04:00
lint_imagefile ci: Add missing COPY for ./test/lint/test_runner 2023-11-13 18:10:51 +01:00
lint_run_all.sh lint: Add missing set -ex to ci/lint/06_script.sh 2023-07-19 11:39:50 +02:00
README.md docs: ci multi-arch requires qemu 2024-02-20 10:55:33 +00:00
test_imagefile ci: Fix macOS-cross SDK rsync 2023-08-16 10:30:50 +02:00
test_run_all.sh ci: move-only CI_CONTAINER_ID to 02_run_container.sh 2023-10-09 16:17:04 +02:00

CI Scripts

This directory contains scripts for each build step in each build stage.

Running a Stage Locally

Be aware that the tests will be built and run in-place, so please run at your own risk. If the repository is not a fresh git clone, you might have to clean files from previous builds or test runs first.

The ci needs to perform various sysadmin tasks such as installing packages or writing to the user's home directory. While it should be fine to run the ci system locally on you development box, the ci scripts can generally be assumed to have received less review and testing compared to other parts of the codebase. If you want to keep the work tree clean, you might want to run the ci system in a virtual machine with a Linux operating system of your choice.

To allow for a wide range of tested environments, but also ensure reproducibility to some extent, the test stage requires bash, docker, and python3 to be installed. To run on different architectures than the host qemu is also required. To install all requirements on Ubuntu, run

sudo apt install bash docker.io python3 qemu-user-static

It is recommended to run the ci system in a clean env. To run the test stage with a specific configuration,

env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'

Configurations

The test files (FILE_ENV) are constructed to test a wide range of configurations, rather than a single pass/fail. This helps to catch build failures and logic errors that present on platforms other than the ones the author has tested.

Some builders use the dependency-generator in ./depends, rather than using the system package manager to install build dependencies. This guarantees that the tester is using the same versions as the release builds, which also use ./depends.

It is also possible to force a specific configuration without modifying the file. For example,

env -i HOME="$HOME" PATH="$PATH" USER="$USER" bash -c 'MAKEJOBS="-j1" FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh'

The files starting with 0n (n greater than 0) are the scripts that are run in order.

Cache

In order to avoid rebuilding all dependencies for each build, the binaries are cached and reused when possible. Changes in the dependency-generator will trigger cache-invalidation and rebuilds as necessary.