diff --git a/contrib/guix/README.md b/contrib/guix/README.md index dffcf99607a..dbe1ea837b9 100644 --- a/contrib/guix/README.md +++ b/contrib/guix/README.md @@ -40,25 +40,27 @@ Otherwise, follow the [Guix installation guide][guix/bin-install]. Guix allows us to achieve better binary security by using our CPU time to build everything from scratch. However, it doesn't sacrifice user choice in pursuit of -this: users can decide whether or not to bootstrap and to use substitutes. +this: users can decide whether or not to bootstrap and to use substitutes +(pre-built packages). After installation, you may want to consider [adding substitute -servers](#speeding-up-builds-with-substitute-servers) to speed up your build if -that fits your security model (say, if you're just testing that this works). -This is skippable if you're using the [Dockerfile][fanquake/guix-docker]. +servers](#speeding-up-builds-with-substitute-servers) from which to download +pre-built packages to speed up your build if that fits your security model (say, +if you're just testing that this works). Substitute servers are set up by +default if you're using the [Dockerfile][fanquake/guix-docker]. -If you prefer not to use any substitutes, make sure to set -`ADDITIONAL_GUIX_ENVIRONMENT_FLAGS` like the following snippet. The first build -will take a while, but the resulting packages will be cached for future builds. +If you prefer not to use any substitutes, make sure to supply `--no-substitutes` +like in the following snippet. The first build will take a while, but the +resulting packages will be cached for future builds. ```sh -export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--no-substitutes' +export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ``` Likewise, to perform a bootstrapped build (takes even longer): ```sh -export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap --no-substitutes' +export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap' ``` ### Using a version of Guix with `guix time-machine` capabilities @@ -82,17 +84,6 @@ export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH" ## Usage -### As a Development Environment - -For a Bitcoin Core depends development environment, simply invoke - -```sh -guix environment --manifest=contrib/guix/manifest.scm -``` - -And you'll land back in your shell with all the build dependencies required for -a `depends` build injected into your environment. - ### As a Tool for Deterministic Builds From the top of a clean Bitcoin Core repository: @@ -113,10 +104,8 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum * _**HOSTS**_ Override the space-separated list of platform triples for which to perform a - bootstrappable build. _(defaults to "x86\_64-linux-gnu - arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_ - - > Windows and OS X platform triplet support are WIP. + bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf + aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32")_ * _**SOURCES_PATH**_ @@ -147,13 +136,29 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum string) is interpreted the same way as not setting `V` at all, and that `V=0` has the same effect as `V=1`. -* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_ +* _**SUBSTITUTE_URLS**_ - Additional flags to be passed to `guix environment`. For a fully-bootstrapped + A whitespace-delimited list of URLs from which to download pre-built packages. + A URL is only used if its signing key is authorized (refer to the [substitute + servers section](#speeding-up-builds-with-substitute-servers) for more + details). + +* _**ADDITIONAL_GUIX_COMMON_FLAGS**_ + + Additional flags to be passed to all `guix` commands. For a fully-bootstrapped build, set this to `--bootstrap --no-substitutes` (refer to the [security model section](#choosing-your-security-model) for more details). Note that a fully-bootstrapped build will take quite a long time on the first run. +* _**ADDITIONAL_GUIX_TIMEMACHINE_FLAGS**_ + + Additional flags to be passed to `guix time-machine`. + +* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_ + + Additional flags to be passed to the invocation of `guix environment` inside + `guix time-machine`. + ## Tips and Tricks ### Speeding up builds with substitute servers @@ -161,14 +166,15 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum _This whole section is automatically done in the convenience [Dockerfiles][fanquake/guix-docker]_ -For those who are used to life in the fast _(and trustful)_ lane, you can use -[substitute servers][guix/substitutes] to enable binary downloads of packages. +For those who are used to life in the fast _(and trustful)_ lane, you can +specify [substitute servers][guix/substitutes] from which to download pre-built +packages. > For those who only want to use substitutes from the official Guix build farm > and have authorized the build farm's signing key during Guix's installation, > you don't need to do anything. -#### Authorize the signing keys +#### Step 1: Authorize the signing keys For the official Guix build farm at https://ci.guix.gnu.org, run as root: @@ -182,7 +188,7 @@ For dongcarl's substitute server at https://guix.carldong.io, run as root: wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize ``` -#### Use the substitute servers +#### Step 2: Specify the substitute servers The official Guix build farm at https://ci.guix.gnu.org is automatically used unless the `--no-substitutes` flag is supplied. @@ -196,7 +202,7 @@ To use dongcarl's substitute server for Bitcoin Core builds after having [authorized his signing key](#authorize-the-signing-keys): ``` -export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"' +export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org' ``` ## FAQ @@ -212,9 +218,9 @@ As mentioned at the bottom of [this manual page][guix/bin-install]: ### When will Guix be packaged in debian? -Vagrant Cascadian has been making good progress on this -[here][debian/guix-package]. We have all the pieces needed to put up an APT -repository and will likely put one up soon. +Thanks to Vagrant Cascadian's diligent work, Guix is now [in debian +experimental][debian/guix-experimental]! Hopefully it will make its way into a +release soon. [b17e]: http://bootstrappable.org/ [r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/ @@ -226,5 +232,5 @@ repository and will likely put one up soon. [guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html [guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html -[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644 +[debian/guix-experimental]: https://packages.debian.org/experimental/guix [fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix diff --git a/contrib/guix/guix-build.sh b/contrib/guix/guix-build.sh index 11d2c8b8672..54cc5793f6e 100755 --- a/contrib/guix/guix-build.sh +++ b/contrib/guix/guix-build.sh @@ -2,6 +2,116 @@ export LC_ALL=C set -e -o pipefail +################### +## Sanity Checks ## +################### + +################ +# Check 1: Make sure that we can invoke required tools +################ +for cmd in git make guix cat mkdir; do + if ! command -v "$cmd" > /dev/null 2>&1; then + echo "ERR: This script requires that '$cmd' is installed and available in your \$PATH" + exit 1 + fi +done + +################ +# Check 2: Make sure GUIX_BUILD_OPTIONS is empty +################ +# +# GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that +# can perform builds. This seems like what we want instead of +# ADDITIONAL_GUIX_COMMON_FLAGS, but the value of GUIX_BUILD_OPTIONS is actually +# _appended_ to normal command-line options. Meaning that they will take +# precedence over the command-specific ADDITIONAL_GUIX__FLAGS. +# +# This seems like a poor user experience. Thus we check for GUIX_BUILD_OPTIONS's +# existence here and direct users of this script to use our (more flexible) +# custom environment variables. +if [ -n "$GUIX_BUILD_OPTIONS" ]; then +cat << EOF +Error: Environment variable GUIX_BUILD_OPTIONS is not empty: + '$GUIX_BUILD_OPTIONS' + +Unfortunately this script is incompatible with GUIX_BUILD_OPTIONS, please unset +GUIX_BUILD_OPTIONS and use ADDITIONAL_GUIX_COMMON_FLAGS to set build options +across guix commands or ADDITIONAL_GUIX__FLAGS to set build options for a +specific guix command. + +See contrib/guix/README.md for more details. +EOF +exit 1 +fi + +################ +# Check 3: Make sure that we're not in a dirty worktree +################ +if ! git diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then +cat << EOF +ERR: The current git worktree is dirty, which may lead to broken builds. + + Aborting... + +Hint: To make your git worktree clean, You may want to: + 1. Commit your changes, + 2. Stash your changes, or + 3. Set the 'FORCE_DIRTY_WORKTREE' environment variable if you insist on + using a dirty worktree +EOF +exit 1 +else + GIT_COMMIT=$(git rev-parse --short=12 HEAD) +fi + +################ +# Check 4: Make sure that build directories do no exist +################ + +# Default to building for all supported HOSTs (overridable by environment) +export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu + x86_64-w64-mingw32}" + +DISTSRC_BASE="${DISTSRC_BASE:-${PWD}}" + +# Usage: distsrc_for_host HOST +# +# HOST: The current platform triple we're building for +# +distsrc_for_host() { + echo "${DISTSRC_BASE}/distsrc-${GIT_COMMIT}-${1}" +} + +# Accumulate a list of build directories that already exist... +hosts_distsrc_exists="" +for host in $HOSTS; do + if [ -e "$(distsrc_for_host "$host")" ]; then + hosts_distsrc_exists+=" ${host}" + fi +done + +if [ -n "$hosts_distsrc_exists" ]; then +# ...so that we can print them out nicely in an error message +cat << EOF +ERR: Build directories for this commit already exist for the following platform + triples you're attempting to build, probably because of previous builds. + Please remove, or otherwise deal with them prior to starting another build. + + Aborting... + +EOF +for host in $hosts_distsrc_exists; do + echo " ${host} '$(distsrc_for_host "$host")'" +done +exit 1 +else + mkdir -p "$DISTSRC_BASE" +fi + +######### +# Setup # +######### + # Determine the maximum number of jobs to run simultaneously (overridable by # environment) MAX_JOBS="${MAX_JOBS:-$(nproc)}" @@ -16,11 +126,23 @@ SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}" # Execute "$@" in a pinned, possibly older version of Guix, for reproducibility # across time. time-machine() { + # shellcheck disable=SC2086 guix time-machine --url=https://github.com/dongcarl/guix.git \ --commit=b066c25026f21fb57677aa34692a5034338e7ee3 \ + --max-jobs="$MAX_JOBS" \ + ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \ + ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \ -- "$@" } +# Make sure an output directory exists for our builds +OUTDIR="${OUTDIR:-${PWD}/output}" +[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR" + +######### +# Build # +######### + # Function to be called when building for host ${1} and the user interrupts the # build int_trap() { @@ -38,9 +160,9 @@ and untracked files and directories will be wiped, allowing you to start anew. EOF } -# Deterministically build Bitcoin Core for HOSTs (overridable by environment) +# Deterministically build Bitcoin Core # shellcheck disable=SC2153 -for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu x86_64-w64-mingw32}; do +for host in $HOSTS; do # Display proper warning when the user interrupts the build trap 'int_trap ${host}' INT @@ -50,6 +172,19 @@ for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv # for the particular $HOST we're building for export HOST="$host" + # shellcheck disable=SC2030 +cat << EOF +INFO: Building commit ${GIT_COMMIT:?not set} for platform triple ${HOST:?not set}: + ...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set} + ...running at most ${MAX_JOBS:?not set} jobs + ...from worktree directory: '${PWD}' + ...bind-mounted in container to: '/bitcoin' + ...in build directory: '$(distsrc_for_host "$HOST")' + ...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")' + ...outputting in: '${OUTDIR:?not set}' + ...bind-mounted in container to: '/outdir' +EOF + # Run the build script 'contrib/guix/libexec/build.sh' in the build # container specified by 'contrib/guix/manifest.scm'. # @@ -99,20 +234,36 @@ for host in ${HOSTS=x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv # make the downloaded depends sources available to it. The sources # should have been downloaded prior to this invocation. # - # shellcheck disable=SC2086 + # ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} + # + # fetch substitute from SUBSTITUTE_URLS if they are + # authorized + # + # Depending on the user's security model, it may be desirable to use + # substitutes (pre-built packages) from servers that the user trusts. + # Please read the README.md in the same directory as this file for + # more information. + # + # shellcheck disable=SC2086,SC2031 time-machine environment --manifest="${PWD}/contrib/guix/manifest.scm" \ --container \ --pure \ --no-cwd \ --share="$PWD"=/bitcoin \ + --share="$DISTSRC_BASE"=/distsrc-base \ + --share="$OUTDIR"=/outdir \ --expose="$(git rev-parse --git-common-dir)" \ ${SOURCES_PATH:+--share="$SOURCES_PATH"} \ - ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \ + --max-jobs="$MAX_JOBS" \ + ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \ + ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \ -- env HOST="$host" \ MAX_JOBS="$MAX_JOBS" \ SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \ ${V:+V=1} \ ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \ + DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \ + OUTDIR=/outdir \ bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh" ) diff --git a/contrib/guix/libexec/build.sh b/contrib/guix/libexec/build.sh index d658c4f6a67..b00c42ce012 100644 --- a/contrib/guix/libexec/build.sh +++ b/contrib/guix/libexec/build.sh @@ -11,9 +11,15 @@ if [ -n "$V" ]; then export VERBOSE="$V" fi -# Check that environment variables assumed to be set by the environment are set -echo "Building for platform triple ${HOST:?not set} with reference timestamp ${SOURCE_DATE_EPOCH:?not set}..." -echo "At most ${MAX_JOBS:?not set} jobs will run at once..." +# Check that required environment variables are set +cat << EOF +Required environment variables as seen inside the container: + HOST: ${HOST:?not set} + SOURCE_DATE_EPOCH: ${SOURCE_DATE_EPOCH:?not set} + MAX_JOBS: ${MAX_JOBS:?not set} + DISTSRC: ${DISTSRC:?not set} + OUTDIR: ${OUTDIR:?not set} +EOF ##################### # Environment Setup # @@ -23,19 +29,6 @@ echo "At most ${MAX_JOBS:?not set} jobs will run at once..." # $HOSTs after successfully building. BASEPREFIX="${PWD}/depends" -# Setup an output directory for our build -OUTDIR="${OUTDIR:-${PWD}/output}" -[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR" - -# Setup the directory where our Bitcoin Core build for HOST will occur -DISTSRC="${DISTSRC:-${PWD}/distsrc-${HOST}}" -if [ -e "$DISTSRC" ]; then - echo "DISTSRC directory '${DISTSRC}' exists, probably because of previous builds... Aborting..." - exit 1 -else - mkdir -p "$DISTSRC" -fi - # Given a package name and an output name, return the path of that output in our # current guix environment store_path() { @@ -189,6 +182,7 @@ esac # Make $HOST-specific native binaries from depends available in $PATH export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}" +mkdir -p "$DISTSRC" ( cd "$DISTSRC" diff --git a/doc/guix.md b/doc/guix.md new file mode 100644 index 00000000000..fa9c74f4860 --- /dev/null +++ b/doc/guix.md @@ -0,0 +1,3 @@ +# Bootstrappable Bitcoin Core Builds + +See [contrib/guix/README.md](../contrib/guix/README.md) diff --git a/share/genbuild.sh b/share/genbuild.sh index d8f3429f7a9..fe89ae1fdef 100755 --- a/share/genbuild.sh +++ b/share/genbuild.sh @@ -31,7 +31,7 @@ if [ "${BITCOIN_GENBUILD_NO_GIT}" != "1" ] && [ -e "$(command -v git)" ] && [ "$ fi # otherwise generate suffix from git, i.e. string like "59887e8-dirty" - GIT_COMMIT=$(git rev-parse --short HEAD) + GIT_COMMIT=$(git rev-parse --short=12 HEAD) git diff-index --quiet HEAD -- || GIT_COMMIT="$GIT_COMMIT-dirty" fi