bitcoin/contrib/macdeploy
fanquake 1ad5d5088d
Merge bitcoin/bitcoin#24866: build: No longer need to hack the PATH variable in config.site
efa3a807a6 build: No longer need to hack the `PATH` variable in `config.site` (Hennadii Stepanov)
f3af4f7a18 build: Let the depends build system define a path to `dsymutil` tool (Hennadii Stepanov)
b0a8ddabe5 build: Pass missed darwin-specific tools via `config.site` (Hennadii Stepanov)
f87594da14 build: No need to provide defaults for darwin-specific tools (Hennadii Stepanov)
80cd99322f scripted-diff: Rename INSTALLNAMETOOL -> INSTALL_NAME_TOOL (Hennadii Stepanov)
a4fd440741 build: Pass missed `strip` tool via `config.site` (Hennadii Stepanov)

Pull request description:

  This PR adds lacking definitions of absolute paths to some tools in the depends build system.

  This improvement makes possible to keep the `PATH` variable untouched during configuration.

  Also see https://github.com/bitcoin/bitcoin/pull/24566#discussion_r851125442.

  #### Guix builds on `x86_64`:
  ```
  $ find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
  93fa58bf2a1f5c15c5a547c014036ac79761e5c9622bd5099408ce570b39f02f  guix-build-efa3a807a677/output/aarch64-linux-gnu/SHA256SUMS.part
  ef5f53348404ba973dceaf088a4d47dfd0f1fa3d3bf75bdd723b043431de005d  guix-build-efa3a807a677/output/aarch64-linux-gnu/bitcoin-efa3a807a677-aarch64-linux-gnu-debug.tar.gz
  6688b2295d564b32ea676c0540c43cdc5211322eddc87e0967b764284e847368  guix-build-efa3a807a677/output/aarch64-linux-gnu/bitcoin-efa3a807a677-aarch64-linux-gnu.tar.gz
  4abb3428be477aa7360611689fd28950f30dbbac6a95c454095367d8df11ad72  guix-build-efa3a807a677/output/arm-linux-gnueabihf/SHA256SUMS.part
  8dd33389170e83812821d5dd68741db96af1376035ba40af0215a7ae95dcf7fc  guix-build-efa3a807a677/output/arm-linux-gnueabihf/bitcoin-efa3a807a677-arm-linux-gnueabihf-debug.tar.gz
  544e97eb88b2a44c8ceb9660399eb5d49d75e07ff59fc03a701a595bacea0491  guix-build-efa3a807a677/output/arm-linux-gnueabihf/bitcoin-efa3a807a677-arm-linux-gnueabihf.tar.gz
  219faf1131bdcffffd5979eafd2beabc4a300081f8b1df184852b7183dfbc0e8  guix-build-efa3a807a677/output/arm64-apple-darwin/SHA256SUMS.part
  602a417bfa7971fb26d0fe9921d2348fd0d01a5bcf0af93f8a9d50112076e0eb  guix-build-efa3a807a677/output/arm64-apple-darwin/bitcoin-efa3a807a677-arm64-apple-darwin-unsigned.dmg
  0dcb197420844da8da3f528a1d986628f7b63adb1e83353d63e8a84da59abc42  guix-build-efa3a807a677/output/arm64-apple-darwin/bitcoin-efa3a807a677-arm64-apple-darwin-unsigned.tar.gz
  10efdd33418234a8288c27a614f50e9ff45efbd681fa1c0e173142b6d267cdb8  guix-build-efa3a807a677/output/arm64-apple-darwin/bitcoin-efa3a807a677-arm64-apple-darwin.tar.gz
  e5ef02adeb9bcb4675972b0dc233a904006b0940d721049eeb94b14cda34872e  guix-build-efa3a807a677/output/dist-archive/bitcoin-efa3a807a677.tar.gz
  a75d2a49b7a8ab1c849e1badff5049a417519f0154b65335a8717d01b8b6ed62  guix-build-efa3a807a677/output/powerpc64-linux-gnu/SHA256SUMS.part
  aef63d196487dd0bc597fd53633ac889149f1a126c651ea55f3bec11a092c460  guix-build-efa3a807a677/output/powerpc64-linux-gnu/bitcoin-efa3a807a677-powerpc64-linux-gnu-debug.tar.gz
  ddc7913eed26270be271a8712bf351d562d57c746810ea3b7b4101aea8cb6d89  guix-build-efa3a807a677/output/powerpc64-linux-gnu/bitcoin-efa3a807a677-powerpc64-linux-gnu.tar.gz
  83bbf4b1af07a2cf7d6014de7c885f0998dd38afacdb5242f5f56505ee704f17  guix-build-efa3a807a677/output/powerpc64le-linux-gnu/SHA256SUMS.part
  c9d5d95de98ed987b63a78d4f6e082cb36d5ec3ba71d130601e03d3ebfbd208d  guix-build-efa3a807a677/output/powerpc64le-linux-gnu/bitcoin-efa3a807a677-powerpc64le-linux-gnu-debug.tar.gz
  88ed6ec82dd4c1c656fe80b4c49f91a4c15c2ab798dbbe16a3d57393f17d6f3a  guix-build-efa3a807a677/output/powerpc64le-linux-gnu/bitcoin-efa3a807a677-powerpc64le-linux-gnu.tar.gz
  7b8e7b3b1e68a2ea0e37c058b284da11e9721ef4d1bc2761ed003b2061358d5f  guix-build-efa3a807a677/output/riscv64-linux-gnu/SHA256SUMS.part
  a5de4bedf2b4bc5ab25db21b942076897cabe8a40ce9b0637488af6b4d90693a  guix-build-efa3a807a677/output/riscv64-linux-gnu/bitcoin-efa3a807a677-riscv64-linux-gnu-debug.tar.gz
  53023994202887778a001ce00daf7cdc135b9e6c3be034f31645ab4ba5f078c6  guix-build-efa3a807a677/output/riscv64-linux-gnu/bitcoin-efa3a807a677-riscv64-linux-gnu.tar.gz
  c6664a61b81dfa080c466d2252a6db70165acbea6cfad51ada16970e9c08bb6f  guix-build-efa3a807a677/output/x86_64-apple-darwin/SHA256SUMS.part
  a528569ae4bf5e19401311649086a2d8e3fa5251b44550e623722968dfb111ea  guix-build-efa3a807a677/output/x86_64-apple-darwin/bitcoin-efa3a807a677-x86_64-apple-darwin-unsigned.dmg
  9b0384cce7605b546ed581074955f2b9c33cf0817453842036e6224b423b814b  guix-build-efa3a807a677/output/x86_64-apple-darwin/bitcoin-efa3a807a677-x86_64-apple-darwin-unsigned.tar.gz
  ebdbd2f3a6406233f27ee48be0ab014991fedba3c0831f79f4a4873f7abf3d7a  guix-build-efa3a807a677/output/x86_64-apple-darwin/bitcoin-efa3a807a677-x86_64-apple-darwin.tar.gz
  05a8f71fe67f7193e71ea8bbe6f8df2e651b8ac7da3075ba25aacdd3515f7757  guix-build-efa3a807a677/output/x86_64-linux-gnu/SHA256SUMS.part
  30a17a1e3d795ea390cd1e0f3ef74c989b5768ae7415740fcca46befe4cb7206  guix-build-efa3a807a677/output/x86_64-linux-gnu/bitcoin-efa3a807a677-x86_64-linux-gnu-debug.tar.gz
  66db846f3fd739089afa5c339659dbf5efb50572f2d29f8288bf24be9e8f1dd0  guix-build-efa3a807a677/output/x86_64-linux-gnu/bitcoin-efa3a807a677-x86_64-linux-gnu.tar.gz
  bd3c44890823badcf6d296fa674de14275684be7593f4ab21c0316873ddd8652  guix-build-efa3a807a677/output/x86_64-w64-mingw32/SHA256SUMS.part
  6ce3ee21212ff2a95e085073a48194476ade2d5ff94cc1c8ec58a8ae7db8f1fa  guix-build-efa3a807a677/output/x86_64-w64-mingw32/bitcoin-efa3a807a677-win64-debug.zip
  5e697c05537cfb2ce2ed95fef25e261e2cfa83a31fd548a98118580c4bbff2e4  guix-build-efa3a807a677/output/x86_64-w64-mingw32/bitcoin-efa3a807a677-win64-setup-unsigned.exe
  4cc2bcff98845c792c0ed12a2ea407b25fb85b2d4250d88dca94ed68f42e714d  guix-build-efa3a807a677/output/x86_64-w64-mingw32/bitcoin-efa3a807a677-win64-unsigned.tar.gz
  9d5d72271dc6b820e63b30c5c3f9015309777793100b4e2b6ab0c8ea0f7b4aed  guix-build-efa3a807a677/output/x86_64-w64-mingw32/bitcoin-efa3a807a677-win64.zip
  ```

ACKs for top commit:
  laanwj:
    Tested ACK efa3a807a6. I get the same build output as in OP:
  vincenzopalazzo:
    Re ACK efa3a807a6

Tree-SHA512: 6d35c11fc307221d61ad250bbdcdc09dbc49adbe43f7a94acb56190ae9f005d23fc22941ea59e3eb62811f8974e39d3617e0c47071232d4b1b0bc2e2e2782e88
2022-05-05 09:36:26 +01:00
..
background.tiff build: use a static .tiff for macOS .dmg over generating 2022-01-02 15:38:19 +08:00
detached-sig-create.sh build, mac: Include arch in codesignature tarball 2022-03-09 10:38:24 -05:00
gen-sdk contrib: macdeploy: fix permissions typo in gen-sdk script 2022-04-22 22:09:44 +02:00
LICENSE Added a copy of the GPLv3 to macdeployqtplus 2011-11-02 19:26:32 +01:00
macdeployqtplus scripted-diff: Rename INSTALLNAMETOOL -> INSTALL_NAME_TOOL 2022-04-21 10:08:45 +02:00
README.md contrib: macdeploy: fix permissions typo in gen-sdk script 2022-04-22 22:09:44 +02:00

MacOS Deployment

The macdeployqtplus script should not be run manually. Instead, after building as usual:

make deploy

When complete, it will have produced Bitcoin-Core.dmg.

SDK Extraction

Step 1: Obtaining Xcode.app

A free Apple Developer Account is required to proceed.

Our current macOS SDK (Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz) can be extracted from Xcode_12.2.xip.

Alternatively, after logging in to your account go to 'Downloads', then 'More' and search for Xcode_12.2.

An Apple ID and cookies enabled for the hostname are needed to download this.

The sha256sum of the downloaded XIP archive should be 28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0.

After Xcode version 7.x, Apple started shipping the Xcode.app in a .xip archive. This makes the SDK less-trivial to extract on non-macOS machines. One approach (tested on Debian Buster) is outlined below:

# Install/clone tools needed for extracting Xcode.app
apt install cpio
git clone https://github.com/bitcoin-core/apple-sdk-tools.git

# Unpack Xcode_12.2.xip and place the resulting Xcode.app in your current
# working directory
python3 apple-sdk-tools/extract_xcode.py -f Xcode_12.2.xip | cpio -d -i

On macOS the process is more straightforward:

xip -x Xcode_12.2.xip

Step 2: Generating Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz from Xcode.app

To generate Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz, run the script gen-sdk with the path to Xcode.app (extracted in the previous stage) as the first argument.

# Generate a Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz from
# the supplied Xcode.app
./contrib/macdeploy/gen-sdk '/path/to/Xcode.app'

The sha256sum of the generated TAR.GZ archive should be df75d30ecafc429e905134333aeae56ac65fac67cb4182622398fd717df77619.

Deterministic macOS DMG Notes

Working macOS DMGs are created in Linux by combining a recent clang, the Apple binutils (ld, ar, etc) and DMG authoring tools.

Apple uses clang extensively for development and has upstreamed the necessary functionality so that a vanilla clang can take advantage. It supports the use of -F, -target, -mmacosx-version-min, and -isysroot, which are all necessary when building for macOS.

Apple's version of binutils (called cctools) contains lots of functionality missing in the FSF's binutils. In addition to extra linker options for frameworks and sysroots, several other tools are needed as well such as install_name_tool, lipo, and nmedit. These do not build under Linux, so they have been patched to do so. The work here was used as a starting point: mingwandroid/toolchain4.

In order to build a working toolchain, the following source packages are needed from Apple: cctools, dyld, and ld64.

These tools inject timestamps by default, which produce non-deterministic binaries. The ZERO_AR_DATE environment variable is used to disable that.

This version of cctools has been patched to use the current version of clang's headers and its libLTO.so rather than those from llvmgcc, as it was originally done in toolchain4.

To complicate things further, all builds must target an Apple SDK. These SDKs are free to download, but not redistributable. See the SDK Extraction notes above for how to obtain it.

The Guix process builds 2 sets of files: Linux tools, then Apple binaries which are created using these tools. The build process has been designed to avoid including the SDK's files in Guix's outputs. All interim tarballs are fully deterministic and may be freely redistributed.

xorrisofs is used to create the DMG.

A background image is added to DMG files by inserting a .DS_Store during creation.

As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in order to satisfy the new Gatekeeper requirements. Because this private key cannot be shared, we'll have to be a bit creative in order for the build process to remain somewhat deterministic. Here's how it works:

  • Builders use Guix to create an unsigned release. This outputs an unsigned DMG which users may choose to bless and run. It also outputs an unsigned app structure in the form of a tarball, which also contains all of the tools that have been previously (deterministically) built in order to create a final DMG.
  • The Apple keyholder uses this unsigned app to create a detached signature, using the script that is also included there. Detached signatures are available from this repository.
  • Builders feed the unsigned app + detached signature back into Guix. It uses the pre-built tools to recombine the pieces into a deterministic DMG.