Commit Graph

164 Commits

Author SHA1 Message Date
fanquake
8f76f9f613
Merge bitcoin/bitcoin#22531: guix: Fixes to guix-{attest,verify}
9b313dfef1 guix: Ensure EPOCH_SOURCE_DATE does not include GPG information (Andrew Chow)
43225f0a2a guix: Remove extra \r from all.SHA256SUMS line ending (Andrew Chow)
d080c27066 guix, doc: Add a note that codesigners need to rebuild after tagging (Andrew Chow)
4a466388a0 guix: Allow changing the base manifest in guix-verify (Andrew Chow)
33455c7696 guix: Make all.SHA256SUMS rather than codesigned.SHA256SUMS (Andrew Chow)

Pull request description:

  `guix-verify` expects `all.SHA256SUMS` but `guix-attest` produces `codesigned.SHA256SUMS`. Since `all.SHA256SUMS` makes more sense (as the file contains all the sha256sums, not just the codesigned ones), `guix-attest` has been changed to output a file of that name.

  As a quality of life improvement, `guix-verify` can take `SIGNER` and use the signer's manifest as the base to compare against. This makes it easier to compare a single person's attestations with everyone else's and can make it more obvious when one builder is clearly mismatching with everyone else.

  Lastly `release-process.md` is updated with a note about a gotcha that can cause a mismatch in the codesigned attestation.

ACKs for top commit:
  fanquake:
    ACK 9b313dfef1

Tree-SHA512: 0d60627def38288dbd3059ad1e72cad224f9205da11b1a561c082ef28250a074df5cc5f2797c91a7be027bc486a3fda3319c2e496a8724e5b539337236c6f990
2021-07-29 11:13:12 +08:00
Andrew Chow
9b313dfef1 guix: Ensure EPOCH_SOURCE_DATE does not include GPG information
If the user has set log.showSignature=true in their git config, then the
git log will always output GPG signature information. Since git log is
used to set EPOCH_SOURCE_DATE, this will mistakenly have GPG signature
information in it which causes issues for the build. To avoid this
issue, we override the config and force log.showSignature=false.
2021-07-28 12:59:47 -04:00
Andrew Chow
43225f0a2a guix: Remove extra \r from all.SHA256SUMS line ending
guix-attest mistakenly added an extra \r to the line endings in
all.SHA256SUMS, causing guix-verify to erroneously fail.

Co-Authored-By: Carl Dong <contact@carldong.me>
2021-07-28 12:59:47 -04:00
jonatack
198ceb82f9 script, doc: guix touchups 2021-07-23 21:23:37 +05:30
h
d7b7f610a5 Updated Readme, Corrected the codesign typo 2021-07-23 20:15:16 +05:30
fanquake
2b5563bb1e
Merge bitcoin/bitcoin#22533: guix/build: Remove vestigial SKIPATTEST.TAG
9f01feda0a guix/build: Remove vestigial SKIPATTEST.TAG (Carl Dong)

Pull request description:

  No longer needed or referenced by anything. A relic from prior to the great hierarchy overhaul of #22182

ACKs for top commit:
  achow101:
    ACK 9f01feda0a
  fanquake:
    ACK 9f01feda0a

Tree-SHA512: a94cf63f0c5cb8dbacf1025b6c0e81b219c2a3c93b3cbcefc239ccde29e602ecd4b717b1d93dbe53cb791a5017236fb09823c034aec42b0c31894fc9e0ab8b21
2021-07-23 14:03:24 +08:00
Andrew Chow
4a466388a0 guix: Allow changing the base manifest in guix-verify
When verifying guix attestations, it is useful to set a particular
signer's manifest as the base to compare against.
2021-07-22 18:48:25 -04:00
Andrew Chow
33455c7696 guix: Make all.SHA256SUMS rather than codesigned.SHA256SUMS 2021-07-22 17:57:35 -04:00
Carl Dong
9f01feda0a guix/build: Remove vestigial SKIPATTEST.TAG 2021-07-22 17:47:22 -04:00
Carl Dong
a884a1edcd guix/INSTALL: Misc fixups 2021-07-22 17:42:46 -04:00
Carl Dong
3c4d2c418e guix: Silence getent(1) invocation 2021-07-20 19:47:28 -04:00
fanquake
201c5e4aec
Merge bitcoin/bitcoin#22465: guix: Pin kernel-header version, time-machine to upstream 1.3.0 commit
e6a94d4446 guix: Bump to version-1.3.0 from upstream (Carl Dong)
90fd13b954 guix: Pin kernel header version (Carl Dong)

Pull request description:

  ```
  - Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
  - Use 4.9 for all others (second-oldest LTS release, released in
    combination with glibc glibc 2.24 in Debian stretch)
  ```

  ```
  The chosen commit is the HEAD of Guix's version-1.3.0 branch as of July
  15th, 2021.

  Also fix visual indenting.
  ```

  -----

  This + the documentation PR should make our Guix system ready for release!

ACKs for top commit:
  MarcoFalke:
    review ACK e6a94d4446 to change to vanilla guix. Did not review the kernel change.
  laanwj:
    ACK e6a94d4446
  fanquake:
    ACK e6a94d4446

Tree-SHA512: a175e4ddb3ee786a39f5e800ce336932ad2f6797a3a28400a6f723875d0f19833fd36cedc41b3580e4604110517211bd9f557be36adf7265fd8e591c434ae032
2021-07-20 12:23:56 +08:00
fanquake
e7441a6a45
Merge bitcoin/bitcoin#21711: guix: Add full installation and usage documentation
fac4814106 doc/release-process: Add torrent creation details (Carl Dong)
5d24cc3d82 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong)
5da2ee49d5 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong)
318c60700b guix: Adapt release-process.md to new Guix process (Carl Dong)
fcab35b229 guix-attest: Produce and sign normalized documents (Carl Dong)
c2541fd0ca guix: Overhaul README (Carl Dong)
46ce6ce378 tree-wide: Rename gitian-keys to builder-keys (Carl Dong)
fc4f8449f3 guix: Update various check_tools lists (Carl Dong)
263220a85c guix: Check for a sane services database (Carl Dong)

Pull request description:

  Based on: #21462

  Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier.

  Current TODOs:
  - [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up
  - [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login
  - [x] Fanquake docker option: complete section
  - [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section
  - [x] Building from source option: Insert dependency tree diagram that I made
  - [x] Building from source option: redo sectioning, kind of a mess right now
  - [x] Optional setup: make clear which parts are only needed if building from source
  - [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key
  - [x] Overall (after everything): Make the links work.

  Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option.

ACKs for top commit:
  fanquake:
    ACK fac4814106 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release.

Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
2021-07-20 11:09:39 +08:00
Carl Dong
5d24cc3d82 guix/INSTALL: Guix installs init scripts in libdir 2021-07-19 18:24:36 -04:00
Carl Dong
5da2ee49d5 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting 2021-07-19 18:24:36 -04:00
Carl Dong
fcab35b229 guix-attest: Produce and sign normalized documents
That way we can easily combine the document and detached signature to
produce cleartext signature files for upload during the release process.

See subsequent commits which modify doc/release-process.md for more
details.
2021-07-19 18:24:36 -04:00
Carl Dong
c2541fd0ca guix: Overhaul README
- Added detailed Guix bootstrap/installation instructions
2021-07-19 18:24:36 -04:00
fanquake
4371e635d6
Merge bitcoin/bitcoin#22410: Avoid GCC 7.1 ABI change warning in guix build
1edddf5de4 Avoid GCC 7.1 ABI change warning in guix build (Pieter Wuille)

Pull request description:

  The arm-linux-gnueabihf guix build output is littered with warnings like:

  ```
  /gnu/store/7a96hdqdb2qi8a39f09n84xjy2hr23rs-gcc-cross-arm-linux-gnueabihf-8.4.0/include/c++/bits/stl_vector.h:1085:4: note:
                   parameter passing for argument of type '__gnu_cxx::__normal_iterator<CRecipient*, std::vector<CRecipient> >' changed in GCC 7.1
  ```

  These are irrelevant for us. Disable them using `-Wno-psabi`.

ACKs for top commit:
  laanwj:
    ACK 1edddf5de4
  hebasto:
    ACK 1edddf5de4, after thorough reading related materials, I agree this change can be merged. As I mentioned above, I have been compiling my arm-32bit binaries with `-Wno-psabi` flag for two years, and no related flaws were observed.

Tree-SHA512: 485c7500547ac5da567ad23847341c18ff832607f5a1002676404cc647e437cf3445b6894ecff5b52929ca52bea946c06bd90eace1997c895e56204e787065e4
2021-07-18 14:33:22 +08:00
Carl Dong
e6a94d4446 guix: Bump to version-1.3.0 from upstream
The chosen commit is the HEAD of Guix's version-1.3.0 branch as of July
15th, 2021.

Also fix visual indenting.
2021-07-15 21:50:48 -04:00
Carl Dong
90fd13b954 guix: Pin kernel header version
- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
- Use 4.9 for all others (second-oldest LTS release, released in
  combination with glibc glibc 2.24 in Debian stretch)
2021-07-15 21:47:57 -04:00
Carl Dong
fc4f8449f3 guix: Update various check_tools lists 2021-07-08 11:29:26 -04:00
Carl Dong
263220a85c guix: Check for a sane services database
On bare systems, it is possible to be lacking a services database. Check
for basic entries before attempting a build.

See the error message in the diff for more context.
2021-07-08 11:28:07 -04:00
Carl Dong
5b4703c6a7
guix: Test security-check sanity before performing them 2021-07-07 20:29:05 +08:00
Carl Dong
678348db51
guix: Patch binutils to add security-related disable flags
We use these flags in our test-security-check make target, but they are
only available because debian patches them in.

We can patch them in for our Guix builds so that we can check the sanity
of our security/symbol checking suite before running them.
2021-07-07 19:31:37 +08:00
Pieter Wuille
1edddf5de4 Avoid GCC 7.1 ABI change warning in guix build 2021-07-06 10:41:43 -07:00
fanquake
15fc9a0299
guix: add additional documentation to patches 2021-07-06 20:50:47 +08:00
fanquake
de6ca41a52
guix: no-longer pass --enable-glibc-back-compat to Guix
Now that our Guix builds are performed on glibc 2.24 and 2.27 (RISCV),
we no-longer need to pass the --enable-glibc-back-compat option.

Replace it with --disable-threadlocal, to prevent the usage of symbols
from glibc 2.18.

None of the binaries produced required symbols later than 2.17, and 2.27
(RISCV).
2021-07-06 20:50:40 +08:00
Carl Dong
647f7e5f1d guix: Also sort SHA256SUMS.part 2021-07-02 14:10:33 -04:00
Carl Dong
dc4137a60c guix: Build depends/qt with our platform definition
Our 'bitcoin-linux-g++' definition better integrates with our depends
system than the stock linux-g++-64 definition.

This fixes a bug whereby Guix builds on x86_64 for x86_64 did not
produce a QMinimalIntegrationPlugin and led to bitcoin-qt not being
built.
2021-07-01 16:20:28 -04:00
Carl Dong
16b0a936e1 guix: Rebase toolchain on glibc 2.24 (2.27 for riscv64)
Support for riscv64 in glibc landed in 2.27 so it's unavoidable that we
use 2.27.

Running a Bitcoin build with toolchains based on 2.24 for platforms
other than riscv64 seem to produce binaries which do not have 2.17
symbols. So use 2.24 since it's more recent and maintained by Debian
Stretch.
2021-07-01 16:17:03 -04:00
fanquake
da69d9965a
Merge bitcoin/bitcoin#21871: scripts: add checks for minimum required OS versions
aa80b5759d scripts: check macOS SDK version is set (fanquake)
c972345bac scripts: check minimum required Windows version is set (fanquake)
29615aef52 scripts: check minimum required macOS vesion is set (fanquake)
8732f7b6c9 scripts: LIEF 0.11.5 (fanquake)

Pull request description:

  macOS:
  We use a compile flag ([-mmacosx-version-min=10.14](https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/darwin.mk#L96)) to set the minimum required version of macOS needed to run our binaries. This adds a sanity check that the version is being set as expected.

  Clangs Darwin driver should infer the SDK version used during compilation, and forward that through to the linker. Add a check that this has been done, and the expected SDK version is set. Should help prevent issues like #21771 in future.

  Windows:
  We use linker flags ([-Wl,--major/minor-subsystem-version](https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L683)) to set the minimum required version of Windows needed to run our binaries. This adds a sanity check that the version is being set as expected.

  Gitian builds:
  ```bash
  # macOS:
  8b6fcd61d75001c37b2af3fceb5ae09f5d2fe85e97d361f684214bd91c27954a  bitcoin-f015e1c2cac9-osx-unsigned.dmg
  3c1e412bc7f5a7a5d0f78e2cd84b7096831414e1304c1307211aa3e135d89bbf  bitcoin-f015e1c2cac9-osx-unsigned.tar.gz
  50b7b2804e8481f63c69c78e3e8a71c0d811bf2db8895dd6d3edae9c46a738ae  bitcoin-f015e1c2cac9-osx64.tar.gz
  fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675  src/bitcoin-f015e1c2cac9.tar.gz
  8a20f21b20673dfc8c23e22b20ae0839bcaf65bf0e02f62381cdf5e7922936f0  bitcoin-core-osx-22-res.yml

  # Windows:
  b01fcdc2a5673387050d6c6c4f96f1d350976a121155fde3f76c2af309111f9d  bitcoin-f015e1c2cac9-win-unsigned.tar.gz
  b95bdcbef638804030671d2332d58011f8c4ed4c1db87d6ffd211515c32c9d02  bitcoin-f015e1c2cac9-win64-debug.zip
  350bf180252d24a3d40f05e22398fec7bb00e06d812204eb5a421100a8e10638  bitcoin-f015e1c2cac9-win64-setup-unsigned.exe
  2730ddabe246d99913c9a779e97edcadb2d55309933d46f1dffd0d23ecf9aae5  bitcoin-f015e1c2cac9-win64.zip
  fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675  src/bitcoin-f015e1c2cac9.tar.gz
  aa60d7a753e8cb2d4323cfbbf4d964ad3645e74c918cccd66862888f8646d80f  bitcoin-core-win-22-res.yml
  ```

ACKs for top commit:
  hebasto:
    ACK aa80b5759d, tested by breaking tests:

Tree-SHA512: 10150219910e8131715fbfe20edaa15778387616ef3bfe1a5152c7acd3958fe8f88c74961c3d3641074eb72824680c22764bb1dc01a19e92e946c2d4962a8d2c
2021-06-18 15:21:47 +08:00
fanquake
d50302625e
Merge bitcoin/bitcoin#22182: guix: Overhaul how guix-{attest,verify} works and hierarchy
e2c40a4ed5 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed5 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b839 Make SHA256SUMS fragment right after build (Carl Dong)

Pull request description:

  Based on:  #22075
  Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff

  The following changes resolve many usability improvements which were pointed out to me:
  1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
  2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
  3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
  4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.

  For those who sign on an separate machine than the one you do builds on, the following steps will work:
  1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
  2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
  3. Sign the `SHA256SUMS` files:
      ```bash
      for i in "<path-to-achow101>/*.SHA256SUMS"; do
          gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
      done
      ```
  5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`

  -----

  After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
  ```
  output
  └── x86_64-w64-mingw32
      ├── bitcoin-4e069f7589da-win64-debug.zip
      ├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
      ├── bitcoin-4e069f7589da-win64.zip
      ├── bitcoin-4e069f7589da-win-unsigned.tar.gz
      └── SHA256SUMS.part
  ```

  These `SHA256SUMS.part` fragments look something like:
  ```
  3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825  dist-archive/bitcoin-4e069f7589da.tar.gz
  def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b  x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
  643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e  x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
  a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288  x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
  fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c  x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
  ```

  Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level

  When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:

  1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
  3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs

  Then both files are signed, resulting in the following `guix.sigs` hierarchy:
  ```
  4e069f7589da/
  └── dongcarl
      ├── all.SHA256SUMS
      ├── all.SHA256SUMS.asc
      ├── noncodesigned.SHA256SUMS
      └── noncodesigned.SHA256SUMS.asc
  ```

ACKs for top commit:
  achow101:
    ACK e2c40a4ed5
  hebasto:
    ACK e2c40a4ed5, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.

Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
2021-06-17 13:10:37 +08:00
Carl Dong
e2c40a4ed5 guix-attest: Error out if SHA256SUMS is unexpected 2021-06-14 17:05:26 -04:00
fanquake
ce8276b531
Merge bitcoin/bitcoin#22190: Use latest signapple commit
683d197970 Use latest signapple commit (Andrew Chow)

Pull request description:

  Update gitian and guix to use the same latest signapple commit.

  Also changed guix to use the actual repo. The changes from the fork were incorporated upstream.

ACKs for top commit:
  fanquake:
    ACK 683d197970 - sanity checked that the updated package is built:

Tree-SHA512: a4981f8bbe33e6c5654632bc9b9f6f2f1e675741a19ac7296205e370f1e64a747101ecb632e0cc82a0134e4c2e9ce47b3f7b4d8c8f75f0f06dd069c078303759
2021-06-11 09:09:26 +08:00
fanquake
8732f7b6c9
scripts: LIEF 0.11.5 2021-06-10 10:40:52 +08:00
Carl Dong
4cc35daed5 Rewrite guix-{attest,verify} for new hier 2021-06-09 11:06:00 -04:00
Carl Dong
28a9c9b839 Make SHA256SUMS fragment right after build 2021-06-09 11:06:00 -04:00
fanquake
7cac26246a
Merge bitcoin/bitcoin#22075: guix: Misc leftover usability improvements
108a6be92a guix: Check for disk space availability before building (Carl Dong)
d7dec89091 guix: Remove dest if OUTDIR mv fails (Carl Dong)

Pull request description:

  There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log

  - Perform an early disk space check for `guix-build`
  - Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails

ACKs for top commit:
  laanwj:
    Tested ACK 108a6be92a
  achow101:
    ACK 108a6be92a

Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
2021-06-09 09:09:38 +08:00
Andrew Chow
683d197970 Use latest signapple commit
Update gitian and guix to use the same latest signapple commit
2021-06-08 16:46:56 -04:00
fanquake
8837f1ebde
Merge bitcoin/bitcoin#21654: build, qt: Make Qt rcc output always deterministic
a58868d201 build: Makes rcc output always deterministic (Hennadii Stepanov)

Pull request description:

  The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2.

  The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732).

  This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the
  `QT_RCC_SOURCE_DATE_OVERRIDE` needless.

  ---

  Also it improves interaction with ccache:

  On master (f6c44e999b):
  ```
  $ make && make clean && ccache --zero-stats && make && ccache --show-stats
  ...
  cache directory                     /home/hebasto/.ccache
  primary config                      /home/hebasto/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Sun Apr 11 15:45:43 2021
  stats zeroed                        Sun Apr 11 15:45:05 2021
  cache hit (direct)                   638
  cache hit (preprocessed)               0
  cache miss                             1
  cache hit rate                     99.84 %
  called for link                       10
  cleanups performed                     0
  files in cache                     20023
  cache size                          13.2 GB
  max cache size                      15.0 GB
  ```

  The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`.

  With this PR:
  ```
  $ make && make clean && ccache --zero-stats && make && ccache --show-stats
  ...
  cache directory                     /home/hebasto/.ccache
  primary config                      /home/hebasto/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Sun Apr 11 15:28:46 2021
  stats zeroed                        Sun Apr 11 15:28:21 2021
  cache hit (direct)                   639
  cache hit (preprocessed)               0
  cache miss                             0
  cache hit rate                    100.00 %
  called for link                       10
  cleanups performed                     0
  files in cache                     20012
  cache size                          13.2 GB
  max cache size                      15.0 GB
  ```

ACKs for top commit:
  fanquake:
    ACK a58868d201

Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
2021-06-03 21:25:30 +08:00
Carl Dong
108a6be92a guix: Check for disk space availability before building 2021-05-25 22:37:50 -04:00
Carl Dong
d7dec89091 guix: Remove dest if OUTDIR mv fails 2021-05-25 16:40:46 -04:00
Carl Dong
ee883201cf guix: repro: Sort find output in libtool for gcc-8
Otherwise the resulting .a static libraries (e.g. libstdc++.a) will not
be reproducible and end up making the Bitcoin binaries non-reproducible
as well.

See: https://reproducible-builds.org/docs/archives/#gnu-libtool
2021-05-19 15:29:10 -04:00
Carl Dong
38eb91eb06 guix: Add codesigning functionality 2021-05-13 15:41:56 -04:00
Carl Dong
bac2690e6f guix: Package codesigning tools 2021-05-13 15:41:56 -04:00
Carl Dong
0a2176d477 guix: Reindent existing manifest.scm 2021-05-13 15:41:56 -04:00
W. J. van der Laan
01624a7db3
Merge bitcoin/bitcoin#21462: guix: Add guix-{attest,verify} scripts
d420e5c1c0 guix-attest: Avoid incomplete sigdirs with ERR traps (Carl Dong)
feda2c8e31 guix: Skip attesting to dist-archive (Carl Dong)
d522d8006b guix: Attest to inputs in inputs.SHA256SUMS (Carl Dong)
f9e2960c01 guix: Construct $OUTDIR in ${DISTSRC}/output (Carl Dong)
022abc85fc guix: Minor quoting fix in libexec/build.sh (Carl Dong)
c83c4fa5b7 guix-attest: Allow skipping GPG signing with NO_SIGN (Carl Dong)
0e1c2e448c guix-attest: Use ascii-armor signatures (Carl Dong)
b5fd89c4c8 guix-attest: Only use cross-platform flags for find+xargs (Carl Dong)
5926432ba6 guix: Add guix-verify script (Carl Dong)
30daf76a97 guix: Add guix-attest script (Carl Dong)

Pull request description:

  Adds replacements for `gsign` and `gverify`.

  Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing.

ACKs for top commit:
  laanwj:
    Code review and tested ACK d420e5c1c0

Tree-SHA512: 93d82d201f4596eaea0e3825aa55b013dfb91790e6ccee79893833d37921513d7b4e735f0641103e1e2ea8308abe4cb6218b73160924708802f2e0e3f7f6caf1
2021-05-12 13:51:38 +02:00
Hennadii Stepanov
a58868d201
build: Makes rcc output always deterministic
The Qt Resource Compiler (rcc) has a command-line option
`--format-version` which has the default value 2.

The only difference from `--format-version 1` is adding a last modified
timestamp to the output file. That, in turn, forces us to use
`QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds.

This change makes rcc output always deterministic by using
`--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless. Also it improves interaction
with ccache.

Co-authored-by: fanquake <fanquake@gmail.com>
2021-05-10 18:31:49 +03:00
W. J. van der Laan
1b9a5236e9
Merge bitcoin/bitcoin#21740: test: add new python linter to check file names and permissions
46b025e00d test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7 test: fix file permissions on various scripts (windsok)

Pull request description:

  Adds a new python linter test which tests for correct filenames and file permissions in the repository.

  Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050

  Summary of tests:
  * Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.

  * Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)

  * Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.

  * Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)

  * Checks every file that contains a shebang line to ensure it has an executable permission

  Additionally updates the permissions on various files to comply with the new tests.

  Fixes #21729

ACKs for top commit:
  practicalswift:
    cr re-ACK 46b025e00d: patch still looks correct
  kiminuo:
    code review ACK 46b025e00d if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
  laanwj:
    Code review ACK 46b025e00d

Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
2021-05-05 17:14:22 +02:00
W. J. van der Laan
b7c2625703
Merge bitcoin/bitcoin#21664: contrib: use LIEF for macOS and Windows symbol & security checks
7fc5e865b9 test: install lief in CI (fanquake)
955140b326 contrib: consolidate PIE and NX security checks (fanquake)
2aa1631822 contrib: use LIEF in PE symbol checks (fanquake)
e93ac26b85 contrib: use LIEF in macOS symbol checks (fanquake)
a632cbcee5 contrib: use f strings in symbol-check.py (fanquake)
0f5d77c8e4 contrib: add PE PIE check to security checks (fanquake)
8e1f40dd9a contrib: use LIEF for PE security checks (fanquake)
a25b2e965c contrib: use LIEF for macOS security checks (fanquake)
7e7eae7aa8 contrib: use f strings in security-check.py (fanquake)
2e7a9f7ade guix: install LIEF in Guix container (fanquake)
465967b5ef gitian: install LIEF in gitian container (fanquake)

Pull request description:

  This PR is a proof of concept for using [LIEF](https://github.com/lief-project/LIEF) for the PE and MACHO symbol and security checks. It replaces our current approach of manually parsing the output of `objdump` & `otool`. If the consensus is that using LIEF is ok, then I also plan on replacing [pixie.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/devtools/pixie.py), and using LIEF for all checks. LIEF for Linux is also currently blocked (on the next release, unless we want to build master) on one change for RISC-V that I [sent upstream](https://github.com/lief-project/LIEF/pull/562).

  LIEF is seemingly well maintained, and is the basis for a number of other tools. It also has some very nice documentation; i.e the [Python API for ELF](https://lief.quarkslab.com/doc/latest/api/python/elf.html). It also has many builtins we can take advantage of. i.e [`is_pie`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.is_pie), [`has_nx`](https://lief.quarkslab.com/doc/latest/api/python/macho.html#lief.MachO.Binary.has_nx) etc. This means we can [consolidate some of our checks](9c5eeb5484). If/when end up using LIEF for lightning then we can consolidate further, and cleanup these scripts. i.e to not parse the binary inside the checks, but once at the start of the script.

  Guix builds:
  ```bash
  # find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
  963a08638c46f9a3d75cd4b0c155d1ca091bbeba27167291adcd3dca03fd4c3d  guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
  a3ce927c46b103789a010c41a6ebfafe4548d90ee7d88f2a735c9183b775da5c  guix-build-f51237d94d98/output/aarch64-linux-gnu/bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
  2503ac8901068805d5e7251fd5cfeb7c1f8ba3528bdfcf3aa1e0c40bfd5c1cbc  guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
  5798697e58e1788df85aa9e2e4d33fef0456169fcbd2521f13b3b5806ac0d84d  guix-build-f51237d94d98/output/arm-linux-gnueabihf/bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
  4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2  guix-build-f51237d94d98/output/dist-archive/bitcoin-f51237d94d98.tar.gz
  9b4b8756c5c84295eb6b61b6b32a07a8d07723fb38aaa8f519b6133935061bda  guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
  cbd821aa464a9c16f7979dbec1a5e66939e777a567f55f7081499a8d528d42c5  guix-build-f51237d94d98/output/powerpc64-linux-gnu/bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
  abed530a82e97e3cf621c90a13c0881b0e39ccce2a6f42a3ff80de76e2abc5f7  guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
  8b6d2bdd8b58ff1f6072bf8693abe3ce773ff3a7d8d2b7218207e69945b9d31b  guix-build-f51237d94d98/output/powerpc64le-linux-gnu/bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
  d99cc705032d22ae819975992216899ed960ba25871a05c8789d00b80418511f  guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
  5240ca4f4ef7c62088185224ac319ad9a4a9b40075df10af18d8a6355bca32fb  guix-build-f51237d94d98/output/riscv64-linux-gnu/bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
  adc16eaee4b51e8615ce8b3be9f6c018698237df4ad6e0886cf0d4ab6bc9e5c4  guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.dmg
  b188af0572ee682d74cc82c7e6e464115205fc130a457cfe19d42ac9ddd267f8  guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx-unsigned.tar.gz
  e764062fde144e6fb5d6dd776c10fc2daa8d775831f7e43247d17a6c6e060c97  guix-build-f51237d94d98/output/x86_64-apple-darwin18/bitcoin-f51237d94d98-osx64.tar.gz
  dab3d26ac94c669140f7329d14e57ef02b0fe92b8a8f9d96c32a416adea0da0f  guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
  ca59d4379fbe2b9a52deebeaf88508e0eda4215f28d319aff0781289dd159712  guix-build-f51237d94d98/output/x86_64-linux-gnu/bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
  52b7c35321a85c4f6c95bf0e687574454b71ede9bec1c9cf17f37c578c888a94  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win-unsigned.tar.gz
  a543895a00f8ffb3ba50ca68396d52ad5a18dd8efe38730e0049dd70d283a092  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-debug.zip
  aec050d03c65268a986148500f7341cceb8c5f85287e0e3cde8933ce4b4dee32  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64-setup-unsigned.exe
  57ba33ed6ee8d3a885e342471359301473e83037d5442895beb686921a4c50e9  guix-build-f51237d94d98/output/x86_64-w64-mingw32/bitcoin-f51237d94d98-win64.zip
  ```

  Gitian builds:
  ```bash
  # macOS:
  2f066e852bdd30ac46e5ecdf7619d19d408035c318a3edf0f1893ec2e25efb69  bitcoin-41a1b3d1b130-osx-unsigned.dmg
  8cf8ac4d21740f490262453c330b5f4a5c5b8139dfc1b322efefce3f3b93d1b2  bitcoin-41a1b3d1b130-osx-unsigned.tar.gz
  cf1b84efdd9d2588a1ce9513580fb56b38bfafe60e18f8adbeedf03521c6c2b2  bitcoin-41a1b3d1b130-osx64.tar.gz
  14995244b0bb3e80e7b79975c9c70fdfb3ee3c04fda3efd5358ce1c4efa3a312  src/bitcoin-41a1b3d1b130.tar.gz
  93881069d5e1dc385c08895a7b035a94eb010325afc2776c99b6aafa21096eb8  bitcoin-core-osx-22-res.yml

  # Windows:
  4d56dd7713121684b7eaa448679c65df2fd0aa5319bf8d12fb6cfa9f0b005cf7  bitcoin-f51237d94d98-win-unsigned.tar.gz
  4558f4173152b084bcba25aa1a53c605208a70fe20392141b63cefb476528c85  bitcoin-f51237d94d98-win64-debug.zip
  b63feaca010e86d514cfe38d716e3c8a8b8058e4f969b868aaaeb8a8a3d3dc81  bitcoin-f51237d94d98-win64-setup-unsigned.exe
  de7d8586cc91ba391fe911853a99d9fd15fc6f9a60f9b91a0447940173aac67a  bitcoin-f51237d94d98-win64.zip
  4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2  src/bitcoin-f51237d94d98.tar.gz
  45efaca35b5fad0a04dfd06e44f7c00b990aa91c7bf2faea57e020d3491a6cf0  bitcoin-core-win-22-res.yml

  # Linux:
  055d646c5f8cf4708008374546176012ff758566a2645a3a01e1a33eab1002fe  bitcoin-f51237d94d98-aarch64-linux-gnu-debug.tar.gz
  bfc8b0efc36b0474c88546b12d2723c04b4dc629ae311082025c7e0b8f0d1aa9  bitcoin-f51237d94d98-aarch64-linux-gnu.tar.gz
  9dfaa5acfffadad8942b32996458013a155d12ed07be76601f232233627b5cb9  bitcoin-f51237d94d98-arm-linux-gnueabihf-debug.tar.gz
  54eb57905ff8513b9f628707b61aa4659c362fb2f6d17e0ee240b4da3674907d  bitcoin-f51237d94d98-arm-linux-gnueabihf.tar.gz
  ad98d876616eff578ad8cfd17dfbabe48ed14200823579687d66694bae3d2fe3  bitcoin-f51237d94d98-powerpc64-linux-gnu-debug.tar.gz
  fe1b421dd1cb6e04d5dc5d341459dc15fa6e15b80906e5d8e0405cf43495e0f7  bitcoin-f51237d94d98-powerpc64-linux-gnu.tar.gz
  9001d95cc7d2722d9d7dd83d9da8e5adf575fddf91b615b76b9bcfece30ecf6f  bitcoin-f51237d94d98-powerpc64le-linux-gnu-debug.tar.gz
  9e0650ad2aba70c0fd1608a077e95f335dc1bb4a79eab9b0b56ac87427a4fd4f  bitcoin-f51237d94d98-powerpc64le-linux-gnu.tar.gz
  fbfde0134944d3dbd32991455b0a8abdd334853ab8a4c1a1a4c060d9de071c50  bitcoin-f51237d94d98-riscv64-linux-gnu-debug.tar.gz
  2fa2cfddce98c44c65305326fc623a7f065129208337503d813a08d51580cb8a  bitcoin-f51237d94d98-riscv64-linux-gnu.tar.gz
  b2d6caeee0e3c350a43165c39876ebed8e588958007af0d06996e341c7060683  bitcoin-f51237d94d98-x86_64-linux-gnu-debug.tar.gz
  bfdb827e75d43d61462513c9a843620b93c9160d9d246cad13278baaa07f64ea  bitcoin-f51237d94d98-x86_64-linux-gnu.tar.gz
  4185adebc6a0abe7241a3cd409a6ab7be031c26f1c4245e30bb5f87eef0925d2  src/bitcoin-f51237d94d98.tar.gz
  34820a093916fa35b0fd98806a50092f46b20271af7422f43e2a4223ef6f9bb7  bitcoin-core-linux-22-res.yml
  ```

ACKs for top commit:
  laanwj:
    re-ACK 7fc5e865b9

Tree-SHA512: 0c30838413448ecfcf55e6273f607fdb01cb1acafa1d2762afad59360fca7d8efa78ec55064f50cba56cb2c9e98741e13665cba8e9b4b8e5b62b8a53f9bf8990
2021-05-05 13:22:59 +02:00