Previously, if the builder exported $VERSION in their environment (as
past Gitian-building docs told them to), but their HEAD does not
actually point to v$VERSION, their build outputs will differ from those
of other builders.
This is because the contrib/guix/guix-* scripts only ever act on the
current git worktree, and does not try to check out $VERSION if $VERSION
is set in the environment.
Setting $VERSION only makes the scripts pretend like the current
worktree is $VERSION.
This problem was seen in jonatack's attestation for all.SHA256SUMS,
where only his bitcoin-22.0rc3-osx-signed.dmg differed from everyone
else's.
Here is my deduced sequence of events:
1. Aug 27th: He guix-builds 22.0rc3 and uploads his attestations up to
guix.sigs
2. Aug 30th, sometime after POSIX time 1630310848: he pulls the latest
changes from master in the same worktree where he guix-built 22.0rc3
and ends up at 7be143a960
3. Aug 30th, sometime before POSIX time 1630315907: With his worktree
still on 7be143a960, he guix-codesigns. Normally, this would result
in outputs going in guix-build-7be143a960e2, but he had
VERSION=22.0rc3 in his environment, so the guix-* scripts pretended
like he was building 22.0rc3, and used 22.0rc3's guix-build directory
to locate un-codesigned outputs and dump codesigned ones.
However, our SOURCE_DATE_EPOCH defaults to the POSIX time of HEAD
(7be143a960), which made all timestamps in the resulting codesigned
DMG 1630310848, 7be143a960e2's POSIX timestamp. This differs from the
POSIX timestamp of 22.0rc3, which is 1630348517. Note that the
windows codesigning procedure does not consider SOURCE_DATE_EPOCH.
We resolve this by only allowing VERSION overrides via the FORCE_VERSION
environment variable.
42dbd9025a contrib: return non-zero status if getcoins.py errors (Sebastian Falbesoner)
8c203cf0e1 contrib: catch bitcoin-cli RPC call errors in getcoins.py (Sebastian Falbesoner)
0eca5ebace contrib: refactor: introduce bitcoin-cli RPC call helper in getcoins.py (Sebastian Falbesoner)
Pull request description:
This PR is based on #22565 ("[script] signet's getcoins.py improvements"), which should be reviewed first.
The signet faucet script `contrib/signet/getcoins.py` currently issues bitcoin-cli RPC calls without catching errors -- the only case tackled is if there is no `bitcoin-cli` file found. Instead of crashing with a stack-trace on a failed RPC call, the changes in this PR aim to produce a more user-friendly output (see also https://github.com/bitcoin/bitcoin/pull/22565#discussion_r683754875). Additionally, in case of any error, a non-zero status is now returned (instead of 0, indicating success), which could be useful for other scripts taking use of signet faucet script.
The most straight-forward way to test this is invoking the script without a `bitcoind` running on signet:
PR22565 branch:
```
$ ./contrib/signet/getcoins.py
error: Could not connect to the server 127.0.0.1:8332
Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
Traceback (most recent call last):
File "./contrib/signet/getcoins.py", line 26, in <module>
curr_signet_hash = subprocess.check_output([args.cmd] + args.bitcoin_cli_args + ['getblockhash', '1']).strip().decode()
File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bitcoin-cli', 'getblockhash', '1']' returned non-zero exit status 1.
```
this PR branch:
```
$ ./contrib/signet/getcoins.py
error: Could not connect to the server 127.0.0.1:38332
Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
-----
Error while calling "bitcoin-cli -signet getblockhash 1" (see output above).
```
ACKs for top commit:
kallewoof:
Code ACK 42dbd9025a
Zero-1729:
tACK 42dbd90🧪
Tree-SHA512: 912240a4ed03c87035e370602f4095c7ffe26806421bbbd6cf86588126f2310a01a6a61606e9e2918fb2c1a0debdd0ce768c69ba2e4b8e7750fa3474a56d01a0
b0c8246cac Add cleaner errors for unsuccessful faucet transactions (NikhilBartwal)
1c612b274b [script] Update signet getcoins.py for custom network (NikhilBartwal)
Pull request description:
Currently, using the getcoins.py with a custom signet executes successfully and shows the transfer of 0.001 testBTC as complete, however for obvious reasons, it should not. In fact, upon verification it does not actually execute the transaction, but rather gives the output that it did, as shown below which can be misleading:
```
[nikhilb@nikhil-PC bitcoin]$ echo $datadir
/home/nikhilb/signet-custom
[nikhilb@nikhil-PC bitcoin]$ contrib/signet/getcoins.py -- -datadir=$datadir
Payment of 0.00100000 BTC sent with txid dd22c7d996e95f3e5baf20f73140d517ff48f1b26d0e4fefd61e3c37991b8f86
[nikhilb@nikhil-PC bitcoin]$ bitcoin-cli -datadir=$datadir getrawtransaction dd22c7d996e95f3e5baf20f73140d517ff48f1b26d0e4fefd61e3c37991b8f86
error code: -5
error message:
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
[nikhilb@nikhil-PC bitcoin]$ bitcoin-cli -datadir=$datadir gettransaction dd22c7d996e95f3e5baf20f73140d517ff48f1b26d0e4fefd61e3c37991b8f86
error code: -5
error message:
Invalid or non-wallet transaction id
```
This PR adds a sanity check for custom signet by comparing the current network's first block hash (the block after the genesis block) with global signet's respective block hash (since all signet networks share the same genesis block) and if a custom network is detected, the user is prompted to either work on the global signet or setup their own faucet.
The PR was checked to be working successfully, giving the output as below:
```
[nikhilb@nikhil-PC bitcoin]$ git checkout update_signet_getcoins
Switched to branch 'update_signet_getcoins'
Your branch is ahead of 'upstream/master' by 1 commit.
(use "git push" to publish your local commits)
[nikhilb@nikhil-PC bitcoin]$ contrib/signet/getcoins.py -- -datadir=$datadir
The global faucet cannot be used with a custom Signet network. Please use the global signet or setup your custom faucet for the same.
You can have a look here for setting up your own faucet: https://en.bitcoin.it/wiki/Signet
```
ACKs for top commit:
prayank23:
utACK b0c8246cac
kallewoof:
ACK b0c8246cac
arnabsen1729:
utACK b0c8246
prakash1512:
utACK b0c8246
0xB10C:
Tested ACK b0c8246cac
theStack:
Tested ACK b0c8246cac
Zero-1729:
crACK b0c8246🧉
Tree-SHA512: 144b47a83008521a5cda13f4c1b12809a125a744f865a8e0f792132d52fdb88926d4f4f4d7230452c2e129b5879892cdbeda981b8af10b789e9fc0cda2905a5d
132cae44f2 doc: Mention the flat directory structure for uploads (Andrew Chow)
fb17c99e35 guix: Don't include directory name in SHA256SUMS (Andrew Chow)
Pull request description:
The SHA256SUMS file can be used in a sha256sum -c command to verify downloaded binaries. However users are likely to download just a single file and not place this file in the correct directory relative to the SHA256SUMS file for the simple verification command to work. By not including the directory name in the SHA256SUMS file, it will be easier for users to verify downloaded binaries.
ACKs for top commit:
Zero-1729:
re-ACK 132cae44f2
fanquake:
ACK 132cae44f2
Tree-SHA512: c9ff416b8dfb2f3ceaf4d63afb84aac9fcaefbbf9092f9e095061b472884ec92c7a809e6530c7132a82cfe3ab115a7328e47994a412072e1d4feb26fc502c8c5
The SHA256SUMS file can be used in a sha256sum -c command to verify
downloaded binaries. However users are likely to download just a single
file and not place this file in the correct directory relative to the
SHA256SUMS file for the simple verification command to work. By not
including the directory name in the SHA256SUMS file, it will be easier
for users to verify downloaded binaries.
Co-authored-by: Carl Dong <contact@carldong.me>
021daedfa1 refactor: replace remaining binascii method calls (Zero-1729)
Pull request description:
This PR removes the remaining `binascii` method calls outside `test/functional` and `test_framework`, as pointed out here https://github.com/bitcoin/bitcoin/pull/22619#pullrequestreview-722153458.
Follow-up to #22593 and #22619Closes#22605
ACKs for top commit:
josibake:
re-ACK 021daedfa1
theStack:
re-ACK 021daedfa1
Tree-SHA512: 2ae9fee8917112c91a5406f219ca70f24cd8902b903db5a61fc2de85ad640d669a772f5c05970be0fcee6ef1cdd32fae2ca5d1ec6dc9798b43352c8160ddde6f
5449d44e37 scripts: prevent GCC optimising test symbols in test-symbol-check (fanquake)
Pull request description:
I noticed in #22381 that when the test-symbol-check target was being built with Clang and run in the CI it would fail due to using a too-new version of `pow` (used [here](d67330d112/contrib/devtools/test-symbol-check.py (L85))). Our CIs use Focal (glibc 2.31) and the version of `pow` was the optimized version introduced in [glibc 2.29](https://lwn.net/Articles/778286/):
```bash
* Optimized generic exp, exp2, log, log2, pow, sinf, cosf, sincosf and tanf.
```
This made sense, except for that if it was failing when built using Clang, why hadn't it also been failing when being built with GCC?
Turns out GCC is optimizing away that call to `pow` at all optimization levels, including `-O0`, see: https://godbolt.org/z/53MhzMxT7, and this has been the case forever, or at least since GCC 5.x. Clang on the other hand, will only optimize away the `pow` call at `-O1` and `-O2`, not `-O0`: https://godbolt.org/z/Wbnqj3q6c. Thus when this test was built with Clang (we don't pass `-O` so we default to `-O0`) it was failing in the CI environment, because it would actually have a call to the "new" `pow`.
Avoid this issue by using a symbol that won't be optimized away, or that we are unlikely to ever have versioning issues with.
ACKs for top commit:
laanwj:
ACK 5449d44e37
Tree-SHA512: 3a26c5c3a5f2905fd0dd90892470e241ba625c0af3be2629d06d5da3a97534c1d6a55b796bbdd41e2e6a26a8fab7d981b98c45d4238565b0eb7edf3c5da02007
7d95777417 builder-keys: Add dongcarl (Carl Dong)
Pull request description:
https://keys.openpgp.org/search?q=04017A2A6D9A0CCDC81D8EC296AB007F1A7ED999
This is my master key, will be bumping the expiration of subkeys or rotating when necessary.
ACKs for top commit:
fanquake:
ACK 7d95777417 - matches what I've got.
Tree-SHA512: 3a76b8eda81821b3221402501cf8191bce73118624b932aa80a7fc1a32a91e3825aeb2b03ed261bbf284b088e927c384f92e08eadddf7f94ed4de579d9f6d2b7
90b3e482e9 release: Release with separate SHA256SUMS and sig files (Carl Dong)
Pull request description:
This allows us to:
- remove the rfc4880 EOL hacks, and
- release with a SHA256SUMS.asc file that's a combination of all signer signatures
ACKs for top commit:
achow101:
ACK 90b3e482e9
laanwj:
Concept and code review ACK 90b3e482e9
Tree-SHA512: 5d5086063d303aa0cbd590e5fdf2ae8f555e25f4e43bf67545e33384449b990e94834c711622530ad0eb3dcc83f52746884a5081dadb0acff8dd799cfadafac7
Currently, using the getcoins.py with a custom signet executes successfully and shows the transaction as complete, however for obvious reasons, it should not.
This PR adds a sanity check for custom signet by comparing the current network's first block hash with global signet's respective hash.
f8f772dc49 macdeploy: alternative info to download the macOS SDK (Antoine Poinsot)
Pull request description:
The previous link wasn't accessible for me, this adds some instructions
given to me by Hebasto on #bitcoin-core-builds as well as a shasum for
the archive to quickly check the downloaded one is the right one before
processing with the entire Guix build.
ACKs for top commit:
fanquake:
ACK f8f772dc49
Tree-SHA512: 620160b593ed8fa4ae4a748b8e72d67b93ff0ec9e6b8ef3c3ac5402c1c48ec0ac325a527b6278cdf84aaf51ba8194d4c366c412ffad141d0412add2710efcff5
The previous link wasn't accessible for me, this adds some instructions
given to me by Hebasto on #bitcoin-core-builds as well as a shasum for
the archive to quickly check the downloaded one is the right one before
processing with the entire Guix build.
This also corrects a link to an older version of the SDK currently in
use.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
9b313dfef1 guix: Ensure EPOCH_SOURCE_DATE does not include GPG information (Andrew Chow)
43225f0a2a guix: Remove extra \r from all.SHA256SUMS line ending (Andrew Chow)
d080c27066 guix, doc: Add a note that codesigners need to rebuild after tagging (Andrew Chow)
4a466388a0 guix: Allow changing the base manifest in guix-verify (Andrew Chow)
33455c7696 guix: Make all.SHA256SUMS rather than codesigned.SHA256SUMS (Andrew Chow)
Pull request description:
`guix-verify` expects `all.SHA256SUMS` but `guix-attest` produces `codesigned.SHA256SUMS`. Since `all.SHA256SUMS` makes more sense (as the file contains all the sha256sums, not just the codesigned ones), `guix-attest` has been changed to output a file of that name.
As a quality of life improvement, `guix-verify` can take `SIGNER` and use the signer's manifest as the base to compare against. This makes it easier to compare a single person's attestations with everyone else's and can make it more obvious when one builder is clearly mismatching with everyone else.
Lastly `release-process.md` is updated with a note about a gotcha that can cause a mismatch in the codesigned attestation.
ACKs for top commit:
fanquake:
ACK 9b313dfef1
Tree-SHA512: 0d60627def38288dbd3059ad1e72cad224f9205da11b1a561c082ef28250a074df5cc5f2797c91a7be027bc486a3fda3319c2e496a8724e5b539337236c6f990
If the user has set log.showSignature=true in their git config, then the
git log will always output GPG signature information. Since git log is
used to set EPOCH_SOURCE_DATE, this will mistakenly have GPG signature
information in it which causes issues for the build. To avoid this
issue, we override the config and force log.showSignature=false.
guix-attest mistakenly added an extra \r to the line endings in
all.SHA256SUMS, causing guix-verify to erroneously fail.
Co-Authored-By: Carl Dong <contact@carldong.me>
Both added files are extended in the following commits.
doc/usdt.md is based on earlier work by laanwj.
Co-authored-by: W. J. van der Laan <laanwj@protonmail.com>
9f01feda0a guix/build: Remove vestigial SKIPATTEST.TAG (Carl Dong)
Pull request description:
No longer needed or referenced by anything. A relic from prior to the great hierarchy overhaul of #22182
ACKs for top commit:
achow101:
ACK 9f01feda0a
fanquake:
ACK 9f01feda0a
Tree-SHA512: a94cf63f0c5cb8dbacf1025b6c0e81b219c2a3c93b3cbcefc239ccde29e602ecd4b717b1d93dbe53cb791a5017236fb09823c034aec42b0c31894fc9e0ab8b21
a884a1edcd guix/INSTALL: Misc fixups (Carl Dong)
3c4d2c418e guix: Silence getent(1) invocation (Carl Dong)
Pull request description:
Otherwise the `getent(1)` checks will print out the default http, https, and ftp ports, making it seem like something is being spawned that is listening on those ports, which is not the case.
ACKs for top commit:
fanquake:
ACK a884a1edcd
Tree-SHA512: 7706a98fe5f2bcd766fd3a16bfffab899ec45e80d72c485b7bed2a83d2024eddbb44ae4a77e2352e308740ca203c163421a11a5a2327fa94d2032ecceef4d63f
e6a94d4446 guix: Bump to version-1.3.0 from upstream (Carl Dong)
90fd13b954 guix: Pin kernel header version (Carl Dong)
Pull request description:
```
- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
- Use 4.9 for all others (second-oldest LTS release, released in
combination with glibc glibc 2.24 in Debian stretch)
```
```
The chosen commit is the HEAD of Guix's version-1.3.0 branch as of July
15th, 2021.
Also fix visual indenting.
```
-----
This + the documentation PR should make our Guix system ready for release!
ACKs for top commit:
MarcoFalke:
review ACK e6a94d4446 to change to vanilla guix. Did not review the kernel change.
laanwj:
ACK e6a94d4446
fanquake:
ACK e6a94d4446
Tree-SHA512: a175e4ddb3ee786a39f5e800ce336932ad2f6797a3a28400a6f723875d0f19833fd36cedc41b3580e4604110517211bd9f557be36adf7265fd8e591c434ae032
0a5723beea macdeploy: cleanup .temp.dmg if present (fanquake)
ecffe8689d macdeploy: remove qt4 related code (fanquake)
639f064253 macdeploy: select the plugins we need, rather than excluding those we don't (fanquake)
3d26b6b9e9 macdeploy: fix framework printing when passing -verbose (fanquake)
dca6c90329 macdeploy: remove unused plistlib import (fanquake)
Pull request description:
This includes [one followup](https://github.com/bitcoin/bitcoin/pull/20422#discussion_r534207899) and [one bug fix](3d26b6b9e9) from #20422, as well as some simplifications to the `macdeployqtplus` code.
ACKs for top commit:
hebasto:
ACK 0a5723beea, tested on macOS Big Sur 11.4 (20F71, x86_64) + Homebrew's Qt 5.15.2.
Tree-SHA512: cfad9505eacd32fe3a9d06eb13b2de0b6d2cad7b17778e90b503501cbf922e53d4e7f7f74952d1aed58410bdae9b0bb3248098583ef5b85689cb27d4dc06c029
fac4814106 doc/release-process: Add torrent creation details (Carl Dong)
5d24cc3d82 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong)
5da2ee49d5 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong)
318c60700b guix: Adapt release-process.md to new Guix process (Carl Dong)
fcab35b229 guix-attest: Produce and sign normalized documents (Carl Dong)
c2541fd0ca guix: Overhaul README (Carl Dong)
46ce6ce378 tree-wide: Rename gitian-keys to builder-keys (Carl Dong)
fc4f8449f3 guix: Update various check_tools lists (Carl Dong)
263220a85c guix: Check for a sane services database (Carl Dong)
Pull request description:
Based on: #21462
Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier.
Current TODOs:
- [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up
- [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login
- [x] Fanquake docker option: complete section
- [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section
- [x] Building from source option: Insert dependency tree diagram that I made
- [x] Building from source option: redo sectioning, kind of a mess right now
- [x] Optional setup: make clear which parts are only needed if building from source
- [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key
- [x] Overall (after everything): Make the links work.
Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option.
ACKs for top commit:
fanquake:
ACK fac4814106 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release.
Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
That way we can easily combine the document and detached signature to
produce cleartext signature files for upload during the release process.
See subsequent commits which modify doc/release-process.md for more
details.
1edddf5de4 Avoid GCC 7.1 ABI change warning in guix build (Pieter Wuille)
Pull request description:
The arm-linux-gnueabihf guix build output is littered with warnings like:
```
/gnu/store/7a96hdqdb2qi8a39f09n84xjy2hr23rs-gcc-cross-arm-linux-gnueabihf-8.4.0/include/c++/bits/stl_vector.h:1085:4: note:
parameter passing for argument of type '__gnu_cxx::__normal_iterator<CRecipient*, std::vector<CRecipient> >' changed in GCC 7.1
```
These are irrelevant for us. Disable them using `-Wno-psabi`.
ACKs for top commit:
laanwj:
ACK 1edddf5de4
hebasto:
ACK 1edddf5de4, after thorough reading related materials, I agree this change can be merged. As I mentioned above, I have been compiling my arm-32bit binaries with `-Wno-psabi` flag for two years, and no related flaws were observed.
Tree-SHA512: 485c7500547ac5da567ad23847341c18ff832607f5a1002676404cc647e437cf3445b6894ecff5b52929ca52bea946c06bd90eace1997c895e56204e787065e4
- Use 4.19 for riscv64 (earliest LTS release w/ riscv64 support)
- Use 4.9 for all others (second-oldest LTS release, released in
combination with glibc glibc 2.24 in Debian stretch)
On bare systems, it is possible to be lacking a services database. Check
for basic entries before attempting a build.
See the error message in the diff for more context.
Now that our release binaries are build in a glibc 2.24 and 2.27
environment, we can't use a symbol from glibc 2.28 to test our checks.
Replace renameat2() with nextup(), which was introduced in 2.24.
Note that this also means re-disabling the test for RISC-V, however
RISC-V is built in a glibc 2.27 environment, and our minimum required
glibc for that binary is 2.27.
We use these flags in our test-security-check make target, but they are
only available because debian patches them in.
We can patch them in for our Guix builds so that we can check the sanity
of our security/symbol checking suite before running them.
This is important to make sure that we're not testing tools different
from the one we're building with.
Introduce determine_wellknown_cmd, which encapsulates how we
should handle well-known tools specification (IFS splitting, env
override, etc.).
Now that our Guix builds are performed on glibc 2.24 and 2.27 (RISCV),
we no-longer need to pass the --enable-glibc-back-compat option.
Replace it with --disable-threadlocal, to prevent the usage of symbols
from glibc 2.18.
None of the binaries produced required symbols later than 2.17, and 2.27
(RISCV).
Our 'bitcoin-linux-g++' definition better integrates with our depends
system than the stock linux-g++-64 definition.
This fixes a bug whereby Guix builds on x86_64 for x86_64 did not
produce a QMinimalIntegrationPlugin and led to bitcoin-qt not being
built.
Support for riscv64 in glibc landed in 2.27 so it's unavoidable that we
use 2.27.
Running a Bitcoin build with toolchains based on 2.24 for platforms
other than riscv64 seem to produce binaries which do not have 2.17
symbols. So use 2.24 since it's more recent and maintained by Debian
Stretch.
bdb8b9a347 test: doc: improve doc for `from_hex` helper (mention `to_hex` alternative) (Sebastian Falbesoner)
1914054208 scripted-diff: test: rename `FromHex` to `from_hex` (Sebastian Falbesoner)
a79396fe5f test: remove `ToHex` helper, use .serialize().hex() instead (Sebastian Falbesoner)
2ce7b47958 test: introduce `tx_from_hex` helper for tx deserialization (Sebastian Falbesoner)
Pull request description:
There are still many functional tests that perform conversions from a hex-string to a message object (deserialization) manually. This PR identifies all those instances and replaces them with a newly introduced helper `tx_from_hex`.
Instances were found via
* `git grep "deserialize.*BytesIO"`
and some of them manually, when it were not one-liners.
Further, the helper `ToHex` was removed and simply replaced by `.serialize().hex()`, since now both variants are in use (sometimes even within the same test) and using the helper doesn't really have an advantage in readability. (see discussion https://github.com/bitcoin/bitcoin/pull/22257#discussion_r652404782)
ACKs for top commit:
MarcoFalke:
review re-ACK bdb8b9a347😁
Tree-SHA512: e25d7dc85918de1d6755a5cea65471b07a743204c20ad1c2f71ff07ef48cc1b9ad3fe5f515c1efaba2b2e3d89384e7980380c5d81895f9826e2046808cd3266e
e8cd3700ee devtools: Integrate ARCH_MIN_GLIBC_VER table into MAX_VERSIONS in symbol-check.py (W. J. van der Laan)
a33381acf5 devtools: Add xkb version to symbol-check (W. J. van der Laan)
19e598bab0 devtools: Fix verneed section parsing in pixie (W. J. van der Laan)
Pull request description:
I misunderstood the ELF specification for version symbols (verneed): The `vn_aux` pointer is relative to the main verneed record, not the start of the section.
This caused many symbols to not be versioned properly in the return value of `elf.dyn_symbols`. This was discovered in #21454.
Fix it by correcting the offset computation.
- xkb versions symbols (using the prefix `V`), as this library is used by bitcoin-qt, add it to the valid versions in `symbol-check.py`
This unfortunately brings to light some symbols that have been introduced since and weren't caught (from a gitian compile of master):
```
bitcoin-cli: symbol getrandom from unsupported version GLIBC_2.25
bitcoin-cli: failed IMPORTED_SYMBOLS
bitcoind: symbol getrandom from unsupported version GLIBC_2.25
bitcoind: symbol log from unsupported version GLIBC_2.29
bitcoind: symbol fcntl64 from unsupported version GLIBC_2.28
bitcoind: symbol pow from unsupported version GLIBC_2.29
bitcoind: symbol exp from unsupported version GLIBC_2.29
bitcoind: failed IMPORTED_SYMBOLS
bitcoin-qt: symbol exp from unsupported version GLIBC_2.29
bitcoin-qt: symbol fcntl64 from unsupported version GLIBC_2.28
bitcoin-qt: symbol log from unsupported version GLIBC_2.29
bitcoin-qt: symbol pow from unsupported version GLIBC_2.29
bitcoin-qt: symbol statx from unsupported version GLIBC_2.28
bitcoin-qt: symbol getrandom from unsupported version GLIBC_2.25
bitcoin-qt: symbol renameat2 from unsupported version GLIBC_2.28
bitcoin-qt: symbol getentropy from unsupported version GLIBC_2.25
bitcoin-qt: failed IMPORTED_SYMBOLS
bitcoin-wallet: symbol exp from unsupported version GLIBC_2.29
bitcoin-wallet: symbol log from unsupported version GLIBC_2.29
bitcoin-wallet: symbol fcntl64 from unsupported version GLIBC_2.28
bitcoin-wallet: failed IMPORTED_SYMBOLS
test_bitcoin: symbol getrandom from unsupported version GLIBC_2.25
test_bitcoin: symbol log from unsupported version GLIBC_2.29
test_bitcoin: symbol fcntl64 from unsupported version GLIBC_2.28
test_bitcoin: symbol pow from unsupported version GLIBC_2.29
test_bitcoin: symbol exp from unsupported version GLIBC_2.29
test_bitcoin: failed IMPORTED_SYMBOLS
```
ACKs for top commit:
hebasto:
ACK e8cd3700ee
Tree-SHA512: 8c15e3478eb642f01a1ddaadef03f80583f088f9fa8e3bf171ce16b0ec05ffb4675ec147d7ffc6a4360637ed47fca517c6ca2bac7bb30d794c03783cfb964b79
The (ancient) versions specified here were deceptive. Entries older than
MAX_VERSIONS['GLIBC'], which is 2.17, are ignored here. So reorganize
the code to avoid confusion for other people reading this code.
aa80b5759d scripts: check macOS SDK version is set (fanquake)
c972345bac scripts: check minimum required Windows version is set (fanquake)
29615aef52 scripts: check minimum required macOS vesion is set (fanquake)
8732f7b6c9 scripts: LIEF 0.11.5 (fanquake)
Pull request description:
macOS:
We use a compile flag ([-mmacosx-version-min=10.14](https://github.com/bitcoin/bitcoin/blob/master/depends/hosts/darwin.mk#L96)) to set the minimum required version of macOS needed to run our binaries. This adds a sanity check that the version is being set as expected.
Clangs Darwin driver should infer the SDK version used during compilation, and forward that through to the linker. Add a check that this has been done, and the expected SDK version is set. Should help prevent issues like #21771 in future.
Windows:
We use linker flags ([-Wl,--major/minor-subsystem-version](https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L683)) to set the minimum required version of Windows needed to run our binaries. This adds a sanity check that the version is being set as expected.
Gitian builds:
```bash
# macOS:
8b6fcd61d75001c37b2af3fceb5ae09f5d2fe85e97d361f684214bd91c27954a bitcoin-f015e1c2cac9-osx-unsigned.dmg
3c1e412bc7f5a7a5d0f78e2cd84b7096831414e1304c1307211aa3e135d89bbf bitcoin-f015e1c2cac9-osx-unsigned.tar.gz
50b7b2804e8481f63c69c78e3e8a71c0d811bf2db8895dd6d3edae9c46a738ae bitcoin-f015e1c2cac9-osx64.tar.gz
fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz
8a20f21b20673dfc8c23e22b20ae0839bcaf65bf0e02f62381cdf5e7922936f0 bitcoin-core-osx-22-res.yml
# Windows:
b01fcdc2a5673387050d6c6c4f96f1d350976a121155fde3f76c2af309111f9d bitcoin-f015e1c2cac9-win-unsigned.tar.gz
b95bdcbef638804030671d2332d58011f8c4ed4c1db87d6ffd211515c32c9d02 bitcoin-f015e1c2cac9-win64-debug.zip
350bf180252d24a3d40f05e22398fec7bb00e06d812204eb5a421100a8e10638 bitcoin-f015e1c2cac9-win64-setup-unsigned.exe
2730ddabe246d99913c9a779e97edcadb2d55309933d46f1dffd0d23ecf9aae5 bitcoin-f015e1c2cac9-win64.zip
fe6b5c0a550096b76b6727efee30e85b60163a41c83f21868c849fdd9876b675 src/bitcoin-f015e1c2cac9.tar.gz
aa60d7a753e8cb2d4323cfbbf4d964ad3645e74c918cccd66862888f8646d80f bitcoin-core-win-22-res.yml
```
ACKs for top commit:
hebasto:
ACK aa80b5759d, tested by breaking tests:
Tree-SHA512: 10150219910e8131715fbfe20edaa15778387616ef3bfe1a5152c7acd3958fe8f88c74961c3d3641074eb72824680c22764bb1dc01a19e92e946c2d4962a8d2c
e2c40a4ed5 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed5 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b839 Make SHA256SUMS fragment right after build (Carl Dong)
Pull request description:
Based on: #22075
Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff
The following changes resolve many usability improvements which were pointed out to me:
1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.
For those who sign on an separate machine than the one you do builds on, the following steps will work:
1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
3. Sign the `SHA256SUMS` files:
```bash
for i in "<path-to-achow101>/*.SHA256SUMS"; do
gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
done
```
5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`
-----
After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
```
output
└── x86_64-w64-mingw32
├── bitcoin-4e069f7589da-win64-debug.zip
├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
├── bitcoin-4e069f7589da-win64.zip
├── bitcoin-4e069f7589da-win-unsigned.tar.gz
└── SHA256SUMS.part
```
These `SHA256SUMS.part` fragments look something like:
```
3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz
def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
```
Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level
When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:
1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs
Then both files are signed, resulting in the following `guix.sigs` hierarchy:
```
4e069f7589da/
└── dongcarl
├── all.SHA256SUMS
├── all.SHA256SUMS.asc
├── noncodesigned.SHA256SUMS
└── noncodesigned.SHA256SUMS.asc
```
ACKs for top commit:
achow101:
ACK e2c40a4ed5
hebasto:
ACK e2c40a4ed5, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.
Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
I misunderstood the ELF specification for version symbols (verneed):
The `vn_aux` pointer is relative to the main verneed record, not the
start of the section.
This caused many symbols to not be versioned properly in the return
value of `elf.dyn_symbols`. This was discovered in #21454.
Fix it by correcting the offset computation.
683d197970 Use latest signapple commit (Andrew Chow)
Pull request description:
Update gitian and guix to use the same latest signapple commit.
Also changed guix to use the actual repo. The changes from the fork were incorporated upstream.
ACKs for top commit:
fanquake:
ACK 683d197970 - sanity checked that the updated package is built:
Tree-SHA512: a4981f8bbe33e6c5654632bc9b9f6f2f1e675741a19ac7296205e370f1e64a747101ecb632e0cc82a0134e4c2e9ce47b3f7b4d8c8f75f0f06dd069c078303759
Clangs Darwin driver should infer the SDK version used during compilation, and
forward that through to the linker. Add a check that this has been done, and the
expected SDK version is set.
Should help prevent issues like #21771 in future.
We use linker flags (-Wl,--major/minor-subsystem-version) to set the
minimum required version of Windows needed to run our binaries. This
adds a sanity check that the version is being set as expected.
We use a compile flag (-mmacosx-version-min) to set the minimum required
version of macOS needed to run our binaries. This adds a sanity check
that the version is being set as expected.
108a6be92a guix: Check for disk space availability before building (Carl Dong)
d7dec89091 guix: Remove dest if OUTDIR mv fails (Carl Dong)
Pull request description:
There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log
- Perform an early disk space check for `guix-build`
- Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails
ACKs for top commit:
laanwj:
Tested ACK 108a6be92a
achow101:
ACK 108a6be92a
Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
5d82a57db4 contrib: remove torv2 seed nodes (Jon Atack)
5f7e086dac contrib: update generate-seeds.py to ignore torv2 addresses (Jon Atack)
8be56f0f8e p2p, refactor: extract OnionToString() from CNetAddr::ToStringIp() (Jon Atack)
5f9d3c09b4 p2p: remove torv2 from CNetAddr::ToStringIP() (Jon Atack)
3d39042144 p2p: remove torv2 in SetIP() and ADDR_TORV2_SIZE constant (Jon Atack)
cff5ec477a p2p: remove pre-addrv2 onions from SerializeV1Array() (Jon Atack)
4192a74413 p2p: ignore torv2-in-ipv6 addresses in SetLegacyIPv6() (Jon Atack)
1d631e956f p2p: remove BIP155Network::TORV2 from GetBIP155Network() (Jon Atack)
7d1769bc45 p2p: remove torv2 from SetNetFromBIP155Network() (Jon Atack)
eba9a94b9f fuzz: rename CNetAddr/CService deserialize targets (Jon Atack)
c56a1c9b18 p2p: drop onions from IsAddrV1Compatible(), no longer relay torv2 (Jon Atack)
f8e94002fc p2p: remove torv2/ADDR_TORV2_SIZE from SetTor() (Jon Atack)
0f1c58ae87 test: update feature_proxy to torv3 (Jon Atack)
Pull request description:
This patch removes support in Bitcoin Core for Tor v2 onions, which are already removed from the release of Tor 0.4.6.
- no longer serialize/deserialize and relay Tor v2 addresses
- ignore incoming Tor v2 addresses
- remove Tor v2 addresses from the addrman and peers.dat on node launch
- update generate-seeds.py to ignore Tor v2 addresses
- remove Tor v2 hard-coded seeds
Tested with tor-0.4.6.1-alpha (no v2 support) and 0.4.5.7 (v2 support). With the latest Tor (no v2 support), this removes all the warnings like those reported with current master in https://github.com/bitcoin/bitcoin/issues/21351
```
<bitcoind debug log>
Socks5() connect to […].onion:8333 failed: general failure
<tor log>
Invalid hostname [scrubbed]; rejecting
```
and the addrman no longer has Tor v2 addresses on launching bitcoind.
```rake
$ ./src/bitcoin-cli -addrinfo
{
"addresses_known": {
"ipv4": 44483,
"ipv6": 8467,
"torv2": 0,
"torv3": 2296,
"i2p": 6,
"total": 55252
}
}
```
After recompiling back to current master and restarting with either of the two Tor versions (0.4.5.7 or 0.4.6.1), -addrinfo initially returns 0 Tor v2 addresses and then begins finding them again.
Ran nodes on this patch over the past week on mainnet/testnet/signet/regtest after building with DEBUG_ADDRMAN.
Verified that this patch bootstraps an onlynet=onion node from the Tor v3 hardcoded fixed seeds on mainnet and testnet and connects to blocks and v3 onion peers: `rm ~/.bitcoin/testnet3/peers.dat ; ./src/bitcoind -testnet -dnsseed=0 -onlynet=onion`
Tested using `addnode`, `getaddednodeinfo`,`addpeeraddress`, `disconnectnode` and `-addrinfo` that a currently valid, connectable Tor v2 peer can no longer be added:
Thanks to Vasil Dimov, Carl Dong, and Wladimir J. van der Laan for their work on BIP155 and Tor v3 that got us here.
ACKs for top commit:
laanwj:
Code review ACK 5d82a57db4
Tree-SHA512: 590ff3d2f6ef682608596facb4b01f44fef69716d2ab3552ae1655aa225f4bf104f9ee08d6769abb9982a8031de93340df553279ce1f5023771f9f2b651178bb
a58868d201 build: Makes rcc output always deterministic (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2.
The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732).
This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless.
---
Also it improves interaction with ccache:
On master (f6c44e999b):
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:45:43 2021
stats zeroed Sun Apr 11 15:45:05 2021
cache hit (direct) 638
cache hit (preprocessed) 0
cache miss 1
cache hit rate 99.84 %
called for link 10
cleanups performed 0
files in cache 20023
cache size 13.2 GB
max cache size 15.0 GB
```
The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`.
With this PR:
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:28:46 2021
stats zeroed Sun Apr 11 15:28:21 2021
cache hit (direct) 639
cache hit (preprocessed) 0
cache miss 0
cache hit rate 100.00 %
called for link 10
cleanups performed 0
files in cache 20012
cache size 13.2 GB
max cache size 15.0 GB
```
ACKs for top commit:
fanquake:
ACK a58868d201
Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
