For testing the USDT tracepoint API in the functional tests we
require:
- that we are on a Linux system*
- that Bitcoin Core is compiled with tracepoints
- that bcc and the the Python bcc module [0] is installed
- that we run the tests with the required permissions**
otherwise we skip the tests.
*: We currently only support tracepoints on Linux. Tracepoints are
not compiled on other platforms.
**: Currently, we check for root permissions via getuid == 0. It's
unclear if it's even possible to run the tests a non-root user
with e.g. CAP_BPF, CAP_PERFMON, and access to /sys/kernel/debug/
tracing/. Anyone running these tests as root should carefully
review them first and then run them in a disposable VM.
[0]: https://github.com/iovisor/bcc/blob/master/INSTALL.md
faef344f84 Print enable_fuzz_binary in configure (MarcoFalke)
Pull request description:
A *disabled* `enable_fuzz` on current master does *not* mean the the fuzz binary is not compiled. This is confusion, so fix it.
* `enable_fuzz` toggles compilation flags for fuzzing and disables all other target. There is no need to print this in the configure result, because the compilation flags are already printed. Also, all other targets are already printed as `no`.
* `enable_fuzz_binary` does what it says it does and is currently not printed. So print it.
ACKs for top commit:
hebasto:
ACK faef344f84, tested on Linux Mint 20.2 (x86_64):
Tree-SHA512: 9b02b05c4b9c5fc92cf3487497392690303c36eace5e217f18b4349f059b5a23a7c0e0d030fb6fa7bbad83e927576a5e81c00099164f9ed8e185c0969dc17689
abc057c603 build: Add Boost.Process usage check (Hennadii Stepanov)
Pull request description:
This PR adds a check that Boost.Process can be used without linking any libraries (header-only).
Disable the functionality if that is not the case.
Fixesbitcoin/bitcoin#24314.
ACKs for top commit:
fanquake:
ACK abc057c603
Tree-SHA512: ed2a32b1f751ec6f88cc7220766edd4cdab93c1d0515c892aa3094ee8d5b13ef569830d6e7a7a00c0197b117585dc526d00d943cc99a1f8c8a66ac4e20fe2061
The bitcoin-chainstate executable serves to surface the dependencies
required by a program wishing to use Bitcoin Core's consensus engine as
it is right now.
More broadly, the _SOURCES list serves as a guiding "North Star" for the
libbitcoinkernel project: as we decouple more and more modules of the
codebase from our consensus engine, this _SOURCES list will grow shorter
and shorter. One day, only what is critical to our consensus engine will
remain. Right now, it's "the minimal list of files to link in to even
use our consensus engine".
[META] In a future commit the libbitcoinkernel library will be extracted
from bitcoin-chainstate, and the libbitcoinkernel library's
_SOURCES list will be the list that we aim to shrink.
0c49e52b22 build: remove unneeded getentropy detection (HAVE_GETENTROPY) (Sebastian Falbesoner)
5cd15ffdce random: use arc4random on OpenBSD (Sebastian Falbesoner)
Pull request description:
Inspired by a discussion on obtaining randomness on various OSes in a secp256k1 PR (https://github.com/bitcoin-core/secp256k1/pull/748#discussion_r524605472, see also https://bitcoincore.reviews/libsecp256k1-748), I think it makes sense to follow best practices and use `arc4random_buf` rather than `getentropy` on OpenBSD in our random module.
The [getentropy(2) man page](https://man.openbsd.org/getentropy.2) states:
```
getentropy() is not intended for regular code; please use the
arc4random(3) family of functions instead.
```
The [arc4random(3) man page](https://man.openbsd.org/arc4random.3) states:
```
Use of these functions is encouraged for almost all random number
consumption because the other interfaces are deficient in either quality,
portability, standardization, or availability.
```
On the linked PR discussion worries about using RC4 internally has been expressed (see https://security.stackexchange.com/questions/85601/is-arc4random-secure-enough/172905#172905), but this would only affect users of OpenBSD <5.5, using a version that was released more than 8 years ago.
ACKs for top commit:
laanwj:
Tested ACK 0c49e52b22
Tree-SHA512: b5ed3d0718962c5a3839db9a28f93d08a0ac93094cc664f83bc4cf1cfad25049e6240b7b81fe06b71e6a3a0ca24a2c337eab088abec5470ad014e10c04fdb216
c62d763fc3 Necessary improvements to make configure work without libevent installed (Perlover)
091ccc38c2 The evhttp_connection_get_peer function from libevent changes the type of the second parameter. Fixing the problem. (Perlover)
Pull request description:
The second parameter of evhttp_connection_get_peer in libevent already has type as `const char **`
The compilation of bitcoind with the fresh libevent occurs errors
Details: https://github.com/bitcoin/bitcoin/issues/23606
ACKs for top commit:
laanwj:
Code review ACK c62d763fc3
luke-jr:
tACK c62d763fc3
Tree-SHA512: d1c8062d90bd0d55c582dae2c3a7e5ee1b6c7ca872bf4aa7fe6f45a52ac4a8f59464215759d961f8efde0efbeeade31b08daf9387d7d50d7622baa1c06992d83
6200fbf54f build: rename --enable-ebpf to --enable-usdt (0xb10c)
e158a2a7aa build: add systemtap's sys/sdt.h as depends (0xb10c)
Pull request description:
There has been light conceptual agreement on including the Userspace, Statically Defined Tracing tracepoints in Bitcoin Core release builds. This, for example, enables user to hook into production deployments, if they need to. Binaries don't have to be switched out. This is possible because we don't do [expensive computations](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#no-expensive-computations-for-tracepoints) only needed for the tracepoints. The tracepoints are NOPs when not used.
Systemtap's `sys/sdt.h` header is required to build Bitcoin Core with USDT support. The header file defines the `DTRACE_PROBE` macros used in [`src/util/trace.h`](https://github.com/bitcoin/bitcoin/blob/master/src/util/trace.h). This PR adds Systemtap 4.5 (May 2021) as dependency. GUIX builds for Linux hosts now include the tracepoints.
Closes https://github.com/bitcoin/bitcoin/issues/23297.
ACKs for top commit:
fanquake:
ACK 6200fbf54f - tested enabling / disabling and with/without SDT from depends. We can follow up with #23819, #23907 and #23296, and if any serious issues arise before feature freeze, it is easy for us to flip depends such that USDT becomes opt-in, rather than opt-out, and thus, releases would be tracepoint free.
Tree-SHA512: 0263f44892bf8450e8a593e4de7a498243687f8d81269e1c3283fa8354922c7cf93fddef4b92cf5192d33798424aa5812e03e68ef8de31af078a32dd34021382
eBPF is a Linux kernel technology used to "extend the capabilities
of the kernel without requiring to change kernel source code or
load kernel modules". While Userspace, Statically Defined Tracing
(USDT) uses eBPF under the hood, --enable-usdt better resembles that
support for USDT is enabled, and tracepoints will be included in the
binary.
e09773d20a build: use a static .tiff for macOS .dmg over generating (fanquake)
Pull request description:
For demonstration, after [discussion in #23778](https://github.com/bitcoin/bitcoin/pull/23778#issuecomment-1003005503), and the question as to why we can't just have a `background.tiff` that we copy into the macOS DMG, and do away with the somewhat convoluted image generation steps.
From my understanding, the only reason we have this image generation as part of our build system is so that forks of Core can adapt the imagery for their own branding via `PACKAGE_NAME`. It don't think it provides much value to us, and could just have a static .tiff that we copy into the dmg (replacing the .svg that currently lives in macdeploy/).
Doing this would eliminate the following build dependencies:
For native macOS:
* `sed` (usage in Makefile.am)
* `librsvg` (rsvg-convert)
* `tiffutil`
Linux macOS cross-compile:
* `sed` (usage in Makefille.am)
* `librsvg`
* `tiffcp`
* `convert` (imagemagick)
* `font-tuffy`
Guix Build:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
c98d67796863f4b1bab0ad600d46bd74e744d94072cbd4bc856a6aeaba3bb329 guix-build-e09773d20a92/output/dist-archive/bitcoin-e09773d20a92.tar.gz
3336f90bab312798cb7665e2b4ae24d1a270fb240647d5fed8dbfcd83e3ed37e guix-build-e09773d20a92/output/x86_64-apple-darwin/SHA256SUMS.part
8fd680c7ee158c64bad212385df7b0b302c6c2143d4e672b4b0eb5da41f9256d guix-build-e09773d20a92/output/x86_64-apple-darwin/bitcoin-e09773d20a92-osx-unsigned.dmg
34f54177c2f0700e8cfaf5d85d91e404807cd9d411e22006cdff82653e5f4af2 guix-build-e09773d20a92/output/x86_64-apple-darwin/bitcoin-e09773d20a92-osx-unsigned.tar.gz
da6b8f54ef755d40330c8eac4f5bd0329637e827be9ee61318600d5d0bdcc3dc guix-build-e09773d20a92/output/x86_64-apple-darwin/bitcoin-e09773d20a92-osx64.tar.gz
```
ACKs for top commit:
hebasto:
ACK e09773d20a
jarolrod:
ACK e09773d20a
Zero-1729:
ACK e09773d20a
Tree-SHA512: 0ad06699a5451daa8cfaaa46759eb7bd85254a72e23f857f70d433a2ffb1a4bf6dd464d9c4ac9f8c20aab045f4e2b61c6dcdcbcceef96ce515b1a0c501665b1f
1bf3809dd1 build: use host_os instead of TARGET_OS in configure output (fanquake)
Pull request description:
`TARGET_OS` was convenient, as a readable host name for most of our
targeted platforms, however unless we add more code to configure to
detect more hosts, it's easier just use `host_os` (it's also more
informative).
i.e FreeBSD master
```bash
target os =
build os = freebsd13.0
```
this PR:
```bash
target os = freebsd13.0
build os = freebsd13.0
```
ACKs for top commit:
hebasto:
ACK 1bf3809dd1
Tree-SHA512: 606d92f60ce3f2f6ab1f54e29b5c179048c62ba51336b272c081b1e009128dd83705b181cfe30991c7a51d9c63e8ba2076bfed9e6112e7d1a74a7f947c5754f5
TARGET_OS was conveninent, as a readable host name for most of our
targetted platforms, however unless we add more code to configure to
detect more hosts, it's easier just use host_os (it's also more
informative).
i.e FreeBSD master
```bash
target os =
build os = freebsd13.0
```
this PR:
```bash
target os = freebsd13.0
build os = freebsd13.0
```
Variables that are declared with AC_ARG_VAR macro are substituted via
AC_SUBST macro.
PKG_CHECK_MODULES macro already has AC_ARG_VAR(${PACKAGE}_CFLAGS) and
AC_ARG_VAR(${PACKAGE}_LIBS).
314195c8be Remove unnecessary cast in CKey::SignSchnorr (Pieter Wuille)
a1f76cdb22 Remove --disable-openssl-tests for libsecp256k1 configure (Pieter Wuille)
86dbc4d075 Squashed 'src/secp256k1/' changes from be8d9c262f..0559fc6e41 (Pieter Wuille)
Pull request description:
The motivation for this bump is getting rid of a cast in `CKey::SignSchnorr`; the `aux_rand` argument isn't modified by the `secp256k1_schnorrsig_sign` function, but was marked as non-`const` anyway. This is fixed now (bitcoin-core/secp256k1#966), and the cast is removed in this PR.
There are a few other relevant changes:
* (bitcoin-core/secp256k1#956): replaces a runtime-computed table with a precomputed one; this adds arouns 1 MiB to the binary size, but is a step towards significantly simplifying the API. If 1 MiB is too much, it can be reduced by 2 or 4 (or more) for a slight verification performance reduction.
* (bitcoin-core/secp256k1#983): removes (test/bench only) OpenSSL support entirely, removing the need to pass `--disable-openssl-tests` (see #23314).
* (bitcoin-core/secp256k1#810): mild performance increase for 64-bit non-x86 platforms.
* (bitcoin-core/secp256k1#1002): Make aux_rnd32==NULL behave identical to 0x0000..00 (which impacts BIP341/BIP342 signing in Bitcoin Core, making it more strictly BIP340 compliant, though not in a manner that affects security).
ACKs for top commit:
fanquake:
ACK 314195c8be - this includes a nice simplification to the lilbsecp build system (and thus our build system), and fixes issues like #22854. Did a Guix build on x86 (above), as well as a build on arm64 (except for the arm64 host):
Tree-SHA512: 0e048390fc148fbbdf5b98d9cce8c71067564e7d69d97b68347808a9bc45a04f4fc653c392c880d79d5d8b9cf282195520955581ac4f1595f6a948080cf5949d
66a20a54a2 build, qt: Drop support for `i686-linux-android` host (Hennadii Stepanov)
Pull request description:
There are no reasons to keep support for `i686-linux-android` host, which is actually broken in master (50c502f54a), and this fact has been unnoticed for months :)
https://github.com/bitcoin/bitcoin/pull/23675#issuecomment-986206434:
> I'm surprised `i686-linux-android` ABI is still supported. I would love to drop it...
https://github.com/bitcoin/bitcoin/pull/23675#issuecomment-991340132
> What is `i686-linux-android`? 32-bit x86 android? is that really a thing?
ACKs for top commit:
prusnak:
utACK 66a20a54a2
Tree-SHA512: 211f794de2fc569f0ade2a4da805b8bfd4ce2ab0930c5d427acc4f5d015fcdc4911f02fc64f6401197f7641aed79944a9594be80c817547be3269cdd721cf79b
We already use this in the blockfilter code, so not sure we need to maintain two
different ways of testing for the same functionality. Consolidate on testing
for __SIZEOF_INT128__, which we already use, is supported by the compilers we
care about, and is also used by libsecp256k1.
Very old shells suffered from bugs which meant that prefixing variables
with an "x" to ensure that the lefthand side of a comparison always
started with an alphanumeric character was needed. Modern shells don't
suffer from this issue (i.e Bash was fixed in 1996).
In any case, we've already got unprefixed checks used in our codebase,
i.e https://github.com/bitcoin/bitcoin/blob/master/configure.ac#L292,
and have dependencies (in depends) that also use unprefixed comparisons.
I think it's time that we can consolidate on not using the x-prefix
workaround. At best it's mostly just confusing.
More info:
https://github.com/koalaman/shellcheck/wiki/SC2268https://www.vidarholen.net/contents/blog/?p=1035
68e5aafde3 build: add `--enable-lto` configuration option (fanquake)
Pull request description:
It's been 5 years since using LTO was first suggested for use when building Bitcoin Core, and it's time to revisit it again. Compilers, and their LTO implementations, have matured, and Bitcoin Core has come a long way in terms of pruning dependencies which may have proved troublesome (i.e Boost previously had issues when using LTO). We'll have even less Boost code after moving to `std::filesystem` (#20744).
Experimenting with LTO came up on IRC last night:
> sipa: jamesob: i'm interested in knowing whether "-flto" and/or "-fdata-sections -ffunction-sections -Wl,--gc-sections" are possible/beneficial with our current compiler suite; what would be a good way to have your test infrastructure benchmark things?
So this PR just adds the bare minimum to make it easier to configure, compile and perform some bench-marking using `-flto`. This PR doesn't do anything depends wise, however if we decide this is what we want to do, I'll expand the changes here.
I had previously had a PR open (#18605) to perform link time garbage collection (`-ffunction-sections -fdata-sections` & `-Wl,--gc-sections`), however moving straight to using LTO would be preferable.
Note that our minimum required set of compilers, GCC 8.1 and Clang 7, all support the `-flto` option.
Related #18579.
Previous discussion: #10616, #14277.
Previous related PRs: #10800 (`-flto`), #16791 (ThinLTO).
Guix build:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
1f3a7c5be4169aaa444b481d3e65a7bb72da9007fee6e6c416ded2e70f97374b guix-build-68e5aafde3e8/output/aarch64-linux-gnu/SHA256SUMS.part
fa8f4cf223d9aaf0b2c1ef55ce61256a19cd1ad7f42b99d0b98c9a52fe6ad8ba guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu-debug.tar.gz
9a9967078cd1849b4e85db619e1f55d305c6d44e9e013067c0e8d62c1ba54087 guix-build-68e5aafde3e8/output/aarch64-linux-gnu/bitcoin-68e5aafde3e8-aarch64-linux-gnu.tar.gz
18c71f30722102baaf3dfda67f7c7aac38723510b142e8df8ee7063c5d499368 guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/SHA256SUMS.part
0854cc0d17c045a118df2a24e4cf36d727e7e7e2dea37c2492ee21b71cb79b4b guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf-debug.tar.gz
215256897dde4e8412ed60473376c694a80c5479fb08039107fb62435f2816ef guix-build-68e5aafde3e8/output/arm-linux-gnueabihf/bitcoin-68e5aafde3e8-arm-linux-gnueabihf.tar.gz
5fad0d9d12bc514ec46ed5d66fd29b7da1376a4a69c3b692936f1ab2356e2f85 guix-build-68e5aafde3e8/output/dist-archive/bitcoin-68e5aafde3e8.tar.gz
4f32989d4ab1946048ca7caee9a983fa875be262282562f5a3e040f4bf92158e guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/SHA256SUMS.part
ae45df309ae8ada52891efac0a369a69fed4ab93847a7bc4150a62230df4c8d7 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu-debug.tar.gz
0ced227de15cb578567131271e2effe80681b4d7a436c92bf1caec735a576fa4 guix-build-68e5aafde3e8/output/powerpc64-linux-gnu/bitcoin-68e5aafde3e8-powerpc64-linux-gnu.tar.gz
26fc5d2ccc1bc17ee0a146cacada6f4909d90c136ae640c8337332adce414ee0 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/SHA256SUMS.part
9956b544d90a62a8ba9fc9dc6b6b7f0efe193357332ec19e88053a89d4aab37e guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu-debug.tar.gz
be8e39ceea1d36086ce5fa93bfb138c68d3bdf0dd6950b192dfa27a65cce3836 guix-build-68e5aafde3e8/output/powerpc64le-linux-gnu/bitcoin-68e5aafde3e8-powerpc64le-linux-gnu.tar.gz
a7755edc394972885c4c77a7798007e5ba4126b177c4ff6224275c4fb8f3b1c4 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/SHA256SUMS.part
b6d252993d8aae7582ad6385fe53c61c54c284c68ece6cb2b2d1ac9554e06139 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu-debug.tar.gz
bb4860f3bbd815f800333124ff901d880741792ab47097f49bda3a6931144da0 guix-build-68e5aafde3e8/output/riscv64-linux-gnu/bitcoin-68e5aafde3e8-riscv64-linux-gnu.tar.gz
3dd17deed5c5935fb28b62dfc7afca5caab0d67862cdcbf3337edae73e1d0c4c guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/SHA256SUMS.part
fa2d68c54fda0816188c81ce2201a77340b82645da2ffe412526f92c297a82df guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.dmg
f6e5accdcd201f522b6426e4d8cc9b3643d4d43a57d268fa0e79ea9a34cfac01 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx-unsigned.tar.gz
4e5a127df957d1c73b65925d685f6620e7bc5667efcb6dcd98be76effc22fc12 guix-build-68e5aafde3e8/output/x86_64-apple-darwin19/bitcoin-68e5aafde3e8-osx64.tar.gz
56ccd216a69acafacbdc6bae0bdcc1faa50b6a51be1aebfa7068206c88b3241a guix-build-68e5aafde3e8/output/x86_64-linux-gnu/SHA256SUMS.part
77b93dd5fad322636853e5b0244ffafd97cc97f3b4b4ee755d5f830b75d77d13 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu-debug.tar.gz
1feda932fc127b900316a232432b91e46e57ee12a81e12a7d888fdc3296219c1 guix-build-68e5aafde3e8/output/x86_64-linux-gnu/bitcoin-68e5aafde3e8-x86_64-linux-gnu.tar.gz
aa7c53ab4164b3736049065c3c24391fc5bd7f26b4bda4aa877c378f0636a125 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/SHA256SUMS.part
5e76148e67aef7e91e70074bfadc08e94373449ac3b966f4343b04d230c778fd guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win-unsigned.tar.gz
34123e3d818beeb70113caeda66945bc7cb9d9e987515d5b149bd17b4b38da90 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-debug.zip
2bba7f40a2b23c6ea3d47c4f564ab54201bf27f7f57103a98cc9bceea4e70c4d guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64-setup-unsigned.exe
0e7e124144af4a92a4344cf70a3b7c06fbd2b8782aee7ede7263893afa3a5ef0 guix-build-68e5aafde3e8/output/x86_64-w64-mingw32/bitcoin-68e5aafde3e8-win64.zip
```
ACKs for top commit:
laanwj:
Code review ACK 68e5aafde3
Tree-SHA512: 5c25249cc178b9d54159e268390c974b739df9458d773e23c14b14d808f87f7afe314058b3c068601a9132042321973b0c9b6f81becb925665eca2738ae9a613
cf7292597e configure.ac: remove Bashism (Matt Whitlock)
Pull request description:
Configure scripts are supposed to adhere to the POSIX shell language. The POSIX `test` builtin does not implement an `==` operator. Bash does, but not all systems have Bash installed as `/bin/sh`. In particular, many systems use the lighter-weight Dash as the default POSIX shell. Dash emits the following error when running `configure`:
```
./configure: 39065: test: xno: unexpected operator
```
This PR removes the Bashism and restores correct operation with POSIX-compliant shells like Dash.
ACKs for top commit:
katesalazar:
ACK cf7292597e.
laanwj:
Code review ACK cf7292597e
Tree-SHA512: 578c873fba52e0472baed9e024bddcf58a0e088600bd5854f3011f1f8d135773ad923bb16baefc960d17ecedee9cc980b36aaa70fb32eb9bc7de93f7fe60541d
From what I can see the only platform this drops support for is CentOS
7. CentOS 7 reached the end of it's "full update" support at the end of
2020. It does receive maintenance updates until 2024, however I don't
think supporting glibc 2.17 until 2024 is realistic. Note that anyone
wanting to self-compile and target a glibc 2.17 runtime could build with
--disable-threadlocal.
glibc 2.18 was released in August 2013.
https://sourceware.org/legacy-ml/libc-alpha/2013-08/msg00160.html
29173d6c6c ubsan: add minisketch exceptions (Cory Fields)
54b5e1aeab Add thin Minisketch wrapper to pick best implementation (Pieter Wuille)
ee9dc71c1b Add basic minisketch tests (Pieter Wuille)
0659f12b13 Add minisketch dependency (Gleb Naumenko)
0eb7928ab8 Add MSVC build configuration for libminisketch (Pieter Wuille)
8bc166d5b1 build: add minisketch build file and include it (Cory Fields)
b2904ceb85 build: add configure checks for minisketch (Cory Fields)
b6487dc4ef Squashed 'src/minisketch/' content from commit 89629eb2c7 (fanquake)
Pull request description:
This takes over #21859, which has [recently switched](https://github.com/bitcoin/bitcoin/pull/21859#issuecomment-921899200) to my integration branch. A few more build issues came up (and have been fixed) since, and after discussing with sipa it was decided I would open a PR to shepherd any final changes through.
> This adds a `src/minisketch` subtree, taken from the master branch of https://github.com/sipa/minisketch, to prepare for Erlay implementation (see #21515). It gets configured for just supporting 32-bit fields (the only ones we're interested in in the context of Erlay), and some code on top is added:
> * A very basic unit test (just to make sure compilation & running works; actual correctness checking is done through minisketch's own tests).
> * A wrapper in `minisketchwrapper.{cpp,h}` that runs a benchmark to determine which field implementation to use.
Only changes since my last update to the branch in the previous PR have been rebasing on master and fixing an issue with a header in an introduced file.
ACKs for top commit:
naumenkogs:
ACK 29173d6c6c
Tree-SHA512: 1217d3228db1dd0de12c2919314e1c3626c18a416cf6291fec99d37e34fb6eec8e28d9e9fb935f8590273b8836cbadac313a15f05b4fd9f9d3024c8ce2c80d02
AC_DEFINE'd values won't be passed down to minisketch because it does not
use bitcoin-config.h. Thus we need a way to know if we should manually add
defines for minisketch files.
17ae2601c7 build: remove build stubs for external leveldb (Cory Fields)
Pull request description:
Presumably these stubs indicate to packagers that external leveldb is meant to be supported in some way. It is not. Remove the stubs to avoid sending any mixed messages.
For context, this was reported on IRC:
> \<Talkless> bitcoind fails to start with undefined symbol: _ZTIN7leveldb6LoggerE in Debian Sid after leveldb upgraded from 1.22 to 1.23: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=996486
ACKs for top commit:
fanquake:
ACK 17ae2601c7
hebasto:
ACK 17ae2601c7. I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 2f1ac2cb30dac64791933a245a2b66ce237bde3955e6f4a6b7ec181248f77a9b1b10597d865d3e2c2b6def696af70de40e905ec274e4ae7cccd1daf461473957
These tests are failing when run against OpenSSL 3, and have been
removed upstream, https://github.com/bitcoin-core/secp256k1/pull/983, so
disabled them for now to avoid `make check` failures.
Note that this will also remove warning output from our build, due to
the use of deprecated OpenSSL API functions. See #23048.
0f95247246 Integrate univalue into our buildsystem (Cory Fields)
9b49ed656f Squashed 'src/univalue/' changes from 98fadc0909..a44caf65fe (fanquake)
Pull request description:
This PR more tightly integrates building Univalue into our build system. This follows the same approach we use for [LevelDB](https://github.com/bitcoin-core/leveldb/), ([`Makefile.leveldb.include`](https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.leveldb.include)), and [CRC32C](https://github.com/bitcoin-core/crc32c) ([`Makefile.crc32c.include`](https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.crc32c.include)), and will be the same approach we use for [minisketch](https://github.com/sipa/minisketch); see #23114.
This approach yields a number of benefits, including:
* Faster configuration due to one less subconfigure being run during `./configure` i.e 22s with this PR vs 26s
* Faster autoconf i.e 13s with this PR vs 17s
* Improved caching
* No more issues with compiler flags i.e https://github.com/bitcoin/bitcoin/pull/12467
* More direct control means we can build exactly the objects we want
There might be one argument against making this change, which is that builders should have the option to use "proper shared/system libraries". However, I think that falls down for a few reasons. The first being that we already don't support building with a number of system libraries (secp256k1, leveldb, crc32c); some for good reason. Univalue is really the odd one out at the moment.
Note that the only fork of Core I'm aware of, that actively patches in support for using system libs, also explicitly marks them as ["DANGEROUS"](a886811721/configure.ac (L1430)) and ["NOT SUPPORTED"](a886811721/configure.ac (L1312)). So it would seem they exist more to satisfy a distro requirement, as opposed to something that anyone should, or would actually use in practice.
PRs like #22412 highlight the "issue" with us operating with our own Univalue fork, where we actively fix bugs, and make improvements, when upstream (https://github.com/jgarzik/univalue) may not be taking those improvements, and by all accounts, is not currently actively maintained. Bitcoin Core should not be hamstrung into not being able to fix bugs in a library, and/or have to litter our source with "workarounds", i.e #22412, for bugs we've already fixed, based on the fact that an upstream project is not actively being maintained. Allowing builders to use system libs is really only exacerbating this problem, with little benefit to our project. Bitcoin Core is not quite like your average piece of distro packaged software.
There is the potential for us to give the same treatment to libsecp256k1, however it seems doing that is currently less straightforward.
ACKs for top commit:
dongcarl:
ACK 0f95247246 less my comment above, always nice to have an include-able `sources.mk` which makes integration easier.
theuni:
ACK 0f95247246. Thanks fanquake for keeping this going.
Tree-SHA512: a7f2e41ee7cba06ae72388638e86b264eca1b9a8b81c15d1d7b45df960c88c3b91578b4ade020f8cc61d75cf8d16914575f9a78fa4cef9c12be63504ed804b99
Presumably these stubs indicate to packagers that external leveldb is meant to
be supported in some way. It is not. Remove the stubs to avoid sending any
mixed messages.
aa69fd6caf build: Drop -Wno-unused-local-typedef (Hennadii Stepanov)
672e8c5d07 build: remove -Wunused-variable (fanquake)
5239af0574 build: remove -Wswitch (fanquake)
0375906e0a build: use loop-analysis over range-loop-analysis (fanquake)
12712fa2c4 build: remove -Wsign-compare (fanquake)
Pull request description:
This remove the addition of flags that are already part of other options, such as `-Wall` or `-Wextra`; see each commit message for details. All of the flags being removed here already exist as part of `-Wall` as of GCC 8, or, for Clang, all exist in `-Wmost` (included in `-Wall)`, or as part of `-Wextra` as of Clang 7. Both of which are our minimum required compilers.
Also cherry-picks one change from #21458.
To give an example of how GCCs `-Wall` has changed over the last few releases:
### 11.x to trunk (12.x)
Added:
```bash
-Wzero-length-bounds
-Wmismatched-dealloc
-Wmismatched-new-delete (only for C/C++)
```
### 10.x to 11.x
Added:
```bash
-Warray-parameter=2 (C and Objective-C only)
-Wrange-loop-construct (only for C++)
-Wsizeof-array-div
-Wvla-parameter (C and Objective-C only)
```
Removed:
```bash
-Wenum-conversion in C/ObjC;
```
### 9.x to 10.x
Added:
```bash
-Wenum-conversion in C/ObjC;
-Wformat-overflow
-Wformat-truncation
-Wzero-length-bounds
```
### 8.x to 9.x
Added:
```bash
-Wpessimizing-move
```
Removed:
```bash
-Wstringop-truncation
```
### 7.x to 8.x
Added:
```bash
-Wcatch-value (C++ and Objective-C++ only)
-Wmissing-attributes
-Wmultistatement-macros
-Wrestrict
-Wsizeof-pointer-div
-Wstringop-truncation
```
[Clang Warning Options](https://clang.llvm.org/docs/DiagnosticsReference.html)
[GCC Warning Options](https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html)
ACKs for top commit:
meshcollider:
utACK aa69fd6caf
Tree-SHA512: 34dde6bd773c864202c151eaa35f902d03fb531c27fe5e1ef659225da03acade2efe5df56df3efb4df5bbded3d395348ce03c25b837fce83be53af3352f0f2bc
ce69e18947 scripts: remove pixie.py (fanquake)
00b85d0b13 scripts: only parse the binary once in security-check.py (fanquake)
cad40a5b16 scripts: use LIEF for ELF checks in security-check.py (fanquake)
8242ae230e scripts: only parse the binary once in symbol-check.py (fanquake)
309eac9019 scripts: use LIEF for ELF checks in symbol-check.py (fanquake)
610a8a8e39 test-*-check: Pass in *FLAGS and compile with them (Carl Dong)
Pull request description:
This finishes the transition to using LIEF for the ELF symbol and security checks.
Note that there's currently a work around used for identifying RISCV binaries (just checking the interpreter). I've sent a PR upstream, https://github.com/lief-project/LIEF/pull/562, and we should be able to drop that when using LIEF 0.12.0 and onwards.
ACKs for top commit:
dongcarl:
Code Review ACK ce69e18947
laanwj:
Code review ACK ce69e18947
Tree-SHA512: 911ba693cd9777ad1fc1f66dff6c4d3630a907351215380cbde5b14a4bbf5cf7eebf52eafa7e86b27deabd2d93d1b403f34aabd356b5ceaab3cc6e9941a01dd4
38fd709fa5 build: make --enable-werror just -Werror (fanquake)
Pull request description:
No longer special case a set of warnings, to make up our own -Werror,
just use -Werror outright. This shouldn't really have any effect on
existing builders, who were already using `--enable-werror`, and is more
inline with what they would expect `--enable-werror` to be, which is
erroring on any/all warnings.
We keep `-Wno-error=return-type` because we know that is broken when using
mingw-w64. It should only be applied when cross-compiling for Windows.
Similar to the change in #20544, but with (hopefully) less work-arounds,
and other bundled changes. A step towards some configure "cleanups".
ACKs for top commit:
hebasto:
ACK 38fd709fa5 (also see https://github.com/bitcoin/bitcoin/pull/23149#issuecomment-940420776), tested:
MarcoFalke:
Concept ACK 38fd709fa5
Tree-SHA512: 37f1857d9408442cab63e40f9280427b73e09cdf03146b19c1339d7e44abd78e93df7f270ca1da0e83b79343cd3ea915f7b9e4e347488b5bc5ceaaa7540e5926
This addresses issues like the one in #12467, where some of our compiler flags
end up being dropped during the subconfigure of Univalue. Specifically, we're
still using the compiler-default c++ version rather than forcing c++17.
We can drop the need subconfigure completely in favor of a tighter build
integration, where the sources are listed separately from the build recipes,
so that they may be included directly by upstream projects. This is
similar to the way leveldb build integration works in Core.
Core benefits of this approach include:
- Better caching (for ex. ccache and autoconf)
- No need for a slow subconfigure
- Faster autoconf
- No more missing compile flags
- Compile only the objects needed
There are no benefits to Univalue itself that I can think of. These changes
should be a no-op there, and to downstreams as well until they take advantage
of the new sources.mk.
This also removes the option to use an external univalue to avoid similar ABI
issues with mystery binaries.
Co-authored-by: fanquake <fanquake@gmail.com>
0bc666b053 doc: add info for debugging with relative paths (S3RK)
a8b515c317 configure: keep relative paths in debug info (S3RK)
Pull request description:
This is a follow-up for #20353 that fixes#21885
It also adds a small section to assist debugging without absolute paths in debug info.
ACKs for top commit:
kallewoof:
Tested ACK 0bc666b053
Zero-1729:
Light crACK 0bc666b053
Tree-SHA512: d4b75183c3d3a0f59fe786841fb230581de87f6fe04cf7224e4b89c520d45513ba729d4ad8c0e62dd1dbaaa7a25741f04d036bc047f92842e76c9cc31ea47fb2
4747da3a5b Add syscall sandboxing (seccomp-bpf) (practicalswift)
Pull request description:
Add experimental syscall sandboxing using seccomp-bpf (Linux secure computing mode).
Enable filtering of system calls using seccomp-bpf: allow only explicitly allowlisted (expected) syscalls to be called.
The syscall sandboxing implemented in this PR is an experimental feature currently available only under Linux x86-64.
To enable the experimental syscall sandbox the `-sandbox=<mode>` option must be passed to `bitcoind`:
```
-sandbox=<mode>
Use the experimental syscall sandbox in the specified mode
(-sandbox=log-and-abort or -sandbox=abort). Allow only expected
syscalls to be used by bitcoind. Note that this is an
experimental new feature that may cause bitcoind to exit or crash
unexpectedly: use with caution. In the "log-and-abort" mode the
invocation of an unexpected syscall results in a debug handler
being invoked which will log the incident and terminate the
program (without executing the unexpected syscall). In the
"abort" mode the invocation of an unexpected syscall results in
the entire process being killed immediately by the kernel without
executing the unexpected syscall.
```
The allowed syscalls are defined on a per thread basis.
I've used this feature since summer 2020 and I find it to be a helpful testing/debugging addition which makes it much easier to reason about the actual capabilities required of each type of thread in Bitcoin Core.
---
Quick start guide:
```
$ ./configure
$ src/bitcoind -regtest -debug=util -sandbox=log-and-abort
…
2021-06-09T12:34:56Z Experimental syscall sandbox enabled (-sandbox=log-and-abort): bitcoind will terminate if an unexpected (not allowlisted) syscall is invoked.
…
2021-06-09T12:34:56Z Syscall filter installed for thread "addcon"
2021-06-09T12:34:56Z Syscall filter installed for thread "dnsseed"
2021-06-09T12:34:56Z Syscall filter installed for thread "net"
2021-06-09T12:34:56Z Syscall filter installed for thread "msghand"
2021-06-09T12:34:56Z Syscall filter installed for thread "opencon"
2021-06-09T12:34:56Z Syscall filter installed for thread "init"
…
# A simulated execve call to show the sandbox in action:
2021-06-09T12:34:56Z ERROR: The syscall "execve" (syscall number 59) is not allowed by the syscall sandbox in thread "msghand". Please report.
…
Aborted (core dumped)
$
```
---
[About seccomp and seccomp-bpf](https://en.wikipedia.org/wiki/Seccomp):
> In computer security, seccomp (short for secure computing mode) is a facility in the Linux kernel. seccomp allows a process to make a one-way transition into a "secure" state where it cannot make any system calls except exit(), sigreturn(), and read() and write() to already-open file descriptors. Should it attempt any other system calls, the kernel will terminate the process with SIGKILL or SIGSYS. In this sense, it does not virtualize the system's resources but isolates the process from them entirely.
>
> […]
>
> seccomp-bpf is an extension to seccomp that allows filtering of system calls using a configurable policy implemented using Berkeley Packet Filter rules. It is used by OpenSSH and vsftpd as well as the Google Chrome/Chromium web browsers on Chrome OS and Linux. (In this regard seccomp-bpf achieves similar functionality, but with more flexibility and higher performance, to the older systrace—which seems to be no longer supported for Linux.)
ACKs for top commit:
laanwj:
Code review and lightly tested ACK 4747da3a5b
Tree-SHA512: e1c28e323eb4409a46157b7cc0fc29a057ba58d1ee2de268962e2ade28ebd4421b5c2536c64a3af6e9bd3f54016600fec88d016adb49864b63edea51ad838e17
No longer special case a set of warnings, to make up our own -Werror,
just use -Werror outright. This shouldn't really have any effect on
existing builders, who were already using --enable-werror, and is more
inline with what they would expect --enable-werror to be, which is
erroring on any/all warnings.
We keep -Wno-error=return-type because we know that is broken when using
mingw-w64. It should only be applied when cross-compiling for Windows.
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
It was [pointed out in #23030](https://github.com/bitcoin/bitcoin/pull/23030#issuecomment-922893367) that we might be able to get rid of our weak linking of [`getauxval()`](https://man7.org/linux/man-pages/man3/getauxval.3.html) (`HAVE_WEAK_GETAUXVAL`) entirely, with only Android being a potential holdout:
> I wonder if it's time to get rid of HAVE_WEAK_GETAUXVAL. I think it's confusing. Either we build against a C library that has this functionality, or not. We don't do this weak linking thing for any other symbols and recently got rid of the other glibc backwards compatibility stuff.
> Unless there is still a current platform that really needs it (Android?), I'd prefer to remove it from the build system, it has caused enough issues.
After looking at Android further, it would seem that given we are moving to using `std::filesystem`, which [requires NDK version 22 and later](https://github.com/android/ndk/wiki/Changelog-r22), and `getauxval` has been available in the since [API version 18](https://developer.android.com/ndk/guides/cpu-features#features_using_libcs_getauxval3), that shouldn't really be an issue. Support for API levels < 19 will be dropped with the NDK 24 release, and according to [one website](https://apilevels.com/), supporting API level 18+ will cover ~99% of devices. Note that in the CI we currently build with NDK version 22 and API level 28.
The other change in this PR is removing the include of headers for ARM intrinsics, from the check for strong `getauxval()` support in configure, as they shouldn't be needed. Including these headers also meant that the check would basically only succeed when building for ARM. This would be an issue if we remove weak linking, as we wouldn't detect `getauxval()` as supported on other platforms. Note that we also use `getauxval()` in our RNG when it's available.
I've checked that with these changes we detect support for strong `getauxval()` on Alpine (muslibc). On Linux, previously we'd be detecting support for weak getauxval(), now we detect strong support. Note that we already require glibc 2.17, and `getauxval()` was introduced in `2.16`.
This is an alternative / supersedes #23030.
`crc32c`'s hardware accelerated code doesn't handle ARM 32-bit at all.
Make the check in `configure.ac` check for this architecture explicitly.
For the release binaries, the current `configure.ac` check happens
to work: it enables it on aarch64 but disables it for armhf. However
some combination of compiler version and settings might ostensibly cause
this check to succeed on armhf (as reported on IRC). So make the 64-bit
platform requirement explicit.
3ec633ef1a build: improve check for ::(w)system (fanquake)
Pull request description:
`AC_DEFINE()` takes `HAVE_STD__SYSTEM || HAVE_WSYSTEM` literally, meaning you
end up with the following in bitcoin-config.h:
```cpp
/* std::system or ::wsystem */
#define HAVE_SYSTEM HAVE_STD__SYSTEM || HAVE_WSYSTEM
```
This works for the preprocessor, because `HAVE_SYSTEM`, is defined, just unusually. Remove this in favor of setting `have_any_system` in either case, given we don't actually use `HAVE_STD__SYSTEM` or `HAVE_WSYSTEM`, and defining `HAVE_SYSTEM` to 1 thereafter.
ACKs for top commit:
laanwj:
Code review ACK 3ec633ef1a
Tree-SHA512: 02c39ba3179136ec1dc28df026b7fa5d732914c85622298ba7ec880f1ae9324208d322a47be451a5c2ff2e165ad1d446bae92e7018db8e517e7ac38fca25a0a3
At this point, or minimum required glibc is implicitly 2.18, due to
thread_local support being enabled by default. However, users can
disable thread_local support to maintain 2.17 ccompat for now, which is
currently done in the Guix build.
`AC_DEFINE()` takes `HAVE_STD__SYSTEM || HAVE_WSYSTEM` literally, meaning you
end up with the following in bitcoin-config.h:
```cpp
/* std::system or ::wsystem */
#define HAVE_SYSTEM HAVE_STD__SYSTEM || HAVE_WSYSTEM
```
This works for the preprocessor, because `HAVE_SYSTEM`, is defined, just unusually.
Remove this in favor of defining `HAVE_SYSTEM` to 1 in either case, given we
don't actually use `HAVE_STD__SYSTEM` or `HAVE_WSYSTEM`. We just use ::system if
we aren't building for Windows.
2445df4eb3 build: Fix macOS Apple Silicon build with miniupnpc and libnatpmp (Hennadii Stepanov)
Pull request description:
On master (7a49fdc581) the `configure` script does not pick up Homebrew's `miniupnpc` and `libnatpmp` packages on macOS Apple Silicon:
```
% ./configure --with-miniupnpc
...
checking for miniupnpc/miniupnpc.h... no
checking for miniupnpc/upnpcommands.h... no
checking for miniupnpc/upnperrors.h... no
...
checking whether to build with support for UPnP... configure: error: "UPnP requested but cannot be built. Use --without-miniupnpc."
```
```
% ./configure --with-natpmp
...
checking for natpmp.h... no
...
checking whether to build with support for NAT-PMP... configure: error: NAT-PMP requested but cannot be built. Use --without-natpmp
```
The preferred Homebrew [prefix for Apple Silicon](https://docs.brew.sh/Installation) is `/opt/homebrew`. Therefore, if we do not use `pkg-config` to detect packages, we should set the `CPPFLAGS` and `LDFLAGS` variables for them explicitly.
ACKs for top commit:
Zero-1729:
re-tACK 2445df4eb3 (re-tested on an M1 Machine running macOS 11.4).
jarolrod:
re-ACK 2445df4eb3
Tree-SHA512: d623d26492f463812bf66ca519847ff4b23d517466b6c51c3caf3642a582d02e5f03ce57915742b29f01bf9bceb731a3978ef9a5fdc82e568bcb62548eda758a
b367745cfe ci: Make Cirrus CI Windows build with --enable-werror (Hennadii Stepanov)
c713bb2b24 Fix Windows build with --enable-werror on Ubuntu Focal (Hennadii Stepanov)
Pull request description:
This PR makes possible to cross-compile Windows build with `--enable-werror --enable-suppress-external-warnings`.
Some problems are fixed, others are silenced.
Also `--enable-werror` is enabled for Cirrus CI Windows build (the last one on Cirrus CI without `--enable-werror`).
ACKs for top commit:
practicalswift:
cr ACK b367745cfe: patch looks correct
laanwj:
Code review ACK b367745cfe
vasild:
ACK b367745cfe
jarolrod:
ACK b367745cfe
Tree-SHA512: 64f5c99b7dad4c0efce80cd45d7074f275bd8411235dc9e0841287bdab64b812c6f8f9d632c35531d0b8210148531f53aaaac77be7699b29d2d6aaae304dbee0
The preferred Homebrew prefix for Apple Silicon is /opt/homebrew.
Therefore, if we do not use pkg-config to detect packages, we should set
the CPPFLAGS and LDFLAGS variables for them explicitly.
e4c8bb62e4 build: Fix undefined reference to __mulodi4 (Hennadii Stepanov)
Pull request description:
When compiling with clang on 32-bit systems the `__mulodi4` symbol is defined in compiler-rt only.
Fixes#21294.
See more:
- https://bugs.llvm.org/show_bug.cgi?id=16404
- https://bugs.llvm.org/show_bug.cgi?id=28629
ACKs for top commit:
MarcoFalke:
tested-only ACK e4c8bb62e4
luke-jr:
utACK e4c8bb62e4
fanquake:
ACK e4c8bb62e4 - it's a bit of an awkward workaround to carry, but at-least it's contained to the fuzzers.
Tree-SHA512: 93edb4ed568027702b1b9aba953ad50889b834ef97fde3cb99d1ce70076d9c00aa13f95c86b12d6f59b24fa90108d93742f920e15119901a2848fb337ab859a1
3c4c8e79ba build: Add -Werror=implicit-fallthrough compile flag (Hennadii Stepanov)
014110c47d Use C++17 [[fallthrough]] attribute, and drop -Wno-implicit-fallthrough (Hennadii Stepanov)
Pull request description:
ACKs for top commit:
fanquake:
ACK 3c4c8e79ba - looks ok to me now. Checked that warnings occur in our code & leveldb by removing a `[[fallthrough]]` or `FALLTHROUGH_INTENDED`.
jarolrod:
ACK 3c4c8e79ba
theStack:
ACK 3c4c8e79ba
Tree-SHA512: 4dce91f0f26b8a3de09bd92bb3d7e1995e078e3a8b3ff861c4fbf6c0b32b2327d063633b07b89c4aa94a1141d7f78d46d9d43ab8df865273e342693ad30645b6
8f7704d032 build: improve detection of eBPF support (fanquake)
Pull request description:
Just checking for the `sys/sdt.h` header isn't enough, as systems like macOS have the header, but it doesn't actually have the `DTRACE_PROBE*` probes, which leads to [compile failures](https://github.com/bitcoin/bitcoin/pull/22006#issuecomment-859559004). The contents of `sys/sdt.h` in the macOS SDK is:
```bash
#ifndef _SYS_SDT_H
#define _SYS_SDT_H
/*
* This is a wrapper header that wraps the mach visible sdt.h header so that
* the header file ends up visible where software expects it to be. We also
* do the C/C++ symbol wrapping here, since Mach headers are technically C
* interfaces.
*
* Note: The process of adding USDT probes to code is slightly different
* than documented in the "Solaris Dynamic Tracing Guide".
* The DTRACE_PROBE*() macros are not supported on Mac OS X -- instead see
* "BUILDING CODE CONTAINING USDT PROBES" in the dtrace(1) manpage
*
*/
#include <sys/cdefs.h>
__BEGIN_DECLS
#include <mach/sdt.h>
__END_DECLS
#endif /* _SYS_SDT_H */
```
The `BUILDING CODE CONTAINING USDT PROBES` section from the dtrace manpage is available [here](https://gist.github.com/fanquake/e56c9866d53b326646d04ab43a8df9e2), and outlines the more involved process of using USDT probes on macOS.
ACKs for top commit:
jb55:
utACK 8f7704d032
practicalswift:
cr ACK 8f7704d032
hebasto:
ACK 8f7704d032, tested on macOS Big Sur 11.4 (20F71) and on Linux Mint 20.1 (x86_64) with depends.
Tree-SHA512: 5f1351d0ac2e655fccb22a5454f415906404fdaa336fd89b54ef49ca50a442c44ab92d063cba3f161cb8ea0679c92ae3cd6cfbbcb19728cac21116247a017df5
2f5bdcbc31 gui: misc external signer fixes and translation hints (Sjors Provoost)
d672404466 refactor: make ExternalSigner NetworkArg() and m_chain private (Sjors Provoost)
4455145e26 refactor: reduce #ifdef ENABLE_EXTERNAL_SIGNER usage (Sjors Provoost)
5be90c907e build: enable external signer by default (Sjors Provoost)
7d9453041b refactor: clean up external_signer.h includes (Sjors Provoost)
fc0eca31b3 fuzz: fix fuzz binary linking order (Sjors Provoost)
Pull request description:
This follows the introduction of GUI support in https://github.com/bitcoin-core/gui/pull/4
I don't think we should expect GUI users to self compile. This also enables external signer support by default for RPC users.
In addition this PR reduces the number of `#ifdef ENABLE_EXTERNAL_SIGNER`, which also fixes#21919. When compiled with `--disable-external-signer` such wallets can't be created in RPC or GUI, but they can be loaded. Attempting any action that calls HWI will trigger an error.
Side-note: this PR may or may not (currently) break CI for the GUI repository, as explained here: https://github.com/bitcoin-core/gui/pull/4#issuecomment-769859001
ACKs for top commit:
achow101:
ACK 2f5bdcbc31
hebasto:
re-ACK 2f5bdcbc31
Tree-SHA512: 1b71c5a8bea2be077ee9fa33a01130c957a0cf90951d4b7b04d3d0ef826bb77e474c3963abddfef2e2c1ea99d9c72cd2302d1eb9b5fcb7ba0bd2a625f006aa05
Just checking for the `sys/sdt.h` header isn't enough, as systems like
macOS have the header, but it doesn't actually have the dtrace probes,
which leads to compile failures.
5c7ee1b2da libsecp256k1 no longer has --with-bignum= configure option (Pieter Wuille)
bdca9bcb6c Squashed 'src/secp256k1/' changes from 3967d96bf1..efad3506a8 (Pieter Wuille)
cabb566123 Disable certain false positive warnings for libsecp256k1 msvc build (Pieter Wuille)
Pull request description:
This updates our src/secp256k1 subtree to the latest upstream master. The changes include:
* The introduction of safegcd-based modular inverses, reducing ECDSA signing time by 25%-30% and ECDSA verification time by 15%-17%.
* [Original paper](https://gcd.cr.yp.to/papers.html) by Daniel J. Bernstein and Bo-Yin Yang
* [Implementation](https://github.com/bitcoin-core/secp256k1/pull/767) by Peter Dettman; [final](https://github.com/bitcoin-core/secp256k1/pull/831) version
* [Explanation](https://github.com/bitcoin-core/secp256k1/blob/master/doc/safegcd_implementation.md) of the algorithm using Python snippets
* [Analysis](https://github.com/sipa/safegcd-bounds) of the maximum number of iterations the algorithm needs
* [Formal proof in Coq](https://medium.com/blockstream/a-formal-proof-of-safegcd-bounds-695e1735a348) by Russell O'Connor, for a high-level equivalent algorithm
* Removal of libgmp as an (optional) dependency (which wasn't used in the Bitcoin Core build)
* CI changes (Travis -> Cirrus)
* Build system improvements
ACKs for top commit:
laanwj:
Tested ACK 5c7ee1b2da
Tree-SHA512: ad8ac3746264d279556a4aa7efdde3733e114fdba8856dd53218588521f04d83950366f5c1ea8fd56329b4c7fe08eedf8e206f8f26dbe3f0f81852e138655431
e9f948c727 build: Convert warnings into errors when testing for -fstack-clash-protection (Hennadii Stepanov)
Pull request description:
Apple clang version 12.0.5 (clang-1205.0.22.9) that is a part of Xcode 12.5, and is based on LLVM clang 11.1.0, fires spammy warnings:
```
clang: warning: argument unused during compilation: '-fstack-clash-protection' [-Wunused-command-line-argument]
```
From the https://github.com/apple/llvm-project:
```
$ git log --oneline | grep 'stack-clash-protection'
00065d5cbd02 Revert "-fstack-clash-protection: Return an actual error when used on unsupported OS"
4d59c8fdb955 -fstack-clash-protection: Return an actual error when used on unsupported OS
df3bfaa39071 [Driver] Change -fnostack-clash-protection to -fno-stack-clash-protection
68e07da3e5d5 [clang][PowerPC] Enable -fstack-clash-protection option for ppc64
515bfc66eace [SystemZ] Implement -fstack-clash-protection
e67cbac81211 Support -fstack-clash-protection for x86
454621160066 Revert "Support -fstack-clash-protection for x86"
0fd51a4554f5 Support -fstack-clash-protection for x86
658495e6ecd4 Revert "Support -fstack-clash-protection for x86"
e229017732bc Support -fstack-clash-protection for x86
b03c3d8c6209 Revert "Support -fstack-clash-protection for x86"
4a1a0690ad68 Support -fstack-clash-protection for x86
f6d98429fcdb Revert "Support -fstack-clash-protection for x86"
39f50da2a357 Support -fstack-clash-protection for x86
```
I suppose, that Apple clang-1205.0.22.9 ends with on of the "Revert..." commits.
This PR prevents using of the `-fstack-clash-protection` flag if it causes warnings.
---
System: macOS Big Sur 11.3 (20E232).
ACKs for top commit:
jarolrod:
re-ACK e9f948c727
Sjors:
tACK e9f948c727 on macOS 11.3.1
Tree-SHA512: 30186da67f9b0f34418014860c766c2e7f622405520f1cbbc1095d4aa4038b0a86014d76076f318a4b1b09170a96d8167c21d7f53a760e26017f486e1a7d39d4
When building with Clang, if `-fstack-clash-protection` is used with an
unsupported target, it may result in hundreds of
`-Wunused-command-line-argument` warnings at compile time. This is
currently the case when building for at least Darwin using Apple or LLVM
Clang.
Unsupported targets may also include *BSD, however that is changing; see
further discussion in https://reviews.llvm.org/D92245 and
https://reviews.freebsd.org/D27366.
Note that this option is already skipped for Windows.
a5491882a0 build: fix configuring when building depends with NO_BDB=1 (fanquake)
Pull request description:
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets), `./configure` will fail as it still tries to find bdb. i.e:
```bash
make -C depends/ NO_QT=1 NO_BDB=1 NO_UPNP=1 NO_ZMQ=1 NO_NATPMP=1 -j8
...
copying packages: native_b2 boost libevent sqlite
./autogen.sh
./configure --prefix=/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu
...
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of bdb, without opting out of wallets entirely, and still configure successfully. I think I've tested across most potential configurations. i.e:
```bash
./configure (bdb and sqlite on system)
bdb & sqlite are both are available
./configure --without-bdb (bdb and sqlite on system)
only sqlite
./configure --without-sqlite (bdb and sqlite on system)
only bdb
./configure --disable-wallet (bdb and sqlite on system)
neither bdb or sqlite
depends NO_WALLET=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
neither bdb or sqlite
depends NO_BDB=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only sqlite
depends NO_SQLITE=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only bdb
depends
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
bdb and sqlite
```
ACKs for top commit:
laanwj:
Code review ACK a5491882a0
jarolrod:
ACK a5491882a0
Tree-SHA512: baf7d2543a401db0d846095415ff449c04ecfb4a74c734dc51e79453702f9051210daeef686970f11fcffd32cdfadbc58acd54f0706aceecfb3edb0ff17310d7
7abac98d3e configure: Support -f{debug,macro}-prefix-map (Anthony Towns)
Pull request description:
When bitcoin is checked out in two directories (eg via git worktree) object files between the two will differ due to the full path being included in the debug section. `-fdebug-prefix-map` is used to replace this with "." to avoid this unnecessary difference and allow ccache to share objects between worktrees (provided the source and compile options are the same).
Also provide `-fmacro-prefix-map` if supported so that the working dir is not encoded in `__FILE__` macros.
ACKs for top commit:
practicalswift:
cr ACK 7abac98d3e: patch looks correct
fanquake:
ACK 7abac98d3e
Tree-SHA512: b6a37c1728ec3b2e552f244da0e66db113c1e7662c7ac502e12ff466f3dbfbfefae12695ca135137c50dbb1c4c5d84059116c0cd09b391a17466dc77b8726679
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets),
./configure will fail as it still tries to find bdb. i.e:
```bash
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of
bdb without opting out of wallets entirely, and still configure
successfully.
a4e970adb6 build: enable -Wdocumentation if suppressing external warnings (fanquake)
3b0078f958 doc: fixup -Wdocumentation issues (fanquake)
c6edcf1c71 build: suppress libevent warnings if supressing external warnings (fanquake)
Pull request description:
Enable `-Wdocumentation` by taking advantage of our `--enable-suppress-external-warnings` flag. Most of the CIs are using this flag now, so any regressions should be caught.
This also required modifying libevents flags when suppressing warnings, as depending on the version being built against, that could generate a large number of warnings. i.e:
```bash
In file included from httpserver.cpp:34:
In file included from ./support/events.h:12:
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:464:11: warning: parameter 'req' not found in the function declaration [-Wdocumentation]
@param req a request object
^~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:465:11: warning: parameter 'databuf' not found in the function declaration [-Wdocumentation]
@param databuf the data chunk to send as part of the reply.
^~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:467:11: warning: parameter 'call' not found in the function declaration [-Wdocumentation]
@param call back's argument.
^~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:939:4: warning: declaration is marked with '@deprecated' command but does not have a deprecation attribute [-Wdocumentation-deprecated-sync]
@deprecated This function is deprecated; you probably want to use
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:946:1: note: add a deprecation attribute to the declaration to silence this warning
char *evhttp_decode_uri(const char *uri);
^
__AVAILABILITY_INTERNAL_DEPRECATED
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:979:5: warning: declaration is marked with '@deprecated' command but does not have a deprecation attribute [-Wdocumentation-deprecated-sync]
@deprecated This function is deprecated as of Libevent 2.0.9. Use
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:987:1: note: add a deprecation attribute to the declaration to silence this warning
int evhttp_parse_query(const char *uri, struct evkeyvalq *headers);
^
__AVAILABILITY_INTERNAL_DEPRECATED
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:1002:11: warning: parameter 'query_parse' not found in the function declaration [-Wdocumentation]
@param query_parse the query portion of the URI
^~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:1002:11: note: did you mean 'uri'?
@param query_parse the query portion of the URI
^~~~~~~~~~~
uri
69 warnings generated.
```
Note that a lot of these have already been fixed upstream.
ACKs for top commit:
laanwj:
Concept and code review ACK a4e970adb6
practicalswift:
cr ACK a4e970adb6: automatic compiler feedback comes sooner and is more reliable than manual reviewer feedback
jonatack:
Light ACK a4e970adb6 skimmed the changes, clang 11 build is clean with the change, verified -Wdocumentation build warnings with this change when a doc fix was reverted
Tree-SHA512: 57a1e30cffcc8bcceee72d85f58ebe29eae525861c70acb237541bd480c51ede89875c033042c0af376fdbb49fb7f588ef9282a47c6e78f9d4501c41f1b21eb6
The register keyword was deprecated in C++11, and removed in C++17. Now
that we require C++17, we shouldn't have to supress warnings for a
non-existant feature.
0eabb2abed build: Remove unused header from the build system (Hennadii Stepanov)
Pull request description:
The only `#include <miniupnpc/miniwget.h>` was removed in #16659.
ACKs for top commit:
practicalswift:
cr ACK 0eabb2abed
fanquake:
ACK 0eabb2abed
Tree-SHA512: 630da03875c851e80286561eae0f966c89624cbb17b90f70e2bec9a69146e79d088fc176e07a4906915770ac1cdb11341a7a431ea7cf6a59d2816e927486f335
246774e264 depends: fix Qt precompiled headers bug (Igor Cota)
8e7ad4146d depends: disable Qt Vulkan support on Android (Igor Cota)
ba46adaa1a CI: add Android APK build to cirrus (Igor Cota)
7563720e30 CI: add Android APK build script (Igor Cota)
ebfb10cb75 Qt: add Android packaging support (Igor Cota)
Pull request description:
This PR is the third and final piece of the basic Android support puzzle - it depends on https://github.com/bitcoin/bitcoin/pull/16110 and is related to https://github.com/bitcoin/bitcoin/pull/16883. It introduces an `android` directory under `qt` and a simple way to build an Android package of `bitcoin-qt`:
1. Build depends for Android as described in the [README](https://github.com/bitcoin/bitcoin/blob/master/depends/README.md)
2. Configure with one of the resulting prefixes
3. Run `make && make apk` in `src/qt`
The resulting APK files will be in `android/build/outputs/apk`. You can install them manually or with [adb](https://developer.android.com/studio/command-line/adb). One can also open the `android` directory in Android Studio for that integrated development and debugging experience. `BitcoinQtActivity` is your starting point.
Under the hood makefile `apk` target:
1. Renames the `bitcoin-qt` binary to `libbitcoin-qt.so` and copies it over to a folder under `android/libs` depending on which prefix and corresponding [ABI](https://developer.android.com/ndk/guides/abis.html#sa) `bitcoin-qt` was built for
2. Takes `libc++_shared.so` from the Android NDK and puts in the same place. It [must be included](https://developer.android.com/ndk/guides/cpp-support) in the APK
3. Extracts Qt for Android Java support files from the `qtbase` archive in `depends/sources` to `android/src`
There is also just a tiny bit of `ifdef`'d code to make the Qt Widgets menus usable. It's not pretty but it works and is a stepping stone towards https://github.com/bitcoin/bitcoin/pull/16883.
ACKs for top commit:
MarcoFalke:
cr ACK 246774e264
laanwj:
Code review ACK 246774e264
Tree-SHA512: ba30a746576a167545223c35a51ae60bb0838818779fc152c210f5af1413961b2a6ab6af520ff92cbc8dcd5dcb663e81ca960f021218430c1f76397ed4cead6c
Introduce an android directory under qt and allow one to package bitcoin-qt for Android by running make apk.
Add bitcoin-qt Android build instructions.
This has never worked with any of the mingw-w64 compilers we use, and
the -O0 is causing issues for builders applying spectre mitigations.
Recent discussion on https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458
also indicates that this should just not be used on Windows.
e017a913d0 bitcoind: Add -daemonwait option to wait for initialization (Wladimir J. van der Laan)
c3e6fdee6d shutdown: Use RAII TokenPipe in shutdown (Wladimir J. van der Laan)
612f746a8f util: Add RAII TokenPipe (Wladimir J. van der Laan)
Pull request description:
This adds a `-daemonwait` flag that does the same as `-daemon` except that it, from a user perspective, backgrounds the process only after initialization is complete. This is similar to the behaviour of some other software such as c-lightning.
This can be useful when the process launching bitcoind wants to guarantee that either the RPC server is running, or that initialization failed, before continuing. The exit code indicates the initialization result.
The use of the libc function `daemon()` is replaced by a custom implementation which is inspired by the [glibc implementation](https://github.com/lattera/glibc/blob/master/misc/daemon.c#L44), but which also creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is that no MACOS-specific pragmas are needed anymore to silence a deprecation warning.
TODO:
- [x] Factor out `token_read` and `token_write` to an utility, and use them in `shutdown.cpp` as well—this is exactly the same kind of communication mechanism.
- [x] RAII-ify pipe endpoints.
- [x] Improve granularity of the `configure.ac` checks. This currently still checks for the function `daemon()` which makes no sense as it's not used. It should check for individual functions such as
`fork()` and `setsid()` etc—the former being required, the second optional.
- [-] ~~Signal propagation during initialization: if say, pressing Ctrl-C during `-daemonwait` it would be good to pass this SIGINT on to the child process instead of detaching the parent process and letting the child run free.~~ This is not necessary, see https://github.com/bitcoin/bitcoin/pull/21007#issuecomment-769007341.
Future:
- Consider if it makes sense to use this in the RPC tests (there would be no more need for "is RPC ready" polling loops). I think this is out of scope for this PR.
ACKs for top commit:
jonatack:
Tested ACK e017a913d0 checked change since previous review is move-only
Tree-SHA512: 53369b8ca2247e4cf3af8cb2cfd5b3399e8e0e3296423d64be987004758162a7ddc1287b01a92d7692328edcb2da4cf05d279b1b4ef61a665b71440ab6a6dbe2
This adds a `-daemonwait` flag that does the same as `-daemon` except
it, from a user perspective, backgrounds the process only after
initialization is complete.
This can be useful when the process launching bitcoind wants to
guarantee that either the RPC server is running, or that initialization
failed, before continuing. The exit code indicates the initialization
result.
This replaces the use of the libc function `daemon()` by a custom
implementation which is inspired by the glibc implementation, but also
creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is
that no MACOS-specific pragmas are needed anymore to silence a
deprecation warning.
faa06ecc9c build: Bump minimum Qt version to 5.9.5 (Hennadii Stepanov)
Pull request description:
Close#20104.
ACKs for top commit:
laanwj:
Code review ACK faa06ecc9c
jarolrod:
ACK faa06ecc9c
fanquake:
ACK faa06ecc9c - this should be ok to do now.
Tree-SHA512: 7295472b5fd37ffb30f044e88c39d375a5a5187d3f2d44d4e73d0eb0c7fd923cf9949c2ddab6cddd8c5da7e375fff38112b6ea9779da4fecce6f024d05ba9c08
fae216a73d scripted-diff: Rename MakeFuzzingContext to MakeNoLogFileContext (MarcoFalke)
fa4fbec03e scripted-diff: Rename PROVIDE_MAIN_FUNCTION -> PROVIDE_FUZZ_MAIN_FUNCTION (MarcoFalke)
Pull request description:
Split out two renames from #21003:
* `PROVIDE_FUZZ_MAIN_FUNCTION`. *Reason*: This in only used by fuzzing, so the name should indicate that.
* `MakeNoLogFileContext`. *Reason*: Better reflects what the helper does. Also, prepares it to be used in non-fuzz tests in the future.
ACKs for top commit:
practicalswift:
cr ACK fae216a73d: scripted-diff looks correct
Tree-SHA512: e5d347746f5da72b0c86fd4f07ac2e4b3016e88e8c97a830c73bd79d0af6d0245fe7712487fc20344d6cc25958941716c1678124a123930407e3a437265b71df
9bac71350d build: make HAVE_O_CLOEXEC available outside LevelDB (bugfix) (Sebastian Falbesoner)
584fd91d2d init: only use pipe2 if availabile, check in configure (Sebastian Falbesoner)
Pull request description:
The result of the O_CLOEXEC availability check is currently only set in the Makefile and passed to LevelDB (see `LEVELDB_CPPFLAGS_INT` in `src/Makefile.leveldb.include`), but not defined to be used in our codebase. This means that code within the preprocessor conditional `#if HAVE_O_CLOEXEC` was actually never compiled. On the master branch this is currently used for pipe creation in `src/shutdown.cpp`, PR #21007 moves this part to a new module (I found the issue while testing that PR).
The fix is similar to the one in #19803, which solved the same problem for HAVE_FDATASYNC.
In the course of working on the PR it turned out that pipe2 is not available an all platforms, hence a configure check and a corresponding define HAVE_PIPE2 is introduced and used.
The PR can be tested by anyone with a system that has pipe2 and O_CLOEXEC available by putting gibberish into the HAVE_O_CLOEXEC block: on master, everything should compile fine, on PR, the compiler should abort with an error. At least that's my naive way of testing preprocessor logic, happy to hear more sophisticated ways :-)
ACKs for top commit:
laanwj:
Code review ACK 9bac71350d
Tree-SHA512: aec89faf6ba52b6f014c610ebef7b725d9e967207d58b42a4a71afc9f1268fcb673ecc85b33a2a3debba8105a304dd7edaba4208c5373fcef2ab83e48a170051
This option replaces --with-boost-process
This prepares external signer support to be disabled by default.
It adds a configure option to enable this feature and to check
if Boost::Process is present.
This also exposes ENABLE_EXTERNAL_SIGNER to the test suite via test/config.ini
c5da2749e2 build: actually stop configure if Boost isn't available (fanquake)
cad8b527ea build: explicitly install libboost-dev package (fanquake)
Pull request description:
If Boost is not found via AX_BOOST_BASE, we don't actually stop
configuring, only a warning is emitted:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version MINIMUM_REQUIRED_BOOST or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
```
Instead we usually fail when one of the other AX_BOOST_* macros fails to find a library. These macros are slowly being
removed, and in any case, it makes more sense to fail earlier if Boost is missing.
If Boost is unavailable, the failure now looks like:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version 1.58.0 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
configure: error: Boost is not available!
```
Note that we now just pass the version into AX_BOOST_BASE, which fixes it's display in the output (rather than showing `MINIMUM_REQUIRED_BOOST`).
This PR also has a commit that adds `libboost-dev` to our install instructions and CI. This package is currently installed as a side-effect of installing our other libboost-*-dev packages. However as those continue to disappear, it makes sense to install boost-dev explicitly.
ACKs for top commit:
laanwj:
Code review ACK c5da2749e2
MarcoFalke:
Concept ACK c5da2749e2
Tree-SHA512: f866062f9d7d3a2316b6c887f17c664b9cfff41fdc0cb99ca79d641240fb01a5ae0d34140e515bc465219e1b43d5ca84f7c55f48b9c5b45a80ff2795dafd072b
If Boost is not found via AX_BOOST_BASE, we don't actually stop
configuring, only a warning is emitted:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version MINIMUM_REQUIRED_BOOST or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
```
Instead we would usually fail when one of the other
AX_BOOST_* macros fails to find a library. These macros are slowly being
removed, and in any case, it makes more sense to fail earlier if Boost
is missing.
If Boost is unavailable, the failure now looks like:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version 1.58.0 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
configure: error: Boost is not available!
```
Note that we now just pass the version into AX_BOOST_BASE, which fixes
it's display in the output (rather than MINIMUM_REQUIRED_BOOST).
Performing a series of link checks for a Boost component that is
header-only doesn't make much sense, and currently means we just have
another confusing Boost macro in our tree. I'm not sure why this was
originally done this way; maybe Sjors or luke-jr can elaborate
(#15382 (929cda5470))?
The macro also has the side-effect of producing confusing error
messages. i.e in #20744, the CI is currently failing with:
```bash
checking for boostlib >= 1.58.0 (105800) lib path in "/tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/lib"... yes
checking for boostlib >= 1.58.0 (105800)... yes
checking whether the Boost::Process library is available... yes
configure: error: Could not find a version of the Boost::Process library!
```
This isn't useful, given there is no such thing as a `Boost::Process`
library.
This PR just removes the macro entirely, but maintains a `--with-boost-process`
(defaulting to off), flag to configure. Hopefully this will also be
removed, in favour of `--enable-disable-external-signer` if/when #16546
is merged.
060a2a64d4 ci: remove boost thread installation (fanquake)
06e1d7d81d build: don't build or use Boost Thread (fanquake)
7097add83c refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef8 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d4
vasild:
ACK 060a2a64d4
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
We are already testing for this, and our test works correctly with a Darwin
target, where the macro does not. Darwin targets do not support "protected"
visibility.
32cbb06676 build: build fuzz tests by default. (Dan Benjamin)
Pull request description:
This fixes issue #19388. The changes are as follows:
- Add a new flag to configure, --enable-fuzz-binary, which allows building test/fuzz/fuzz regardless of whether we are building to do actual fuzzing
- Set -DPROVIDE_MAIN_FUNCTION whenever --enable-fuzz is no
- Add the following libraries to FUZZ_SUITE_LD_COMMON:
- LIBBITCOIN_WALLET
- SQLLITE_LIBS
- BDB_LIBS
- if necessary, some or all of:
- NATPMP_LIBS
- MINIUPNPC_LIBS
- LIBBITCOIN_ZMQ / ZMQ_LIBS
Fixes #19388
ACKs for top commit:
MarcoFalke:
review ACK 32cbb06676📭
Tree-SHA512: c91d713ffe54a3d055daaec02c4317d7e13eed6688821ddc10d894224950b18e276fbdd4acc758c7103b50f34a132b1882b68bc8b60409f97438e0759ced77e1
e9189a750b build: more robustly check for fcf-protection support (fanquake)
Pull request description:
When using Clang 7, we may end up trying to use the flag when it won't
work properly, which can lead to confusing errors. i.e:
```bash
/usr/bin/ld: error: ... <corrupt x86 feature size: 0x8>
```
Use `AX_CHECK_LINK_FLAG` & `--fatal-warnings` to ensure we wont use the flag in this case.
We do this as even when the error is emitted, compilation succeeds, and the binaries produced will run. This means we can't just check if the compiler accepts the flag, or if compilation succeeds (without or without `-Werror`, and/or passing `-Wl,--fatal-warnings`, which may not be passed through to the linker).
This was reported by someone configuring for fuzzing, on Debian 10, where Clang 7 is the default.
See here for a minimal example of the problematic behaviour:
https://gist.github.com/fanquake/9b33555fcfebef8eb8c0795a71732bc6
ACKs for top commit:
pstratem:
tested ACK e9189a750b
MarcoFalke:
not an ACK e9189a750b , I only tested configure on my system (gcc-10, clang-11):
hebasto:
ACK e9189a750b, tested with clang-7, clang-10 and gcc: the `-fcf-protection=full` is not applied for clang-7, but applied for others compilers.
Tree-SHA512: ec24b0cc5523b90139c96cbb33bb98d1e6a24d858c466aa7dfb3c474caf8c50aca53e570fdbc0ff88378406b0ac5d687542452637b1b5fa062e829291b886fc1
This fixes issue #19388. The changes are as follows:
- Add a new flag to configure, --enable-fuzz-binary, which allows building test/fuzz/fuzz regardless of whether we are building to do actual fuzzing
- Set -DPROVIDE_MAIN_FUNCTION whenever --enable-fuzz is no
- Add the following libraries to FUZZ_SUITE_LD_COMMON:
- LIBBITCOIN_WALLET
- SQLLITE_LIBS
- BDB_LIBS
- if necessary, some or all of:
- NATPMP_LIBS
- MINIUPNPC_LIBS
- LIBBITCOIN_ZMQ / ZMQ_LIBS
22eb7930a6 tracing: add tracing framework (William Casarin)
933ab8a720 build: detect sys/sdt.h for eBPF tracing (William Casarin)
Pull request description:
Instead of writing ad-hoc logging everywhere (eg: #19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin 😅 )
The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages:
# Examples
## Network Messages
```
#!/usr/bin/env bpftrace
BEGIN
{
printf("bitcoin net msgs\n");
@start = nsecs;
}
usdt:./src/bitcoind:net:push_message
{
$ip = str(arg0);
$peer_id = (int64)arg1;
$command = str(arg2);
$data_len = arg3;
$data = buf(arg3,arg4);
$t = (nsecs - @start) / 100000;
printf("%zu outbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);
@outbound[$command]++;
}
usdt:./src/bitcoind:net:process_message
{
$ip = str(arg0);
$peer_id = (int64)arg1;
$command = str(arg2);
$data_len = arg3;
$data = buf(arg3,arg4);
$t = (nsecs - @start) / 100000;
printf("%zu inbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);
@inbound[$ip, $command]++;
}
```
$ sudo bpftrace netmsg.bt
output: https://jb55.com/s/b11312484b601fb3.txt
if you look at the bottom of the output you can see a histogram of all the messages grouped by message type and IP. nice!
## IBD Benchmarking
```
#!/usr/bin/env bpftrace
BEGIN
{
printf("IBD to 500,000 bench\n");
}
usdt:./src/bitcoind:CChainState:ConnectBlock
{
$height = (uint32)arg0;
if ($height == 1) {
printf("block 1 found, starting benchmark\n");
@start = nsecs;
}
if ($height >= 500000) {
@end = nsecs;
@duration = @end - @start;
exit();
}
}
END {
printf("duration %d ms\n", @duration / 1000000)
}
```
This one hooks into ConnectBlock and prints the IBD time to height 500,000 starting from the first call to ConnectBlock
Userspace static tracepoints give lots of flexibility without invasive logging code. It's also more flexible than ad-hoc logging code, allowing you to instrument many different aspects of the system without having to enable per-subsystem logging.
Other ideas: tracepoints for lock contention, threads, what else?
Let me know what ya'll think and if this is worth adding to bitcoin.
## TODO
- [ ] docs?
- [x] Integrate systemtap-std-dev/libsystemtap into build (provides the <sys/sdt.h> header)
- [x] ~dtrace macos support? (is this still a thing?)~ going to focus on linux for now
ACKs for top commit:
laanwj:
Tested ACK 22eb7930a6
0xB10C:
Tested ACK 22eb7930a6
Tree-SHA512: 69242242112b679c8a12a22b3bc50252c305894fb3055ae6e13d5f56221d858e58af1d698af55e23b69bdb7abedb5565ac6b45fa5144087b77a17acd04646a75
595a34dbea contrib/signet: Document miner script in README.md (Anthony Towns)
ff7dbdc08a contrib/signet: Add script for generating a signet chain (Anthony Towns)
13762bcc96 Add bitcoin-util command line utility (Anthony Towns)
95d5d5e625 rpc: allow getblocktemplate for test chains when unconnected or in IBD (Anthony Towns)
81c54dec20 rpc: update getblocktemplate with signet rule, include signet_challenge (Anthony Towns)
Pull request description:
Adds `contrib/signet/miner` for mining signet blocks.
Adds `bitcoin-util` cli utility, with the idea being it can provide bitcoin related functionality that does not rely on the ability to access a running node. Only subcommand currently is "grind" which takes a hex-encoded header and grinds its nonce until its nBits is satisfied.
Updates `getblocktemplate` to include `signet_challenge` field, and makes `getblocktemplate` require the signet rule when invoked on the signet change. Removes connectivity and IBD checks from `getblocktemplate` when applied to a test chain (regtest, testnet, signet).
ACKs for top commit:
laanwj:
code review ACK 595a34dbea
Tree-SHA512: 8d43297710fdc1edc58acd9b53e1bd1671e5724f7097b40ab73653715dc8becc70534c4496cbba9290f4dd6538a7a3d5830eb85f83391ea31a3bb5b9d3378cc3
a191e23b8e doc: Add release notes (Hennadii Stepanov)
ae749d12dd doc: Add libnatpmp stuff (Hennadii Stepanov)
e28f9be87a ci: Add libnatpmp-dev package to some builds (Hennadii Stepanov)
5a0185b6c9 gui: Add NAT-PMP network option (Hennadii Stepanov)
a39f7336a3 net: Add -natpmp command line option (Hennadii Stepanov)
28acffd9d5 net: Add NAT-PMP to port mapping loop (Hennadii Stepanov)
a8d9f275d0 net: Add libnatpmp support (Hennadii Stepanov)
58e8364dcd gui: Apply port mapping changes on dialog exit (Hennadii Stepanov)
cf151cc68c scripted-diff: Rename UPnP stuff (Hennadii Stepanov)
4e91b1e24d net: Add flags for port mapping protocols (Hennadii Stepanov)
8b50d1b5bb net: Keep trying to use UPnP when -upnp=1 (Hennadii Stepanov)
28e2961fd6 refactor: Replace magic number with named constant (Hennadii Stepanov)
02ccf69dd6 refactor: Move port mapping code to its own module (Hennadii Stepanov)
Pull request description:
Close#11902
This PR is an alternative to:
- #12288
- #15717
To compile with NAT-PMP support on Ubuntu [`libnatpmp-dev`](https://packages.ubuntu.com/source/bionic/libnatpmp) should be available.
Log excerpt:
```
2020-02-05T20:12:28Z [mapport] NAT-PMP: public address = 95.164.65.194
2020-02-05T20:12:28Z [mapport] AddLocal(95.164.65.194:18333,3)
2020-02-05T20:12:28Z [mapport] NAT-PMP: port mapping successful.
```
See: [`libnatpmp`](https://miniupnp.tuxfamily.org/libnatpmp.html)
---
Some follow-ups are out of this PR's scope:
- mention NAT-PMP library in the version message
- ~integrate NAT-PMP into the GUI~ (already [added](https://github.com/bitcoin/bitcoin/pull/18077#issuecomment-589405068))
ACKs for top commit:
laanwj:
Tested and code review ACK a191e23b8e
Tree-SHA512: 10e19267c21bf30f20ff1abfc882d526049f0e790b95e12f109dc2bed7c0aef45de03eaf967f4e667e7509be04f1873a5c508087393d947205f3aab2ad6d7cf1
9815332d51 test: Change MuHash Python implementation to match cpp version again (Fabian Jahr)
01297fb3ca fuzz: Add MuHash consistency fuzz test (Fabian Jahr)
b111410914 test: Add MuHash3072 fuzz test (Fabian Jahr)
c122527385 bench: Add Muhash benchmarks (Fabian Jahr)
7b1242229d test: Add MuHash3072 unit tests (Fabian Jahr)
adc708c98d crypto: Add MuHash3072 implementation (Fabian Jahr)
0b4d290bf5 crypto: Add Num3072 implementation (Fabian Jahr)
589f958662 build: Check for 128 bit integer support (Fabian Jahr)
Pull request description:
This is the first split of #18000 which implements the Muhash algorithm and uses it to calculate the UTXO set hash in `gettxoutsetinfo`.
ACKs for top commit:
laanwj:
Code review ACK 9815332d51
Tree-SHA512: 4bc090738f0e3d80b74bdd8122e24a8ce80121120fd37c7e4335a73e7ba4fcd7643f2a2d559e2eebf54b8e3a3bd5f12cfb27ba61ded135fda210a07a233eae45
819d03b932 refactor: took out unused member functions (Zero)
ed69213c2b build: enable unused member function diagnostic (Zero)
Pull request description:
This PR enables the `-Wunused-member-function` compiler diagnostic, as discussed in #19702.
> **Notice**: The `unused-member-function` diagnostic is only available on clang. Therefore, clang should be used to test this PR.
- [x] Include the `-Wunused-member-function`diagnostic in `./configure.ac`. (ed69213c2b)
- [x] Resolve the reported warnings. (819d03b932)
Currently, enabling this flag no longer reports the following warnings:
> **Note**: output from `make 2>&1 | grep "warning: unused member function" | sort | uniq -c`
```
1 index/blockfilterindex.cpp:54:5: warning: unused member function 'DBHeightKey' [-Wunused-member-function]
2 script/bitcoinconsensus.cpp:50:9: warning: unused member function 'GetType' [-Wunused-member-function]
1 test/util_tests.cpp:1975:14: warning: unused member function 'operator=' [-Wunused-member-function]
```
All tests have passed locally (from `make check` & `src/test/test_bitcoin`).
This PR closes#19702.
ACKs for top commit:
practicalswift:
ACK 819d03b932 - patch still looks correct :)
MarcoFalke:
ACK 819d03b932
pox:
Tested ACK 819d03b932 with clang after `make clean`. No unused member function warnings.
theStack:
tested ACK 819d03b932
Tree-SHA512: 5fdfbbb02b3dc618a90a874a5caa5e01e596fc1d14a209e75a6981f01b253f9bca0cfac8fdd758dd7151986609fb76571c3745124a29cfd4f8cbb8d82a07272e
When using Clang 7, we may end up trying to use the flag when it won't
work properly, which can lead to confusing errors. i.e:
```bash
/usr/bin/ld: error: ... <corrupt x86 feature size: 0x8>
```
Use CHECK_LINK_FLAG & --fatal-warnings to ensure we wont use the flag in this case.
a0a771843f contrib: Changes to checks for PowerPC64 (Luke Dashjr)
634f6ec4eb contrib: Parse ELF directly for symbol and security checks (Wladimir J. van der Laan)
Pull request description:
Instead of the ever-messier text parsing of the output of the readelf tool (which is clearly meant for human consumption not to be machine parseable), parse the ELF binaries directly.
Add a small dependency-less ELF parser specific to the checks.
This is slightly more secure, too, because it removes potential ambiguity due to misparsing and changes in the output format of `elfread`. It also allows for stricter and more specific ELF format checks in the future.
This removes the build-time dependency for `readelf`.
It passes the test-security-check for me locally, ~~though I haven't checked on all platforms~~. I've checked that this works on the cross-compile output for all ELF platforms supported by Bitcoin Core at the moment, as well as PPC64 LE and BE.
Top commit has no ACKs.
Tree-SHA512: 7f9241fec83ee512642fecf5afd90546964561efd8c8c0f99826dcf6660604a4db2b7255e1afb1e9bb0211fd06f5dbad18a6175dfc03e39761a40025118e7bfc
7587d11ec9 build: remove cdrkit package from depends (fanquake)
0df9819126 build: Replace genisoimage with xorriso (fanquake)
22437fc72e build: Run libdmg-hfsplus's DMG tool in make deploy (Carl Dong)
Pull request description:
This is a redo of fanquake's https://github.com/bitcoin/bitcoin/pull/18151, which, aside from switching us from the deprecated `genisoimage` to the maintained `xorriso`, is also necessary for Guix to achieve determinism without using faketime.
> xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
> more maintained, and has the right toggles for us to achieve output
> determinism without using blunt tools like faketime.
>
> In this commit, we use xorrisofs from the build environment rather than
> building it ourselves using depends. This is not necessary and can be
> changed in the future.
>
> From wiki.debian.org/genisoimage?action=recall&rev=11 :
>
> > The classical command line interface for production of ISO 9660
> > filesystem images is the option set established by program mkisofs.
> > For reasons of licensing and other problems with its author, Debian
> > ships a fork of mkisofs, called genisoimage, which was split off in
> > 2006 and then developed independently.
> >
> > Meanwhile, genisoimage gets no new features and not even bug fixes. It
> > is first choice only if its options -udf or -hfs are needed.
> >
> > Replacement in most uses cases, especially for bootable ISO 9660
> > filesystems, archiving, and backup, is xorrisofs which starts the -as
> > mkisofs emulation mode of program xorriso.
ACKs for top commit:
laanwj:
ACK 7587d11ec9
Tree-SHA512: 62f3aad08fa8bf21192e951d7dd33b24975586d76834cfa3498f4b8cdb586cefec8cab2c073d1951a0884b5e182fd71ef2cf3accad98f84455016776ad3c5422
xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.
In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.
From https://wiki.debian.org/genisoimage?action=recall&rev=11 :
> The classical command line interface for production of ISO 9660
> filesystem images is the option set established by program mkisofs.
> For reasons of licensing and other problems with its author, Debian
> ships a fork of mkisofs, called genisoimage, which was split off in
> 2006 and then developed independently.
>
> Meanwhile, genisoimage gets no new features and not even bug fixes. It
> is first choice only if its options -udf or -hfs are needed.
>
> Replacement in most uses cases, especially for bootable ISO 9660
> filesystems, archiving, and backup, is xorrisofs which starts the -as
> mkisofs emulation mode of program xorriso.
fa13e1b0c5 build: Add option --enable-danger-fuzz-link-all (MarcoFalke)
44444ba759 fuzz: Link all targets once (MarcoFalke)
Pull request description:
Currently the linker is invoked more than 150 times when compiling with `--enable-fuzz`. This is problematic for several reasons:
* It wastes disk space north of 20 GB, as all libraries and sanitizers are linked more than 150 times
* It wastes CPU time, as the link step can practically not be cached (similar to ccache for object files)
* It makes it a blocker to compile the fuzz tests by default for non-fuzz builds #19388, for the aforementioned reasons
* The build file is several thousand lines of code, without doing anything meaningful except listing each fuzz target in a highly verbose manner
* It makes writing new fuzz tests unnecessarily hard, as build system knowledge is required; Compare that to boost unit tests, which can be added by simply editing an existing cpp file
* It encourages fuzz tests that re-use the `buffer` or assume the `buffer` to be concatenations of seeds, which increases complexity of seeds and complexity for the fuzz engine to explore; Thus reducing the effectiveness of the affected fuzz targets
Fixes#20088
ACKs for top commit:
practicalswift:
Tested ACK fa13e1b0c5
sipa:
ACK fa13e1b0c5. Reviewed the code changes, and tested the 3 different test_runner.py modes (run once, merge, generate). I also tested building with the new --enable-danger-fuzz-link-all
Tree-SHA512: 962ab33269ebd51810924c51266ecc62edd6ddf2fcd9a8c359ed906766f58c3f73c223f8d3cc49f2c60f0053f65e8bdd86ce9c19e673f8c2b3cd676e913f2642
836a3dc02c Avoid weak-linked getauxval support on non-linux platforms (like macOS) (Jonas Schnelli)
41a413b317 Define correct symbols for getauxval (Jonas Schnelli)
Pull request description:
PR #20358 made use of the two preprocessor symbols `HAVE_STRONG_GETAUXVAL` as well as `HAVE_WEAK_GETAUXVAL`.
These symbols have not been defined in configure.ac. They where only passed selective as CRC32 CPPFLAGS in https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.crc32c.include#L16.
PR #20358 would have broken the macOS build since `getauxval` is not supported on macOS (but weak-linking does pass).
This PR defines the two symbols correctly and reduces calls to `getauxval` to linux.
ACKs for top commit:
laanwj:
Code review ACK 836a3dc02c
jonatack:
utACK 836a3dc02c
Tree-SHA512: 6527f4a617b937f4c368a3cb1c162f1ac38a6f5e6341295554961eaf322906e9b27398a6f7b00819854ceebb5c828d3e6ce0a779edd769adc4053ce8beda3739
Previously, the compression of the .iso file to a .dmg file was done
outside of `make deploy' in order to use the faketime-wrapped version of
libdmg-hfsplus's DMG tool.
Specifying the faketime-wrapped version of the DMG tool to ./configure
fixes this and simplifies build scripts.
c932e0d67e doc: Update wallet database installation guide for macOS (Hennadii Stepanov)
ee7b84e63c build: Use Homebrew's sqlite package if it is available (Hennadii Stepanov)
c96d1f65a5 build, refactor: Check that Homebrew's qt5 package is actually installed (Hennadii Stepanov)
Pull request description:
On master (7ae86b3c68) installed Homebrew `sqlite` package is ignored during build on macOS.
This PR fixes this issue and update macOS build docs.
Closes#20498.
ACKs for top commit:
willcl-ark:
> > That said, another tACK of [c932e0d](c932e0d67e)
hebasto:
> That said, another tACK of [c932e0d](c932e0d67e)
laanwj:
Code review ACK c932e0d67e
jonasschnelli:
code review re-ACK c932e0d67e
Tree-SHA512: 2563f25534d065556b17ee8c0fca957aea61b5ae288a2aa72743e77607843a45c39f209321e0f05b34283a74d2edcf961cf1dc54a35ed0cc21182304bb961505
206f74e88c Support make src/bitcoin-node and src/bitcoin-gui (João Barbosa)
Pull request description:
This change adds the following configure output variables
```
dnl Multi Process
BITCOIN_MP_NODE_NAME=bitcoin-node
BITCOIN_MP_GUI_NAME=bitcoin-gui
```
and adds support for
```sh
make src/bitcoin-node src/bitcoin-gui
```
ACKs for top commit:
laanwj:
Code review ACK 206f74e88c
Tree-SHA512: 4d1a694b9010ecc267ee955f4475127a58e6da72f30179ec740285ee6fe03cd91dcb6847317a47460dbd548edb88b7da6c7a98eac10f0dabe3ce4e83e0aa8093
ed1bbcefea contrib: add MACHO tests to symbol-check tests (fanquake)
5bab08df17 contrib: Add test for ELF symbol-check (Wladimir J. van der Laan)
Pull request description:
Check both failure cases:
- Use a glibc symbol from a version that is too new
- Use a symbol from a library that is not in the allowlist
And also check a conforming binary.
Adding a similar check for Windows PE can be done in a separate PR.
ACKs for top commit:
fanquake:
ACK ed1bbcefea
Tree-SHA512: fd437612e003922465fe1396efa1fa3a64bd1c7b0a514d2a0a7a0caaaa9fb5cb43e0ed7caec15eb0a3508692c9eb3212d7ba3c7e8180b942dd3e50616ad6e557
faa05854f8 util: Remove probably misleading TODO (MarcoFalke)
fac5efe730 util: Add Assume() identity function (MarcoFalke)
fa861569dc util: Allow Assert(...) to be used in all contexts (practicalswift)
Pull request description:
This is needed for #20138. Please refer to the added documentation for motivation.
ACKs for top commit:
practicalswift:
cr ACK faa05854f8
jnewbery:
utACK faa05854f8
hebasto:
ACK faa05854f8, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 72165fbd898b92ab9a79b070993fa1faa86c2e3545b6645e72c652bda295d5107bc298d0482bf3aaf0926fc0c3e6418a445c0e073b08568c44231f547f76a688
Check both failure cases:
- Use a glibc symbol from a version that is too new
- Use a symbol from a library that is not in the allowlist
And also check a conforming binary.
Adding a similar check for Windows PE can be done in a separate PR.
d52f502b1e Fix mock SQLiteDatabases (Andrew Chow)
99309ab3e9 Allow disabling BDB in configure with --without-bdb (Andrew Chow)
ee47f11f73 GUI: Force descriptor wallets when BDB is not compiled (Andrew Chow)
71e40b33bd RPC: Require descriptors=True for createwallet when BDB is not compiled (Andrew Chow)
6ebc41bf9c Enforce salvage is only for BDB wallets (Andrew Chow)
a58b719cf7 Do not compile BDB things when USE_BDB is defined (Andrew Chow)
b33af48210 Include wallet/bdb.h where it is actually being used (Andrew Chow)
Pull request description:
Adds a `--without-bdb` option to `configure` which disables the compilation of the BDB stuff. Legacy wallets will not be created when BDB is not compiled. A legacy-sqlite wallet can be loaded, but we will not create them.
Based on #20156 to resolve the situation where both `--without-sqlite` and `--without-bdb` are provided. In that case, the wallet is disabled and `--disable-wallet` is effectively set.
ACKs for top commit:
laanwj:
Code review ACK d52f502b1e
Tree-SHA512: 5a92ba7a542acc2e27003e9d4e5940e0d02d5c1f110db06cdcab831372bfd83e8d89c269caff31dd5bff062c1cf5f04683becff12bd23a33be731676f346553d
Instead of the ever-messier text parsing of the output of the readelf
tool (which is clearly meant for human consumption not to be machine
parseable), parse the ELF binaries directly.
Add a small dependency-less ELF parser specific to the checks.
This is slightly more secure, too, because it removes potential
ambiguity due to misparsing and changes in the output format of `elfread`. It
also allows for stricter and more specific ELF format checks in the future.
This removes the build-time dependency for `readelf`.
It passes the test-security-check for me locally, though I haven't
checked on all platforms.
8f7b930475 Drop the leading 0 from the version number (Andrew Chow)
Pull request description:
Removes the leading 0 from the version number. The minor version, which we had been using as the major version, is now the major version. The revision, which we had been using as the minor version, is now the minor version. The revision number is dropped. The build number is promoted to being part of the version number. This also avoids issues where it was accidentally not included in the version number.
The CLIENT_VERSION remains the same format as previous as previously, as the Major version was 0 so it never actually got included in it.
The user agent string formatter is updated to follow this new versioning.
***
Honestly I'm just tired of all of the people asking for "1.0" that maybe this'll shut them up. Skip the whole 1.0 thing and go straight to version 22.0!
Also, this means that the terminology we commonly use lines up with how the variables are named. So major versions are actually bumping the major version number, etc.
ACKs for top commit:
jnewbery:
Code review ACK 8f7b930475
MarcoFalke:
review ACK 8f7b930475🎻
Tree-SHA512: b5c3fae14d4c0a9c0ab3b1db7c949ecc0ac3537646306b13d98dd0efc17c489cdd16d43f0a24aaa28e9c4a92ea360500e05480a335b03f9fb308010cdd93a436
Removes the leading 0 from the version number. The minor version, which
we had been using as the major version, is now the major version. The
revision, which we had been using as the minor version, is now the minor
version. The revision number is dropped. The build number is promoted to
being part of the version number. This also avoids issues where it was
accidentally not included in the version number.
The CLIENT_VERSION remains the same format as previous as previously,
the Major version was 0 so that was never a factor in CLIENT_VERSION.
97c738ff1b [tests] Recommend f-strings for formatting, update feature_block to use them (Anthony Towns)
8ae9d314e9 Bump minimum python version to 3.6 (Anthony Towns)
Pull request description:
Python 3.5 has reached [end-of-life](https://devguide.python.org/#status-of-python-branches) as of September 2020, and 3.6 has some moderately nice [features](https://docs.python.org/3/whatsnew/3.6.html):
- `f'x = {x}'` as an alternative to `'x = {}'.format(x)` format strings (cf https://github.com/bitcoin/bitcoin/pull/13718#issuecomment-406591027)
- underscore separators for large numbers, like `1_234_567`
- improvements to async
- improvements to typing module
Note that 3.6 is not available in xenial (16.04), but is available in bionic (18.04), while focal (20.04) has 3.8. CentOS 7 and 8 have 3.6.8, Debian stable has 3.7.3, and [gentoo and arch already had 3.6 and 3.7 in 2018](https://github.com/bitcoin/bitcoin/pull/14954#issuecomment-447118707).
ACKs for top commit:
MarcoFalke:
re-ACK 97c738ff1b
Tree-SHA512: ec7fce68845edde4d61a42de12c065fd49e5217311a6fda1323206f091a0afd50f293645dffc27d420127e4e5deb864e953f1b67eff735a0dfbbedd7899a9d60
