8f7704d032 build: improve detection of eBPF support (fanquake)
Pull request description:
Just checking for the `sys/sdt.h` header isn't enough, as systems like macOS have the header, but it doesn't actually have the `DTRACE_PROBE*` probes, which leads to [compile failures](https://github.com/bitcoin/bitcoin/pull/22006#issuecomment-859559004). The contents of `sys/sdt.h` in the macOS SDK is:
```bash
#ifndef _SYS_SDT_H
#define _SYS_SDT_H
/*
* This is a wrapper header that wraps the mach visible sdt.h header so that
* the header file ends up visible where software expects it to be. We also
* do the C/C++ symbol wrapping here, since Mach headers are technically C
* interfaces.
*
* Note: The process of adding USDT probes to code is slightly different
* than documented in the "Solaris Dynamic Tracing Guide".
* The DTRACE_PROBE*() macros are not supported on Mac OS X -- instead see
* "BUILDING CODE CONTAINING USDT PROBES" in the dtrace(1) manpage
*
*/
#include <sys/cdefs.h>
__BEGIN_DECLS
#include <mach/sdt.h>
__END_DECLS
#endif /* _SYS_SDT_H */
```
The `BUILDING CODE CONTAINING USDT PROBES` section from the dtrace manpage is available [here](https://gist.github.com/fanquake/e56c9866d53b326646d04ab43a8df9e2), and outlines the more involved process of using USDT probes on macOS.
ACKs for top commit:
jb55:
utACK 8f7704d032
practicalswift:
cr ACK 8f7704d032
hebasto:
ACK 8f7704d032, tested on macOS Big Sur 11.4 (20F71) and on Linux Mint 20.1 (x86_64) with depends.
Tree-SHA512: 5f1351d0ac2e655fccb22a5454f415906404fdaa336fd89b54ef49ca50a442c44ab92d063cba3f161cb8ea0679c92ae3cd6cfbbcb19728cac21116247a017df5
2f5bdcbc31 gui: misc external signer fixes and translation hints (Sjors Provoost)
d672404466 refactor: make ExternalSigner NetworkArg() and m_chain private (Sjors Provoost)
4455145e26 refactor: reduce #ifdef ENABLE_EXTERNAL_SIGNER usage (Sjors Provoost)
5be90c907e build: enable external signer by default (Sjors Provoost)
7d9453041b refactor: clean up external_signer.h includes (Sjors Provoost)
fc0eca31b3 fuzz: fix fuzz binary linking order (Sjors Provoost)
Pull request description:
This follows the introduction of GUI support in https://github.com/bitcoin-core/gui/pull/4
I don't think we should expect GUI users to self compile. This also enables external signer support by default for RPC users.
In addition this PR reduces the number of `#ifdef ENABLE_EXTERNAL_SIGNER`, which also fixes#21919. When compiled with `--disable-external-signer` such wallets can't be created in RPC or GUI, but they can be loaded. Attempting any action that calls HWI will trigger an error.
Side-note: this PR may or may not (currently) break CI for the GUI repository, as explained here: https://github.com/bitcoin-core/gui/pull/4#issuecomment-769859001
ACKs for top commit:
achow101:
ACK 2f5bdcbc31
hebasto:
re-ACK 2f5bdcbc31
Tree-SHA512: 1b71c5a8bea2be077ee9fa33a01130c957a0cf90951d4b7b04d3d0ef826bb77e474c3963abddfef2e2c1ea99d9c72cd2302d1eb9b5fcb7ba0bd2a625f006aa05
Just checking for the `sys/sdt.h` header isn't enough, as systems like
macOS have the header, but it doesn't actually have the dtrace probes,
which leads to compile failures.
5c7ee1b2da libsecp256k1 no longer has --with-bignum= configure option (Pieter Wuille)
bdca9bcb6c Squashed 'src/secp256k1/' changes from 3967d96bf1..efad3506a8 (Pieter Wuille)
cabb566123 Disable certain false positive warnings for libsecp256k1 msvc build (Pieter Wuille)
Pull request description:
This updates our src/secp256k1 subtree to the latest upstream master. The changes include:
* The introduction of safegcd-based modular inverses, reducing ECDSA signing time by 25%-30% and ECDSA verification time by 15%-17%.
* [Original paper](https://gcd.cr.yp.to/papers.html) by Daniel J. Bernstein and Bo-Yin Yang
* [Implementation](https://github.com/bitcoin-core/secp256k1/pull/767) by Peter Dettman; [final](https://github.com/bitcoin-core/secp256k1/pull/831) version
* [Explanation](https://github.com/bitcoin-core/secp256k1/blob/master/doc/safegcd_implementation.md) of the algorithm using Python snippets
* [Analysis](https://github.com/sipa/safegcd-bounds) of the maximum number of iterations the algorithm needs
* [Formal proof in Coq](https://medium.com/blockstream/a-formal-proof-of-safegcd-bounds-695e1735a348) by Russell O'Connor, for a high-level equivalent algorithm
* Removal of libgmp as an (optional) dependency (which wasn't used in the Bitcoin Core build)
* CI changes (Travis -> Cirrus)
* Build system improvements
ACKs for top commit:
laanwj:
Tested ACK 5c7ee1b2da
Tree-SHA512: ad8ac3746264d279556a4aa7efdde3733e114fdba8856dd53218588521f04d83950366f5c1ea8fd56329b4c7fe08eedf8e206f8f26dbe3f0f81852e138655431
e9f948c727 build: Convert warnings into errors when testing for -fstack-clash-protection (Hennadii Stepanov)
Pull request description:
Apple clang version 12.0.5 (clang-1205.0.22.9) that is a part of Xcode 12.5, and is based on LLVM clang 11.1.0, fires spammy warnings:
```
clang: warning: argument unused during compilation: '-fstack-clash-protection' [-Wunused-command-line-argument]
```
From the https://github.com/apple/llvm-project:
```
$ git log --oneline | grep 'stack-clash-protection'
00065d5cbd02 Revert "-fstack-clash-protection: Return an actual error when used on unsupported OS"
4d59c8fdb955 -fstack-clash-protection: Return an actual error when used on unsupported OS
df3bfaa39071 [Driver] Change -fnostack-clash-protection to -fno-stack-clash-protection
68e07da3e5d5 [clang][PowerPC] Enable -fstack-clash-protection option for ppc64
515bfc66eace [SystemZ] Implement -fstack-clash-protection
e67cbac81211 Support -fstack-clash-protection for x86
454621160066 Revert "Support -fstack-clash-protection for x86"
0fd51a4554f5 Support -fstack-clash-protection for x86
658495e6ecd4 Revert "Support -fstack-clash-protection for x86"
e229017732bc Support -fstack-clash-protection for x86
b03c3d8c6209 Revert "Support -fstack-clash-protection for x86"
4a1a0690ad68 Support -fstack-clash-protection for x86
f6d98429fcdb Revert "Support -fstack-clash-protection for x86"
39f50da2a357 Support -fstack-clash-protection for x86
```
I suppose, that Apple clang-1205.0.22.9 ends with on of the "Revert..." commits.
This PR prevents using of the `-fstack-clash-protection` flag if it causes warnings.
---
System: macOS Big Sur 11.3 (20E232).
ACKs for top commit:
jarolrod:
re-ACK e9f948c727
Sjors:
tACK e9f948c727 on macOS 11.3.1
Tree-SHA512: 30186da67f9b0f34418014860c766c2e7f622405520f1cbbc1095d4aa4038b0a86014d76076f318a4b1b09170a96d8167c21d7f53a760e26017f486e1a7d39d4
When building with Clang, if `-fstack-clash-protection` is used with an
unsupported target, it may result in hundreds of
`-Wunused-command-line-argument` warnings at compile time. This is
currently the case when building for at least Darwin using Apple or LLVM
Clang.
Unsupported targets may also include *BSD, however that is changing; see
further discussion in https://reviews.llvm.org/D92245 and
https://reviews.freebsd.org/D27366.
Note that this option is already skipped for Windows.
a5491882a0 build: fix configuring when building depends with NO_BDB=1 (fanquake)
Pull request description:
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets), `./configure` will fail as it still tries to find bdb. i.e:
```bash
make -C depends/ NO_QT=1 NO_BDB=1 NO_UPNP=1 NO_ZMQ=1 NO_NATPMP=1 -j8
...
copying packages: native_b2 boost libevent sqlite
./autogen.sh
./configure --prefix=/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu
...
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of bdb, without opting out of wallets entirely, and still configure successfully. I think I've tested across most potential configurations. i.e:
```bash
./configure (bdb and sqlite on system)
bdb & sqlite are both are available
./configure --without-bdb (bdb and sqlite on system)
only sqlite
./configure --without-sqlite (bdb and sqlite on system)
only bdb
./configure --disable-wallet (bdb and sqlite on system)
neither bdb or sqlite
depends NO_WALLET=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
neither bdb or sqlite
depends NO_BDB=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only sqlite
depends NO_SQLITE=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only bdb
depends
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
bdb and sqlite
```
ACKs for top commit:
laanwj:
Code review ACK a5491882a0
jarolrod:
ACK a5491882a0
Tree-SHA512: baf7d2543a401db0d846095415ff449c04ecfb4a74c734dc51e79453702f9051210daeef686970f11fcffd32cdfadbc58acd54f0706aceecfb3edb0ff17310d7
7abac98d3e configure: Support -f{debug,macro}-prefix-map (Anthony Towns)
Pull request description:
When bitcoin is checked out in two directories (eg via git worktree) object files between the two will differ due to the full path being included in the debug section. `-fdebug-prefix-map` is used to replace this with "." to avoid this unnecessary difference and allow ccache to share objects between worktrees (provided the source and compile options are the same).
Also provide `-fmacro-prefix-map` if supported so that the working dir is not encoded in `__FILE__` macros.
ACKs for top commit:
practicalswift:
cr ACK 7abac98d3e: patch looks correct
fanquake:
ACK 7abac98d3e
Tree-SHA512: b6a37c1728ec3b2e552f244da0e66db113c1e7662c7ac502e12ff466f3dbfbfefae12695ca135137c50dbb1c4c5d84059116c0cd09b391a17466dc77b8726679
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets),
./configure will fail as it still tries to find bdb. i.e:
```bash
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of
bdb without opting out of wallets entirely, and still configure
successfully.
a4e970adb6 build: enable -Wdocumentation if suppressing external warnings (fanquake)
3b0078f958 doc: fixup -Wdocumentation issues (fanquake)
c6edcf1c71 build: suppress libevent warnings if supressing external warnings (fanquake)
Pull request description:
Enable `-Wdocumentation` by taking advantage of our `--enable-suppress-external-warnings` flag. Most of the CIs are using this flag now, so any regressions should be caught.
This also required modifying libevents flags when suppressing warnings, as depending on the version being built against, that could generate a large number of warnings. i.e:
```bash
In file included from httpserver.cpp:34:
In file included from ./support/events.h:12:
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:464:11: warning: parameter 'req' not found in the function declaration [-Wdocumentation]
@param req a request object
^~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:465:11: warning: parameter 'databuf' not found in the function declaration [-Wdocumentation]
@param databuf the data chunk to send as part of the reply.
^~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:467:11: warning: parameter 'call' not found in the function declaration [-Wdocumentation]
@param call back's argument.
^~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:939:4: warning: declaration is marked with '@deprecated' command but does not have a deprecation attribute [-Wdocumentation-deprecated-sync]
@deprecated This function is deprecated; you probably want to use
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:946:1: note: add a deprecation attribute to the declaration to silence this warning
char *evhttp_decode_uri(const char *uri);
^
__AVAILABILITY_INTERNAL_DEPRECATED
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:979:5: warning: declaration is marked with '@deprecated' command but does not have a deprecation attribute [-Wdocumentation-deprecated-sync]
@deprecated This function is deprecated as of Libevent 2.0.9. Use
~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:987:1: note: add a deprecation attribute to the declaration to silence this warning
int evhttp_parse_query(const char *uri, struct evkeyvalq *headers);
^
__AVAILABILITY_INTERNAL_DEPRECATED
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:1002:11: warning: parameter 'query_parse' not found in the function declaration [-Wdocumentation]
@param query_parse the query portion of the URI
^~~~~~~~~~~
/usr/local/Cellar/libevent/2.1.12/include/event2/http.h:1002:11: note: did you mean 'uri'?
@param query_parse the query portion of the URI
^~~~~~~~~~~
uri
69 warnings generated.
```
Note that a lot of these have already been fixed upstream.
ACKs for top commit:
laanwj:
Concept and code review ACK a4e970adb6
practicalswift:
cr ACK a4e970adb6: automatic compiler feedback comes sooner and is more reliable than manual reviewer feedback
jonatack:
Light ACK a4e970adb6 skimmed the changes, clang 11 build is clean with the change, verified -Wdocumentation build warnings with this change when a doc fix was reverted
Tree-SHA512: 57a1e30cffcc8bcceee72d85f58ebe29eae525861c70acb237541bd480c51ede89875c033042c0af376fdbb49fb7f588ef9282a47c6e78f9d4501c41f1b21eb6
The register keyword was deprecated in C++11, and removed in C++17. Now
that we require C++17, we shouldn't have to supress warnings for a
non-existant feature.
0eabb2abed build: Remove unused header from the build system (Hennadii Stepanov)
Pull request description:
The only `#include <miniupnpc/miniwget.h>` was removed in #16659.
ACKs for top commit:
practicalswift:
cr ACK 0eabb2abed
fanquake:
ACK 0eabb2abed
Tree-SHA512: 630da03875c851e80286561eae0f966c89624cbb17b90f70e2bec9a69146e79d088fc176e07a4906915770ac1cdb11341a7a431ea7cf6a59d2816e927486f335
246774e264 depends: fix Qt precompiled headers bug (Igor Cota)
8e7ad4146d depends: disable Qt Vulkan support on Android (Igor Cota)
ba46adaa1a CI: add Android APK build to cirrus (Igor Cota)
7563720e30 CI: add Android APK build script (Igor Cota)
ebfb10cb75 Qt: add Android packaging support (Igor Cota)
Pull request description:
This PR is the third and final piece of the basic Android support puzzle - it depends on https://github.com/bitcoin/bitcoin/pull/16110 and is related to https://github.com/bitcoin/bitcoin/pull/16883. It introduces an `android` directory under `qt` and a simple way to build an Android package of `bitcoin-qt`:
1. Build depends for Android as described in the [README](https://github.com/bitcoin/bitcoin/blob/master/depends/README.md)
2. Configure with one of the resulting prefixes
3. Run `make && make apk` in `src/qt`
The resulting APK files will be in `android/build/outputs/apk`. You can install them manually or with [adb](https://developer.android.com/studio/command-line/adb). One can also open the `android` directory in Android Studio for that integrated development and debugging experience. `BitcoinQtActivity` is your starting point.
Under the hood makefile `apk` target:
1. Renames the `bitcoin-qt` binary to `libbitcoin-qt.so` and copies it over to a folder under `android/libs` depending on which prefix and corresponding [ABI](https://developer.android.com/ndk/guides/abis.html#sa) `bitcoin-qt` was built for
2. Takes `libc++_shared.so` from the Android NDK and puts in the same place. It [must be included](https://developer.android.com/ndk/guides/cpp-support) in the APK
3. Extracts Qt for Android Java support files from the `qtbase` archive in `depends/sources` to `android/src`
There is also just a tiny bit of `ifdef`'d code to make the Qt Widgets menus usable. It's not pretty but it works and is a stepping stone towards https://github.com/bitcoin/bitcoin/pull/16883.
ACKs for top commit:
MarcoFalke:
cr ACK 246774e264
laanwj:
Code review ACK 246774e264
Tree-SHA512: ba30a746576a167545223c35a51ae60bb0838818779fc152c210f5af1413961b2a6ab6af520ff92cbc8dcd5dcb663e81ca960f021218430c1f76397ed4cead6c
Introduce an android directory under qt and allow one to package bitcoin-qt for Android by running make apk.
Add bitcoin-qt Android build instructions.
This has never worked with any of the mingw-w64 compilers we use, and
the -O0 is causing issues for builders applying spectre mitigations.
Recent discussion on https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458
also indicates that this should just not be used on Windows.
e017a913d0 bitcoind: Add -daemonwait option to wait for initialization (Wladimir J. van der Laan)
c3e6fdee6d shutdown: Use RAII TokenPipe in shutdown (Wladimir J. van der Laan)
612f746a8f util: Add RAII TokenPipe (Wladimir J. van der Laan)
Pull request description:
This adds a `-daemonwait` flag that does the same as `-daemon` except that it, from a user perspective, backgrounds the process only after initialization is complete. This is similar to the behaviour of some other software such as c-lightning.
This can be useful when the process launching bitcoind wants to guarantee that either the RPC server is running, or that initialization failed, before continuing. The exit code indicates the initialization result.
The use of the libc function `daemon()` is replaced by a custom implementation which is inspired by the [glibc implementation](https://github.com/lattera/glibc/blob/master/misc/daemon.c#L44), but which also creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is that no MACOS-specific pragmas are needed anymore to silence a deprecation warning.
TODO:
- [x] Factor out `token_read` and `token_write` to an utility, and use them in `shutdown.cpp` as well—this is exactly the same kind of communication mechanism.
- [x] RAII-ify pipe endpoints.
- [x] Improve granularity of the `configure.ac` checks. This currently still checks for the function `daemon()` which makes no sense as it's not used. It should check for individual functions such as
`fork()` and `setsid()` etc—the former being required, the second optional.
- [-] ~~Signal propagation during initialization: if say, pressing Ctrl-C during `-daemonwait` it would be good to pass this SIGINT on to the child process instead of detaching the parent process and letting the child run free.~~ This is not necessary, see https://github.com/bitcoin/bitcoin/pull/21007#issuecomment-769007341.
Future:
- Consider if it makes sense to use this in the RPC tests (there would be no more need for "is RPC ready" polling loops). I think this is out of scope for this PR.
ACKs for top commit:
jonatack:
Tested ACK e017a913d0 checked change since previous review is move-only
Tree-SHA512: 53369b8ca2247e4cf3af8cb2cfd5b3399e8e0e3296423d64be987004758162a7ddc1287b01a92d7692328edcb2da4cf05d279b1b4ef61a665b71440ab6a6dbe2
This adds a `-daemonwait` flag that does the same as `-daemon` except
it, from a user perspective, backgrounds the process only after
initialization is complete.
This can be useful when the process launching bitcoind wants to
guarantee that either the RPC server is running, or that initialization
failed, before continuing. The exit code indicates the initialization
result.
This replaces the use of the libc function `daemon()` by a custom
implementation which is inspired by the glibc implementation, but also
creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is
that no MACOS-specific pragmas are needed anymore to silence a
deprecation warning.
faa06ecc9c build: Bump minimum Qt version to 5.9.5 (Hennadii Stepanov)
Pull request description:
Close#20104.
ACKs for top commit:
laanwj:
Code review ACK faa06ecc9c
jarolrod:
ACK faa06ecc9c
fanquake:
ACK faa06ecc9c - this should be ok to do now.
Tree-SHA512: 7295472b5fd37ffb30f044e88c39d375a5a5187d3f2d44d4e73d0eb0c7fd923cf9949c2ddab6cddd8c5da7e375fff38112b6ea9779da4fecce6f024d05ba9c08
fae216a73d scripted-diff: Rename MakeFuzzingContext to MakeNoLogFileContext (MarcoFalke)
fa4fbec03e scripted-diff: Rename PROVIDE_MAIN_FUNCTION -> PROVIDE_FUZZ_MAIN_FUNCTION (MarcoFalke)
Pull request description:
Split out two renames from #21003:
* `PROVIDE_FUZZ_MAIN_FUNCTION`. *Reason*: This in only used by fuzzing, so the name should indicate that.
* `MakeNoLogFileContext`. *Reason*: Better reflects what the helper does. Also, prepares it to be used in non-fuzz tests in the future.
ACKs for top commit:
practicalswift:
cr ACK fae216a73d: scripted-diff looks correct
Tree-SHA512: e5d347746f5da72b0c86fd4f07ac2e4b3016e88e8c97a830c73bd79d0af6d0245fe7712487fc20344d6cc25958941716c1678124a123930407e3a437265b71df
9bac71350d build: make HAVE_O_CLOEXEC available outside LevelDB (bugfix) (Sebastian Falbesoner)
584fd91d2d init: only use pipe2 if availabile, check in configure (Sebastian Falbesoner)
Pull request description:
The result of the O_CLOEXEC availability check is currently only set in the Makefile and passed to LevelDB (see `LEVELDB_CPPFLAGS_INT` in `src/Makefile.leveldb.include`), but not defined to be used in our codebase. This means that code within the preprocessor conditional `#if HAVE_O_CLOEXEC` was actually never compiled. On the master branch this is currently used for pipe creation in `src/shutdown.cpp`, PR #21007 moves this part to a new module (I found the issue while testing that PR).
The fix is similar to the one in #19803, which solved the same problem for HAVE_FDATASYNC.
In the course of working on the PR it turned out that pipe2 is not available an all platforms, hence a configure check and a corresponding define HAVE_PIPE2 is introduced and used.
The PR can be tested by anyone with a system that has pipe2 and O_CLOEXEC available by putting gibberish into the HAVE_O_CLOEXEC block: on master, everything should compile fine, on PR, the compiler should abort with an error. At least that's my naive way of testing preprocessor logic, happy to hear more sophisticated ways :-)
ACKs for top commit:
laanwj:
Code review ACK 9bac71350d
Tree-SHA512: aec89faf6ba52b6f014c610ebef7b725d9e967207d58b42a4a71afc9f1268fcb673ecc85b33a2a3debba8105a304dd7edaba4208c5373fcef2ab83e48a170051
This option replaces --with-boost-process
This prepares external signer support to be disabled by default.
It adds a configure option to enable this feature and to check
if Boost::Process is present.
This also exposes ENABLE_EXTERNAL_SIGNER to the test suite via test/config.ini
c5da2749e2 build: actually stop configure if Boost isn't available (fanquake)
cad8b527ea build: explicitly install libboost-dev package (fanquake)
Pull request description:
If Boost is not found via AX_BOOST_BASE, we don't actually stop
configuring, only a warning is emitted:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version MINIMUM_REQUIRED_BOOST or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
```
Instead we usually fail when one of the other AX_BOOST_* macros fails to find a library. These macros are slowly being
removed, and in any case, it makes more sense to fail earlier if Boost is missing.
If Boost is unavailable, the failure now looks like:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version 1.58.0 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
configure: error: Boost is not available!
```
Note that we now just pass the version into AX_BOOST_BASE, which fixes it's display in the output (rather than showing `MINIMUM_REQUIRED_BOOST`).
This PR also has a commit that adds `libboost-dev` to our install instructions and CI. This package is currently installed as a side-effect of installing our other libboost-*-dev packages. However as those continue to disappear, it makes sense to install boost-dev explicitly.
ACKs for top commit:
laanwj:
Code review ACK c5da2749e2
MarcoFalke:
Concept ACK c5da2749e2
Tree-SHA512: f866062f9d7d3a2316b6c887f17c664b9cfff41fdc0cb99ca79d641240fb01a5ae0d34140e515bc465219e1b43d5ca84f7c55f48b9c5b45a80ff2795dafd072b
If Boost is not found via AX_BOOST_BASE, we don't actually stop
configuring, only a warning is emitted:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version MINIMUM_REQUIRED_BOOST or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
```
Instead we would usually fail when one of the other
AX_BOOST_* macros fails to find a library. These macros are slowly being
removed, and in any case, it makes more sense to fail earlier if Boost
is missing.
If Boost is unavailable, the failure now looks like:
```bash
checking for boostlib >= 1.58.0 (105800)... configure: We could not detect the boost libraries (version 1.58.0 or higher). If you have a staged boost library (still not installed) please specify $BOOST_ROOT in your environment and do not give a PATH to --with-boost option. If you are sure you have boost installed, then check your version number looking in <boost/version.hpp>. See http://randspringer.de/boost for more documentation.
configure: error: Boost is not available!
```
Note that we now just pass the version into AX_BOOST_BASE, which fixes
it's display in the output (rather than MINIMUM_REQUIRED_BOOST).
Performing a series of link checks for a Boost component that is
header-only doesn't make much sense, and currently means we just have
another confusing Boost macro in our tree. I'm not sure why this was
originally done this way; maybe Sjors or luke-jr can elaborate
(#15382 (929cda5470))?
The macro also has the side-effect of producing confusing error
messages. i.e in #20744, the CI is currently failing with:
```bash
checking for boostlib >= 1.58.0 (105800) lib path in "/tmp/cirrus-ci-build/depends/x86_64-pc-linux-gnu/lib"... yes
checking for boostlib >= 1.58.0 (105800)... yes
checking whether the Boost::Process library is available... yes
configure: error: Could not find a version of the Boost::Process library!
```
This isn't useful, given there is no such thing as a `Boost::Process`
library.
This PR just removes the macro entirely, but maintains a `--with-boost-process`
(defaulting to off), flag to configure. Hopefully this will also be
removed, in favour of `--enable-disable-external-signer` if/when #16546
is merged.
060a2a64d4 ci: remove boost thread installation (fanquake)
06e1d7d81d build: don't build or use Boost Thread (fanquake)
7097add83c refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef8 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d4
vasild:
ACK 060a2a64d4
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
We are already testing for this, and our test works correctly with a Darwin
target, where the macro does not. Darwin targets do not support "protected"
visibility.
32cbb06676 build: build fuzz tests by default. (Dan Benjamin)
Pull request description:
This fixes issue #19388. The changes are as follows:
- Add a new flag to configure, --enable-fuzz-binary, which allows building test/fuzz/fuzz regardless of whether we are building to do actual fuzzing
- Set -DPROVIDE_MAIN_FUNCTION whenever --enable-fuzz is no
- Add the following libraries to FUZZ_SUITE_LD_COMMON:
- LIBBITCOIN_WALLET
- SQLLITE_LIBS
- BDB_LIBS
- if necessary, some or all of:
- NATPMP_LIBS
- MINIUPNPC_LIBS
- LIBBITCOIN_ZMQ / ZMQ_LIBS
Fixes #19388
ACKs for top commit:
MarcoFalke:
review ACK 32cbb06676ðŸ“
Tree-SHA512: c91d713ffe54a3d055daaec02c4317d7e13eed6688821ddc10d894224950b18e276fbdd4acc758c7103b50f34a132b1882b68bc8b60409f97438e0759ced77e1
e9189a750b build: more robustly check for fcf-protection support (fanquake)
Pull request description:
When using Clang 7, we may end up trying to use the flag when it won't
work properly, which can lead to confusing errors. i.e:
```bash
/usr/bin/ld: error: ... <corrupt x86 feature size: 0x8>
```
Use `AX_CHECK_LINK_FLAG` & `--fatal-warnings` to ensure we wont use the flag in this case.
We do this as even when the error is emitted, compilation succeeds, and the binaries produced will run. This means we can't just check if the compiler accepts the flag, or if compilation succeeds (without or without `-Werror`, and/or passing `-Wl,--fatal-warnings`, which may not be passed through to the linker).
This was reported by someone configuring for fuzzing, on Debian 10, where Clang 7 is the default.
See here for a minimal example of the problematic behaviour:
https://gist.github.com/fanquake/9b33555fcfebef8eb8c0795a71732bc6
ACKs for top commit:
pstratem:
tested ACK e9189a750b
MarcoFalke:
not an ACK e9189a750b , I only tested configure on my system (gcc-10, clang-11):
hebasto:
ACK e9189a750b, tested with clang-7, clang-10 and gcc: the `-fcf-protection=full` is not applied for clang-7, but applied for others compilers.
Tree-SHA512: ec24b0cc5523b90139c96cbb33bb98d1e6a24d858c466aa7dfb3c474caf8c50aca53e570fdbc0ff88378406b0ac5d687542452637b1b5fa062e829291b886fc1
This fixes issue #19388. The changes are as follows:
- Add a new flag to configure, --enable-fuzz-binary, which allows building test/fuzz/fuzz regardless of whether we are building to do actual fuzzing
- Set -DPROVIDE_MAIN_FUNCTION whenever --enable-fuzz is no
- Add the following libraries to FUZZ_SUITE_LD_COMMON:
- LIBBITCOIN_WALLET
- SQLLITE_LIBS
- BDB_LIBS
- if necessary, some or all of:
- NATPMP_LIBS
- MINIUPNPC_LIBS
- LIBBITCOIN_ZMQ / ZMQ_LIBS
22eb7930a6 tracing: add tracing framework (William Casarin)
933ab8a720 build: detect sys/sdt.h for eBPF tracing (William Casarin)
Pull request description:
Instead of writing ad-hoc logging everywhere (eg: #19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin 😅 )
The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages:
# Examples
## Network Messages
```
#!/usr/bin/env bpftrace
BEGIN
{
printf("bitcoin net msgs\n");
@start = nsecs;
}
usdt:./src/bitcoind:net:push_message
{
$ip = str(arg0);
$peer_id = (int64)arg1;
$command = str(arg2);
$data_len = arg3;
$data = buf(arg3,arg4);
$t = (nsecs - @start) / 100000;
printf("%zu outbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);
@outbound[$command]++;
}
usdt:./src/bitcoind:net:process_message
{
$ip = str(arg0);
$peer_id = (int64)arg1;
$command = str(arg2);
$data_len = arg3;
$data = buf(arg3,arg4);
$t = (nsecs - @start) / 100000;
printf("%zu inbound %s %s %zu %d %r\n", $t, $command, $ip, $peer_id, $data_len, $data);
@inbound[$ip, $command]++;
}
```
$ sudo bpftrace netmsg.bt
output: https://jb55.com/s/b11312484b601fb3.txt
if you look at the bottom of the output you can see a histogram of all the messages grouped by message type and IP. nice!
## IBD Benchmarking
```
#!/usr/bin/env bpftrace
BEGIN
{
printf("IBD to 500,000 bench\n");
}
usdt:./src/bitcoind:CChainState:ConnectBlock
{
$height = (uint32)arg0;
if ($height == 1) {
printf("block 1 found, starting benchmark\n");
@start = nsecs;
}
if ($height >= 500000) {
@end = nsecs;
@duration = @end - @start;
exit();
}
}
END {
printf("duration %d ms\n", @duration / 1000000)
}
```
This one hooks into ConnectBlock and prints the IBD time to height 500,000 starting from the first call to ConnectBlock
Userspace static tracepoints give lots of flexibility without invasive logging code. It's also more flexible than ad-hoc logging code, allowing you to instrument many different aspects of the system without having to enable per-subsystem logging.
Other ideas: tracepoints for lock contention, threads, what else?
Let me know what ya'll think and if this is worth adding to bitcoin.
## TODO
- [ ] docs?
- [x] Integrate systemtap-std-dev/libsystemtap into build (provides the <sys/sdt.h> header)
- [x] ~dtrace macos support? (is this still a thing?)~ going to focus on linux for now
ACKs for top commit:
laanwj:
Tested ACK 22eb7930a6
0xB10C:
Tested ACK 22eb7930a6
Tree-SHA512: 69242242112b679c8a12a22b3bc50252c305894fb3055ae6e13d5f56221d858e58af1d698af55e23b69bdb7abedb5565ac6b45fa5144087b77a17acd04646a75
595a34dbea contrib/signet: Document miner script in README.md (Anthony Towns)
ff7dbdc08a contrib/signet: Add script for generating a signet chain (Anthony Towns)
13762bcc96 Add bitcoin-util command line utility (Anthony Towns)
95d5d5e625 rpc: allow getblocktemplate for test chains when unconnected or in IBD (Anthony Towns)
81c54dec20 rpc: update getblocktemplate with signet rule, include signet_challenge (Anthony Towns)
Pull request description:
Adds `contrib/signet/miner` for mining signet blocks.
Adds `bitcoin-util` cli utility, with the idea being it can provide bitcoin related functionality that does not rely on the ability to access a running node. Only subcommand currently is "grind" which takes a hex-encoded header and grinds its nonce until its nBits is satisfied.
Updates `getblocktemplate` to include `signet_challenge` field, and makes `getblocktemplate` require the signet rule when invoked on the signet change. Removes connectivity and IBD checks from `getblocktemplate` when applied to a test chain (regtest, testnet, signet).
ACKs for top commit:
laanwj:
code review ACK 595a34dbea
Tree-SHA512: 8d43297710fdc1edc58acd9b53e1bd1671e5724f7097b40ab73653715dc8becc70534c4496cbba9290f4dd6538a7a3d5830eb85f83391ea31a3bb5b9d3378cc3
a191e23b8e doc: Add release notes (Hennadii Stepanov)
ae749d12dd doc: Add libnatpmp stuff (Hennadii Stepanov)
e28f9be87a ci: Add libnatpmp-dev package to some builds (Hennadii Stepanov)
5a0185b6c9 gui: Add NAT-PMP network option (Hennadii Stepanov)
a39f7336a3 net: Add -natpmp command line option (Hennadii Stepanov)
28acffd9d5 net: Add NAT-PMP to port mapping loop (Hennadii Stepanov)
a8d9f275d0 net: Add libnatpmp support (Hennadii Stepanov)
58e8364dcd gui: Apply port mapping changes on dialog exit (Hennadii Stepanov)
cf151cc68c scripted-diff: Rename UPnP stuff (Hennadii Stepanov)
4e91b1e24d net: Add flags for port mapping protocols (Hennadii Stepanov)
8b50d1b5bb net: Keep trying to use UPnP when -upnp=1 (Hennadii Stepanov)
28e2961fd6 refactor: Replace magic number with named constant (Hennadii Stepanov)
02ccf69dd6 refactor: Move port mapping code to its own module (Hennadii Stepanov)
Pull request description:
Close#11902
This PR is an alternative to:
- #12288
- #15717
To compile with NAT-PMP support on Ubuntu [`libnatpmp-dev`](https://packages.ubuntu.com/source/bionic/libnatpmp) should be available.
Log excerpt:
```
2020-02-05T20:12:28Z [mapport] NAT-PMP: public address = 95.164.65.194
2020-02-05T20:12:28Z [mapport] AddLocal(95.164.65.194:18333,3)
2020-02-05T20:12:28Z [mapport] NAT-PMP: port mapping successful.
```
See: [`libnatpmp`](https://miniupnp.tuxfamily.org/libnatpmp.html)
---
Some follow-ups are out of this PR's scope:
- mention NAT-PMP library in the version message
- ~integrate NAT-PMP into the GUI~ (already [added](https://github.com/bitcoin/bitcoin/pull/18077#issuecomment-589405068))
ACKs for top commit:
laanwj:
Tested and code review ACK a191e23b8e
Tree-SHA512: 10e19267c21bf30f20ff1abfc882d526049f0e790b95e12f109dc2bed7c0aef45de03eaf967f4e667e7509be04f1873a5c508087393d947205f3aab2ad6d7cf1
9815332d51 test: Change MuHash Python implementation to match cpp version again (Fabian Jahr)
01297fb3ca fuzz: Add MuHash consistency fuzz test (Fabian Jahr)
b111410914 test: Add MuHash3072 fuzz test (Fabian Jahr)
c122527385 bench: Add Muhash benchmarks (Fabian Jahr)
7b1242229d test: Add MuHash3072 unit tests (Fabian Jahr)
adc708c98d crypto: Add MuHash3072 implementation (Fabian Jahr)
0b4d290bf5 crypto: Add Num3072 implementation (Fabian Jahr)
589f958662 build: Check for 128 bit integer support (Fabian Jahr)
Pull request description:
This is the first split of #18000 which implements the Muhash algorithm and uses it to calculate the UTXO set hash in `gettxoutsetinfo`.
ACKs for top commit:
laanwj:
Code review ACK 9815332d51
Tree-SHA512: 4bc090738f0e3d80b74bdd8122e24a8ce80121120fd37c7e4335a73e7ba4fcd7643f2a2d559e2eebf54b8e3a3bd5f12cfb27ba61ded135fda210a07a233eae45
819d03b932 refactor: took out unused member functions (Zero)
ed69213c2b build: enable unused member function diagnostic (Zero)
Pull request description:
This PR enables the `-Wunused-member-function` compiler diagnostic, as discussed in #19702.
> **Notice**: The `unused-member-function` diagnostic is only available on clang. Therefore, clang should be used to test this PR.
- [x] Include the `-Wunused-member-function`diagnostic in `./configure.ac`. (ed69213c2b)
- [x] Resolve the reported warnings. (819d03b932)
Currently, enabling this flag no longer reports the following warnings:
> **Note**: output from `make 2>&1 | grep "warning: unused member function" | sort | uniq -c`
```
1 index/blockfilterindex.cpp:54:5: warning: unused member function 'DBHeightKey' [-Wunused-member-function]
2 script/bitcoinconsensus.cpp:50:9: warning: unused member function 'GetType' [-Wunused-member-function]
1 test/util_tests.cpp:1975:14: warning: unused member function 'operator=' [-Wunused-member-function]
```
All tests have passed locally (from `make check` & `src/test/test_bitcoin`).
This PR closes#19702.
ACKs for top commit:
practicalswift:
ACK 819d03b932 - patch still looks correct :)
MarcoFalke:
ACK 819d03b932
pox:
Tested ACK 819d03b932 with clang after `make clean`. No unused member function warnings.
theStack:
tested ACK 819d03b932
Tree-SHA512: 5fdfbbb02b3dc618a90a874a5caa5e01e596fc1d14a209e75a6981f01b253f9bca0cfac8fdd758dd7151986609fb76571c3745124a29cfd4f8cbb8d82a07272e
When using Clang 7, we may end up trying to use the flag when it won't
work properly, which can lead to confusing errors. i.e:
```bash
/usr/bin/ld: error: ... <corrupt x86 feature size: 0x8>
```
Use CHECK_LINK_FLAG & --fatal-warnings to ensure we wont use the flag in this case.
a0a771843f contrib: Changes to checks for PowerPC64 (Luke Dashjr)
634f6ec4eb contrib: Parse ELF directly for symbol and security checks (Wladimir J. van der Laan)
Pull request description:
Instead of the ever-messier text parsing of the output of the readelf tool (which is clearly meant for human consumption not to be machine parseable), parse the ELF binaries directly.
Add a small dependency-less ELF parser specific to the checks.
This is slightly more secure, too, because it removes potential ambiguity due to misparsing and changes in the output format of `elfread`. It also allows for stricter and more specific ELF format checks in the future.
This removes the build-time dependency for `readelf`.
It passes the test-security-check for me locally, ~~though I haven't checked on all platforms~~. I've checked that this works on the cross-compile output for all ELF platforms supported by Bitcoin Core at the moment, as well as PPC64 LE and BE.
Top commit has no ACKs.
Tree-SHA512: 7f9241fec83ee512642fecf5afd90546964561efd8c8c0f99826dcf6660604a4db2b7255e1afb1e9bb0211fd06f5dbad18a6175dfc03e39761a40025118e7bfc
7587d11ec9 build: remove cdrkit package from depends (fanquake)
0df9819126 build: Replace genisoimage with xorriso (fanquake)
22437fc72e build: Run libdmg-hfsplus's DMG tool in make deploy (Carl Dong)
Pull request description:
This is a redo of fanquake's https://github.com/bitcoin/bitcoin/pull/18151, which, aside from switching us from the deprecated `genisoimage` to the maintained `xorriso`, is also necessary for Guix to achieve determinism without using faketime.
> xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
> more maintained, and has the right toggles for us to achieve output
> determinism without using blunt tools like faketime.
>
> In this commit, we use xorrisofs from the build environment rather than
> building it ourselves using depends. This is not necessary and can be
> changed in the future.
>
> From wiki.debian.org/genisoimage?action=recall&rev=11 :
>
> > The classical command line interface for production of ISO 9660
> > filesystem images is the option set established by program mkisofs.
> > For reasons of licensing and other problems with its author, Debian
> > ships a fork of mkisofs, called genisoimage, which was split off in
> > 2006 and then developed independently.
> >
> > Meanwhile, genisoimage gets no new features and not even bug fixes. It
> > is first choice only if its options -udf or -hfs are needed.
> >
> > Replacement in most uses cases, especially for bootable ISO 9660
> > filesystems, archiving, and backup, is xorrisofs which starts the -as
> > mkisofs emulation mode of program xorriso.
ACKs for top commit:
laanwj:
ACK 7587d11ec9
Tree-SHA512: 62f3aad08fa8bf21192e951d7dd33b24975586d76834cfa3498f4b8cdb586cefec8cab2c073d1951a0884b5e182fd71ef2cf3accad98f84455016776ad3c5422
xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.
In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.
From https://wiki.debian.org/genisoimage?action=recall&rev=11 :
> The classical command line interface for production of ISO 9660
> filesystem images is the option set established by program mkisofs.
> For reasons of licensing and other problems with its author, Debian
> ships a fork of mkisofs, called genisoimage, which was split off in
> 2006 and then developed independently.
>
> Meanwhile, genisoimage gets no new features and not even bug fixes. It
> is first choice only if its options -udf or -hfs are needed.
>
> Replacement in most uses cases, especially for bootable ISO 9660
> filesystems, archiving, and backup, is xorrisofs which starts the -as
> mkisofs emulation mode of program xorriso.
fa13e1b0c5 build: Add option --enable-danger-fuzz-link-all (MarcoFalke)
44444ba759 fuzz: Link all targets once (MarcoFalke)
Pull request description:
Currently the linker is invoked more than 150 times when compiling with `--enable-fuzz`. This is problematic for several reasons:
* It wastes disk space north of 20 GB, as all libraries and sanitizers are linked more than 150 times
* It wastes CPU time, as the link step can practically not be cached (similar to ccache for object files)
* It makes it a blocker to compile the fuzz tests by default for non-fuzz builds #19388, for the aforementioned reasons
* The build file is several thousand lines of code, without doing anything meaningful except listing each fuzz target in a highly verbose manner
* It makes writing new fuzz tests unnecessarily hard, as build system knowledge is required; Compare that to boost unit tests, which can be added by simply editing an existing cpp file
* It encourages fuzz tests that re-use the `buffer` or assume the `buffer` to be concatenations of seeds, which increases complexity of seeds and complexity for the fuzz engine to explore; Thus reducing the effectiveness of the affected fuzz targets
Fixes#20088
ACKs for top commit:
practicalswift:
Tested ACK fa13e1b0c5
sipa:
ACK fa13e1b0c5. Reviewed the code changes, and tested the 3 different test_runner.py modes (run once, merge, generate). I also tested building with the new --enable-danger-fuzz-link-all
Tree-SHA512: 962ab33269ebd51810924c51266ecc62edd6ddf2fcd9a8c359ed906766f58c3f73c223f8d3cc49f2c60f0053f65e8bdd86ce9c19e673f8c2b3cd676e913f2642
836a3dc02c Avoid weak-linked getauxval support on non-linux platforms (like macOS) (Jonas Schnelli)
41a413b317 Define correct symbols for getauxval (Jonas Schnelli)
Pull request description:
PR #20358 made use of the two preprocessor symbols `HAVE_STRONG_GETAUXVAL` as well as `HAVE_WEAK_GETAUXVAL`.
These symbols have not been defined in configure.ac. They where only passed selective as CRC32 CPPFLAGS in https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.crc32c.include#L16.
PR #20358 would have broken the macOS build since `getauxval` is not supported on macOS (but weak-linking does pass).
This PR defines the two symbols correctly and reduces calls to `getauxval` to linux.
ACKs for top commit:
laanwj:
Code review ACK 836a3dc02c
jonatack:
utACK 836a3dc02c
Tree-SHA512: 6527f4a617b937f4c368a3cb1c162f1ac38a6f5e6341295554961eaf322906e9b27398a6f7b00819854ceebb5c828d3e6ce0a779edd769adc4053ce8beda3739
Previously, the compression of the .iso file to a .dmg file was done
outside of `make deploy' in order to use the faketime-wrapped version of
libdmg-hfsplus's DMG tool.
Specifying the faketime-wrapped version of the DMG tool to ./configure
fixes this and simplifies build scripts.
c932e0d67e doc: Update wallet database installation guide for macOS (Hennadii Stepanov)
ee7b84e63c build: Use Homebrew's sqlite package if it is available (Hennadii Stepanov)
c96d1f65a5 build, refactor: Check that Homebrew's qt5 package is actually installed (Hennadii Stepanov)
Pull request description:
On master (7ae86b3c68) installed Homebrew `sqlite` package is ignored during build on macOS.
This PR fixes this issue and update macOS build docs.
Closes#20498.
ACKs for top commit:
willcl-ark:
> > That said, another tACK of [c932e0d](c932e0d67e)
hebasto:
> That said, another tACK of [c932e0d](c932e0d67e)
laanwj:
Code review ACK c932e0d67e
jonasschnelli:
code review re-ACK c932e0d67e
Tree-SHA512: 2563f25534d065556b17ee8c0fca957aea61b5ae288a2aa72743e77607843a45c39f209321e0f05b34283a74d2edcf961cf1dc54a35ed0cc21182304bb961505
206f74e88c Support make src/bitcoin-node and src/bitcoin-gui (João Barbosa)
Pull request description:
This change adds the following configure output variables
```
dnl Multi Process
BITCOIN_MP_NODE_NAME=bitcoin-node
BITCOIN_MP_GUI_NAME=bitcoin-gui
```
and adds support for
```sh
make src/bitcoin-node src/bitcoin-gui
```
ACKs for top commit:
laanwj:
Code review ACK 206f74e88c
Tree-SHA512: 4d1a694b9010ecc267ee955f4475127a58e6da72f30179ec740285ee6fe03cd91dcb6847317a47460dbd548edb88b7da6c7a98eac10f0dabe3ce4e83e0aa8093
ed1bbcefea contrib: add MACHO tests to symbol-check tests (fanquake)
5bab08df17 contrib: Add test for ELF symbol-check (Wladimir J. van der Laan)
Pull request description:
Check both failure cases:
- Use a glibc symbol from a version that is too new
- Use a symbol from a library that is not in the allowlist
And also check a conforming binary.
Adding a similar check for Windows PE can be done in a separate PR.
ACKs for top commit:
fanquake:
ACK ed1bbcefea
Tree-SHA512: fd437612e003922465fe1396efa1fa3a64bd1c7b0a514d2a0a7a0caaaa9fb5cb43e0ed7caec15eb0a3508692c9eb3212d7ba3c7e8180b942dd3e50616ad6e557
faa05854f8 util: Remove probably misleading TODO (MarcoFalke)
fac5efe730 util: Add Assume() identity function (MarcoFalke)
fa861569dc util: Allow Assert(...) to be used in all contexts (practicalswift)
Pull request description:
This is needed for #20138. Please refer to the added documentation for motivation.
ACKs for top commit:
practicalswift:
cr ACK faa05854f8
jnewbery:
utACK faa05854f8
hebasto:
ACK faa05854f8, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 72165fbd898b92ab9a79b070993fa1faa86c2e3545b6645e72c652bda295d5107bc298d0482bf3aaf0926fc0c3e6418a445c0e073b08568c44231f547f76a688
Check both failure cases:
- Use a glibc symbol from a version that is too new
- Use a symbol from a library that is not in the allowlist
And also check a conforming binary.
Adding a similar check for Windows PE can be done in a separate PR.
d52f502b1e Fix mock SQLiteDatabases (Andrew Chow)
99309ab3e9 Allow disabling BDB in configure with --without-bdb (Andrew Chow)
ee47f11f73 GUI: Force descriptor wallets when BDB is not compiled (Andrew Chow)
71e40b33bd RPC: Require descriptors=True for createwallet when BDB is not compiled (Andrew Chow)
6ebc41bf9c Enforce salvage is only for BDB wallets (Andrew Chow)
a58b719cf7 Do not compile BDB things when USE_BDB is defined (Andrew Chow)
b33af48210 Include wallet/bdb.h where it is actually being used (Andrew Chow)
Pull request description:
Adds a `--without-bdb` option to `configure` which disables the compilation of the BDB stuff. Legacy wallets will not be created when BDB is not compiled. A legacy-sqlite wallet can be loaded, but we will not create them.
Based on #20156 to resolve the situation where both `--without-sqlite` and `--without-bdb` are provided. In that case, the wallet is disabled and `--disable-wallet` is effectively set.
ACKs for top commit:
laanwj:
Code review ACK d52f502b1e
Tree-SHA512: 5a92ba7a542acc2e27003e9d4e5940e0d02d5c1f110db06cdcab831372bfd83e8d89c269caff31dd5bff062c1cf5f04683becff12bd23a33be731676f346553d
Instead of the ever-messier text parsing of the output of the readelf
tool (which is clearly meant for human consumption not to be machine
parseable), parse the ELF binaries directly.
Add a small dependency-less ELF parser specific to the checks.
This is slightly more secure, too, because it removes potential
ambiguity due to misparsing and changes in the output format of `elfread`. It
also allows for stricter and more specific ELF format checks in the future.
This removes the build-time dependency for `readelf`.
It passes the test-security-check for me locally, though I haven't
checked on all platforms.
8f7b930475 Drop the leading 0 from the version number (Andrew Chow)
Pull request description:
Removes the leading 0 from the version number. The minor version, which we had been using as the major version, is now the major version. The revision, which we had been using as the minor version, is now the minor version. The revision number is dropped. The build number is promoted to being part of the version number. This also avoids issues where it was accidentally not included in the version number.
The CLIENT_VERSION remains the same format as previous as previously, as the Major version was 0 so it never actually got included in it.
The user agent string formatter is updated to follow this new versioning.
***
Honestly I'm just tired of all of the people asking for "1.0" that maybe this'll shut them up. Skip the whole 1.0 thing and go straight to version 22.0!
Also, this means that the terminology we commonly use lines up with how the variables are named. So major versions are actually bumping the major version number, etc.
ACKs for top commit:
jnewbery:
Code review ACK 8f7b930475
MarcoFalke:
review ACK 8f7b930475🎻
Tree-SHA512: b5c3fae14d4c0a9c0ab3b1db7c949ecc0ac3537646306b13d98dd0efc17c489cdd16d43f0a24aaa28e9c4a92ea360500e05480a335b03f9fb308010cdd93a436
Removes the leading 0 from the version number. The minor version, which
we had been using as the major version, is now the major version. The
revision, which we had been using as the minor version, is now the minor
version. The revision number is dropped. The build number is promoted to
being part of the version number. This also avoids issues where it was
accidentally not included in the version number.
The CLIENT_VERSION remains the same format as previous as previously,
the Major version was 0 so that was never a factor in CLIENT_VERSION.
97c738ff1b [tests] Recommend f-strings for formatting, update feature_block to use them (Anthony Towns)
8ae9d314e9 Bump minimum python version to 3.6 (Anthony Towns)
Pull request description:
Python 3.5 has reached [end-of-life](https://devguide.python.org/#status-of-python-branches) as of September 2020, and 3.6 has some moderately nice [features](https://docs.python.org/3/whatsnew/3.6.html):
- `f'x = {x}'` as an alternative to `'x = {}'.format(x)` format strings (cf https://github.com/bitcoin/bitcoin/pull/13718#issuecomment-406591027)
- underscore separators for large numbers, like `1_234_567`
- improvements to async
- improvements to typing module
Note that 3.6 is not available in xenial (16.04), but is available in bionic (18.04), while focal (20.04) has 3.8. CentOS 7 and 8 have 3.6.8, Debian stable has 3.7.3, and [gentoo and arch already had 3.6 and 3.7 in 2018](https://github.com/bitcoin/bitcoin/pull/14954#issuecomment-447118707).
ACKs for top commit:
MarcoFalke:
re-ACK 97c738ff1b
Tree-SHA512: ec7fce68845edde4d61a42de12c065fd49e5217311a6fda1323206f091a0afd50f293645dffc27d420127e4e5deb864e953f1b67eff735a0dfbbedd7899a9d60
When bitcoin is checked out in two directories (eg via git worktree)
object files between the two will differ due to the full path being
included in the debug section. -fdebug-prefix-map is used to replace
this with "." to avoid this unnecessary difference and allow ccache to
share objects between worktrees (provided the source and compile options
are the same).
Also provide -fmacro-prefix-map if supported so that the working dir is
not encoded in __FILE__ macros.
0e2a5e448f tests: dumping and minimizing of script assets data (Pieter Wuille)
4567ba034c tests: add generic qa-asset-based script verification unit test (Pieter Wuille)
f06e6d0345 tests: functional tests for Schnorr/Taproot/Tapscript (Pieter Wuille)
3c226639eb tests: add BIP340 Schnorr signature support to test framework (Pieter Wuille)
206fb180ec --- [TAPROOT] Tests --- (Pieter Wuille)
d7ff237f29 Activate Taproot/Tapscript on regtest (BIP 341, BIP 342) (Pieter Wuille)
e9a021d7e6 Make Taproot spends standard + policy limits (Pieter Wuille)
865d2c37e2 --- [TAPROOT] Regtest activation and policy --- (Pieter Wuille)
72422ce396 Implement Tapscript script validation rules (BIP 342) (Johnson Lau)
330de894a9 Use ScriptExecutionData to pass through annex hash (Pieter Wuille)
8bbed4b7ac Implement Taproot validation (BIP 341) (Pieter Wuille)
0664f5fe1f Support for Schnorr signatures and integration in SignatureCheckers (BIP 340) (Pieter Wuille)
5de246ca81 Implement Taproot signature hashing (BIP 341) (Johnson Lau)
9eb590894f Add TaggedHash function (BIP 340) (Pieter Wuille)
450d2b2371 --- [TAPROOT] BIP340/341/342 consensus rules --- (Pieter Wuille)
5d62e3a68b refactor: keep spent outputs in PrecomputedTransactionData (Pieter Wuille)
8bd2b4e784 refactor: rename scriptPubKey in VerifyWitnessProgram to exec_script (Pieter Wuille)
107b57df9f scripted-diff: put ECDSA in name of signature functions (Pieter Wuille)
f8c099e220 --- [TAPROOT] Refactors --- (Pieter Wuille)
Pull request description:
This is an implementation of the Schnorr/taproot consensus rules proposed by BIPs [340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), [341](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), and [342](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
See the list of commits [below](https://github.com/bitcoin/bitcoin/pull/19953#issuecomment-691815830). No signing or wallet support of any kind is included, as testing is done entirely through the Python test framework.
This is a successor to https://github.com/bitcoin/bitcoin/pull/17977 (see discussion following [this comment](https://github.com/bitcoin/bitcoin/pull/17977#issuecomment-682285983)), and will have further changes squashed/rebased. The history of this PR can be found in #19997.
ACKs for top commit:
instagibbs:
reACK 0e2a5e448f
benthecarman:
reACK 0e2a5e4
kallewoof:
reACK 0e2a5e448f
jonasnick:
ACK 0e2a5e448f almost only looked at bip340/libsecp related code
jonatack:
ACK 0e2a5e448f modulo the last four commits (tests) that I plan to finish reviewing tomorrow
fjahr:
reACK 0e2a5e448f
achow101:
ACK 0e2a5e448f
Tree-SHA512: 1b00314450a2938a22bccbb4e177230cf08bd365d72055f9d526891f334b364c997e260c10bc19ca78440b6767712c9feea7faad9a1045dd51a5b96f7ca8146e
ba8950ee01 build: optionally skip external warnings (Vasil Dimov)
Pull request description:
Add an option to `./configure` to suppress compilation warnings from
external headers. The option is off by default (no change in behavior,
show warnings from external headers).
This option is useful if e.g. Boost or Qt is installed outside of
`/usr/include` (warnings from headers in `/usr/include` are already
suppressed by default) and those warnings stand in the way of compiling
Bitcoin Core with `-Werror[=...]` or they just clutter the build output
too much and make our own warnings hard to spot.
`-isystem /usr/include` bricks GCC's `#include_next`, so we use
`-idirafter` instead. This way we don't have to treat `/usr/include`
specially.
ACKs for top commit:
practicalswift:
ACK ba8950ee01: diff looks correct!
hebasto:
ACK ba8950ee01, tested on Linux Mint 20 (x86_64).
luke-jr:
utACK ba8950ee01
Tree-SHA512: 9b54fae8590be6c79f2688a5aca09e0a9067f481dabecdd49bb278c08a62ac2b0cc704c894fbd53240e77ac84da0c7a237845df0a696cfbdb0359e1c8e2e10c9
This enables the schnorrsig module in libsecp256k1, adds the relevant types
and functions to src/pubkey, as well as in higher-level `SignatureChecker`
classes. The (verification side of the) BIP340 test vectors is also added.
Add an option to `./configure` to suppress compilation warnings from
external headers. The option is off by default (no change in behavior,
show warnings from external headers).
This option is useful if e.g. Boost or Qt is installed outside of
`/usr/include` (warnings from headers in `/usr/include` are already
suppressed by default) and those warnings stand in the way of compiling
Bitcoin Core with `-Werror[=...]` or they just clutter the build output
too much and make our own warnings hard to spot.
c4b85ba704 Bugfix: Define and use HAVE_FDATASYNC correctly outside LevelDB (Luke Dashjr)
Pull request description:
Fixes a bug introduced in #19614
The LevelDB-specific fdatasync check was only using `AC_SUBST`, which works for Makefiles, but doesn't define anything for C++. Furthermore, the #define is typically 0 or 1, never undefined.
This fixes both issues by defining it and checking its value instead of whether it is merely defined.
Pulled out of #14501 by fanquake's request
ACKs for top commit:
fanquake:
ACK c4b85ba704 - thanks for catching and fixing my mistake.
laanwj:
Code review ACK c4b85ba704
Tree-SHA512: 91d5d426ba000b4f3ee7e2315635e24bbb23ceff16269ddf4f65a63d25fc9e9cf94a3b236eed2f8031cc36ddcf78aeb5916efcb244f415943a8a12f907ede8f9
b536813cef build: add -fstack-clash-protection to hardening flags (fanquake)
076183b36b build: add -fcf-protection=full to hardening options (fanquake)
Pull request description:
Beginning with Ubuntu `19.10`, it's packaged GCC now has some additional hardening options enabled by default (in addition to existing defaults like `-fstack-protector-strong` and reducing the minimum ssp buffer size). The new additions are`-fcf-protection=full` and `-fstack-clash-protection`.
> -fcf-protection=[full|branch|return|none]
> Enable code instrumentation of control-flow transfers to increase program security by checking that target addresses of control-flow transfer instructions (such as indirect function call, function return, indirect jump) are valid. This prevents diverting the flow of control to an unexpected target. This is intended to protect against such threats as Return-oriented Programming (ROP), and similarly call/jmp-oriented programming (COP/JOP).
> -fstack-clash-protection
> Generate code to prevent stack clash style attacks. When this option is enabled, the compiler will only allocate one page of stack space at a time and each page is accessed immediately after allocation. Thus, it prevents allocations from jumping over any stack guard page provided by the operating system.
If your interested you can grab `gcc-9_9.3.0-10ubuntu2.debian.tar.xz` from https://packages.ubuntu.com/focal/g++-9. The relevant changes are part of the `gcc-distro-specs` patches, along with the relevant additions to the gcc manages:
> NOTE: In Ubuntu 19.10 and later versions, -fcf-protection is enabled by default for C, C++, ObjC, ObjC++, if none of -fno-cf-protection nor -fcf-protection=* are found.
> NOTE: In Ubuntu 19.10 and later versions, -fstack-clash-protection is enabled by default for C, C++, ObjC, ObjC++, unless -fno-stack-clash-protection is found.
So, if you're C++ using GCC on Ubuntu 19.10 or later, these options will be active unless you explicitly opt out. This can be observed with a small test:
```c++
int main() { return 0; }
```
```bash
g++ --version
g++ (Ubuntu 9.3.0-10ubuntu2) 9.3.0
g++ test.cpp
objdump -dC a.out
..
0000000000001129 <main>:
1129: f3 0f 1e fa endbr64
112d: 55 push %rbp
112e: 48 89 e5 mov %rsp,%rbp
1131: b8 00 00 00 00 mov $0x0,%eax
1136: 5d pop %rbp
1137: c3 retq
1138: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
113f: 00
# recompile opting out of control flow protection
g++ test.cpp -fcf-protection=none
objdump -dC a.out
...
0000000000001129 <main>:
1129: 55 push %rbp
112a: 48 89 e5 mov %rsp,%rbp
112d: b8 00 00 00 00 mov $0x0,%eax
1132: 5d pop %rbp
1133: c3 retq
1134: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
113b: 00 00 00
113e: 66 90 xchg %ax,%ax
```
Note the insertion of an `endbr64` instruction when compiling and _not_ opting out. This instruction is part of the Intel Control-flow Enforcement Technology [spec](https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf), which the GCC control flow implementation is based on.
If we're still doing gitian builds for the `0.21.0` and `0.22.0` releases, we'd likely update the gitian image to Ubuntu Focal, which would mean that the GCC used for gitian builds would also be using these options by default. So we should decide whether we want to explicitly turn these options on as part of our hardening options (although not just for this reason), or, we should be opting-out.
GCC has supported both options since 8.0.0. Clang has supported `-fcf-protection` from 7.0.0 and will support `-fstack-clash-protection` in it's upcoming [11.0.0 release](https://clang.llvm.org/docs/ReleaseNotes.html#id6).
ACKs for top commit:
jamesob:
ACK b536813cef ([`jamesob/ackr/18921.1.fanquake.build_add_stack_clash_an`](https://github.com/jamesob/bitcoin/tree/ackr/18921.1.fanquake.build_add_stack_clash_an))
laanwj:
Code review ACK b536813cef
Tree-SHA512: abc9adf23cdf1be384f5fb9aa5bfffdda86b9ecd671064298d4cda0440828b509f070f9b19c88c7ce50ead9ff32afff9f14c5e78d75f01241568fbfa077be0b7
1ccb9f30c0 Move Win32 defines to configure.ac to ensure they are globally defined (Luke Dashjr)
Pull request description:
#9245 no longer needs this, since the main `_WIN32_WINNT` got bumped by something else.
So rather than just lose it, might as well get it merged in independently.
I'm not aware of any practical effects, but it seems safer to use the same API versions everywhere.
ACKs for top commit:
fanquake:
ACK 1ccb9f30c0 - checked that the binaries produced are the same.
Tree-SHA512: 273e9186579197be01b443b6968e26b9a8031d356fabc5b73aa967fcdb837df195b7ce0fc4e4529c85d9b86da6f2d7ff1bf56a3ff0cbbcd8cee8a9c2bf70a244
2f8a4c9a06 build: Enable some commonly enabled compiler diagnostics (practicalswift)
Pull request description:
Enable some commonly enabled compiler diagnostics as discussed in #17344.
| Compiler diagnostic | no# of emitted unique GCC warnings in `master` | no# of emitted unique Clang warnings in `master` |
| ------------- | ------------- | ------------- |
| `-Wduplicated-branches`: Warn if `if`/`else` branches have duplicated code | 0 | Not supported |
| `-Wduplicated-cond`: Warn if `if`/`else` chain has duplicated conditions | 0 | Not supported |
| `-Wlogical-op`: Warn about logical operations being used where bitwise were probably wanted | 0 | Not supported |
| `-Woverloaded-virtual`: Warn if you overload (not `override`) a virtual function | 0 | 0 |
| ~~`-Wunused-member-function`: Warn on unused member function~~ | Not supported | 2 |
| ~~`-Wunused-template`: Warn on unused template~~ | Not supported | 1 |
There is a large overlap between this list and [Jason Turner's list of recommended compiler diagnostics in the Collaborative Collection of C++ Best Practices (`cppbestpractices`) project](https://github.com/lefticus/cppbestpractices/blob/master/02-Use_the_Tools_Available.md#gcc--clang). There is also an overlap with the recommendations given in the [C++ Core Guidelines](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines) (with editors Bjarne Stroustrup and Herb Sutter).
Closes#17344.
ACKs for top commit:
jonatack:
ACK 2f8a4c9a06 no warnings for me with these locally on debian 5.7.10-1 (2020-07-26) x86_64 with gcc 10 and clang 12
fanquake:
ACK 2f8a4c9a06 - no-longer seeing any obvious issues with doing this.
hebasto:
ACK 2f8a4c9a06, no new warnings in Travis jobs.
Tree-SHA512: f669ea22b31263a555f999eff6a9d65750662e95831b188c3192a2cf0127fb7b5136deb762a6b0b7bbdfb0dc6a40caf48251a62b164fffb81dd562bdd15ec3c8
70452a070b build: set minimum required Boost to 1.58 (fanquake)
Pull request description:
Any systems which only have an older installable Boost can use depends.
1.58.0 retains compatibility with the packages [installable on Ubuntu 16.04](https://packages.ubuntu.com/xenial/libboost-dev).
The projects usage of Boost wont be going away any time soon, if ever (i.e #15382), and our usage of the test framework.
Fixes: #19506
ACKs for top commit:
practicalswift:
ACK 70452a070b -- patch looks correct
laanwj:
ACK 70452a070b
hebasto:
ACK 70452a070b, tested on Linux Mint 20 (x86_64).
Tree-SHA512: d290415e3c70a394b3d7659c0480a35b4082bdce8d48b1c64a0025f7ad6e21567b4dc85813869513ad246d27f950706930410587c11c1aa3693ae6245084765c
This flag was added to binutils/ld in the 2.30 release,
see commit c11c786f0b45617bb8807ab6a57220d5ff50e414:
> The new "-z separate-code" option will generate separate code LOAD
segment which must be in wholly disjoint pages from any other data.
It was made the default for Linux/x86 targets in the 2.31 release, see commit
f6aec96dce1ddbd8961a3aa8a2925db2021719bb:
> This patch adds --enable-separate-code to ld configure to turn on
-z separate-code by default and enables it by default for Linux/x86.
This avoids mixing code pages with data to improve cache performance
as well as security.
> To reduce x86-64 executable and shared object sizes, the maximum page
size is reduced from 2MB to 4KB when -z separate-code is turned on by
default. Note: -z max-page-size= can be used to set the maximum page
size.
> We compared SPEC CPU 2017 performance before and after this change on
Skylake server. There are no any significant performance changes.
Everything is mostly below +/-1%.
Support was also added to LLVMs lld: https://reviews.llvm.org/D64903, however
there is remains off by default.
There were concerns about an increase in binary size, however in our case, the
increase (1 page worth of bytes) would seem negligible, given we are shipping a
multi-megabyte binary, which then downloads 100's of GBs of data.
Also note that most recent versions of distros are shipping a new enough version
of binutils that this is available and/or on by default (assuming the distro has
not turned it off, I haven't checked everywhere):
CentOS 8: 2.30
Debian Buster 2.31.1
Fedora 29: 2.31.1
FreeBSD: 2.33
GNU Guix: 2.33 / 2.34
Ubuntu 18.04: 2.30
Related threads / discussion:
https://bugzilla.redhat.com/show_bug.cgi?id=1623218
While testing #19530 I noticed that we couldn't call dsymutil after LTO:
```bash
../libtool: line 10643: x86_64-apple-darwin16-dsymutil: command not found
```
This updates configure to call `AC_PATH_TOOL` so that we end up with the
full path to dsymutil, similar to `otool` and `install_name_tool`, ie:
`/bitcoin/depends/x86_64-apple-darwin16/share/../native/bin/x86_64-apple-darwin16-otool`.
Debian GCC ignores -Wformat-security, without -Wformat, which
means when we test for it, it currently fails:
```bash
checking whether C++ compiler accepts -Wformat-security... no
...
configure:15907: checking whether C++ compiler accepts -Wformat-security
configure:15926: g++ -std=c++11 -c -g -O2 -Werror -Wformat-security conftest.cpp >&5
cc1plus: error: '-Wformat-security' ignored without '-Wformat' [-Werror=format-security]
cc1plus: all warnings being treated as errors
```
Fix this by just combining the -Wformat and -Wformat-security checks
together.
92bc268e4a build: Detect missed pkg-config early (Hennadii Stepanov)
1739eb23d8 build: Drop unused use_pkgconfig variable (Hennadii Stepanov)
a661449a2e build: Drop use_pkgconfig check for libmultiprocess check (Hennadii Stepanov)
90b95e7929 build: Drop dead non-pkg-config code for libevent check (Hennadii Stepanov)
44a14afbb8 build: Drop dead non-pkg-config code for qrencode check (Hennadii Stepanov)
10cbae0c39 build: Drop dead non-pkg-config code for ZMQ check (Hennadii Stepanov)
06cfc9cadf build: Fix indentation in UNIVALUE check (Hennadii Stepanov)
6fd2118e77 build: Drop dead non-pkg-config code for UNIVALUE check (Hennadii Stepanov)
e9edbe4dbd build: Always use pkg-config (Hennadii Stepanov)
9e2e753b06 build: Always define ZMQ_STATIC for MinGW (Hennadii Stepanov)
Pull request description:
This PR:
- is based on #18297 (already merged)
- drops all of the non-pkg-config paths from the `configure` script
Ref: #17768
ACKs for top commit:
fanquake:
ACK 92bc268e4a. I re-gitian-built. There are a couple follow-ups that I'll PR shortly. Thanks for addressing my feedback above. I took too long to get back to this.
laanwj:
ACK 92bc268e4a
Tree-SHA512: 83c2d9cf03518867a1ebf7e26a8fc5b6dd8962ef983fe0d84e0c7eb74717f4c36a834da02faf0e503ffd87167005351671cf040c0d4ddae57ee152a6ff84012b
c4ffcf07af build: remove BIP70 configure option (fanquake)
Pull request description:
This was left in after #17165, so that anyone who had been compiling
with (already disabled by default) BIP70 would realise that support
had been completely removed in 0.20.0. However we should be able to
remove it for 0.21.0.
ACKs for top commit:
jnewbery:
utACK c4ffcf07af
MarcoFalke:
ACK c4ffcf07af with or without the "catch-all reject"
Tree-SHA512: a5dd4231ed97c9dd1984fb90d69a8725df2fdda0b963269b0575601c74528e5d820a4a863c428f8ede86eaae2a1606671fe1fcebdeb96b1023f7a5f899270284
9952242c03 build: improve builtin_clz* detection (fanquake)
Pull request description:
Fixes#19402.
The way we currently test for `__builtin_clz*` support with `AC_CHECK_DECLS` does not work with Clang:
```bash
configure:21492: clang++-10 -std=c++11 -c -g -O2 -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS conftest.cpp >&5
conftest.cpp💯10: error: builtin functions must be directly called
(void) __builtin_clz;
^
1 error generated.
```
This also removes the `__builtin_clz()` check, as we don't actually use it anywhere, and it's trvial to re-add detection if we do start using it at some point. If this is controversial then I'll add a test for it as well.
ACKs for top commit:
sipa:
ACK 9952242c03
laanwj:
ACK 9952242c03
Tree-SHA512: 695abb1a694a01a25aaa483b4fffa7d598842f2ba4fe8630fbed9ce5450b915c33bf34bb16ad16a16b702dd7c91ebf49fe509a2498b9e28254fe0ec5177bbac0
The way we currently test with AC_CHECK_DECLS do not work with Clang:
```bash
configure:21492: clang++-10 -std=c++11 -c -g -O2 -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS conftest.cpp >&5
conftest.cpp💯10: error: builtin functions must be directly called
(void) __builtin_clz;
^
1 error generated.
```
This also removes the __builtin_clz() check, as we don't actually use
it anywhere, and it's trvial to re-add detection if we do start using
it at some point.
On OS X, when searching Homebrew keg-only packages for BDB 4.8, if we find it,
use BDB_CPPFLAGS and BDB_LIBS instead of CFLAGS and LIBS for the result. This
is (1) more correct, and (2) necessary in order to give this location
priority over other directories in the include search path, which may include
system include directories with other versions of BDB.
This option causes the compiler to insert probes whenever stack space
is allocated statically or dynamically to reliably detect stack overflows
and thus mitigate the attack vector that relies on jumping over a stack
guard page as provided by the operating system.
This option is now enabled by default in Ubuntu GCC as of 19.10.
Available in GCC 8 and Clang 11.
8a26848c46 build: Fix m4 escaping (Hennadii Stepanov)
9123ec15db build: Remove extra tokens warning (Hennadii Stepanov)
fded4f48c3 build: Remove duplicated QT_STATICPLUGIN define (Hennadii Stepanov)
05a93d5d96 build: Fix indentation in bitcoin_qt.m4 (Hennadii Stepanov)
ddbb419310 build: Use pkg-config in BITCOIN_QT_CONFIGURE for all hosts (Hennadii Stepanov)
492971de35 build: Fix mingw pkgconfig file and dependency naming (Hennadii Stepanov)
Pull request description:
This PR makes `bitcoin_qt.m4` to use `pkg-config` for all hosts and removes non-pkg-config paths from it. This is a step towards the idea which was clear [stated](https://github.com/bitcoin/bitcoin/pull/8314#issue-76644643) by Cory Fields:
> I believe the consensus is to treat Windows like the others and require pkg-config across the board. We can drop all of the non-pkg-config paths, and simply AC_REQUIRE(PKG_PROG_PKG_CONFIG)
There are two unsolved problems with this PR. If depends is built with `DEBUG=1` the `configure` script fails to pickup Qt:
- for macOS host (similar to, but not the same as #16391)
- for Windows host (regression)
The fix is ~on its way~ submitted in #18298 (as a followup).
Also this PR picks some small improvements from #17820.
ACKs for top commit:
theuni:
Code review ACK 8a26848c46
dongcarl:
Code Review ACK 8a26848c46
laanwj:
Code review ACK 8a26848c46
Tree-SHA512: 3b25990934b939121983df7707997b31d61063b1207d909f539d69494c7cb85212f353092956d09ecffebb9fef28b869914dd1216a596d102fcb9744bb5487f7
This was left in after #17165, so that anyone who had been compiling
with (already disabled by default) BIP70 would realise that support
had been completely removed in 0.20.0. However we should be able to
remove it for 0.21.0.
Fuzzing code uses C++17 specific code (e.g. std::optional), so it is not
possible to compile with --enable-fuzz and without --enable-c++17.
Thus, turn on --enable-c++17 whenever --enable-fuzz is used.
eea8114657 build: Enable unreachable-code-loop-increment (Jonathan Schoeller)
d15db4b1fc refactor: Fix unreachable code in init arg checks (Jonathan Schoeller)
Pull request description:
Closes: #19017
In #19015 it's been suggested that we add some new compiler warnings to our build. Some of these, such as `-Wunreachable-code-loop-increment`, generate warnings. We'll likely want to fix these up if we're going to turn these warnings on.
```shell
init.cpp:969:5: warning: loop will run at most once (loop increment never executed) [-Wunreachable-code-loop-increment]
for (const auto& arg : gArgs.GetUnsuitableSectionOnlyArgs()) {
^~~
1 warning generated.
```
aa8d76806c/src/init.cpp (L968-L972)
To fix this, collect all errors, and output them in a single error message after the loop completes. This resolves the unreachable code warning, and avoids popup hell that could result from outputting a seperate message for each error or warning one by one.
ACKs for top commit:
laanwj:
Code review ACK eea8114657
hebasto:
re-ACK eea8114657, only suggested changes applied since the [previous](https://github.com/bitcoin/bitcoin/pull/19131#pullrequestreview-421772387) review.
Tree-SHA512: 2aa3ceb7fab581b6ba2580900668388d8eba1c3001c8ff9c11c1f4a9a10fbc37f30e590249862676858446e3f4950140a252953ba1643ba3bfd772f8eae20583
facef3d413 doc: Explain that anyone can work on good first issues, move text to CONTRIBUTING.md (MarcoFalke)
fae2fb2a19 doc: Expand section on Getting Started (MarcoFalke)
100000d1b2 doc: Add headings to CONTRIBUTING.md (MarcoFalke)
fab893e0ca doc: Fix unrelated typos reported by codespell (MarcoFalke)
Pull request description:
Some random doc changes:
* Add sections to docs, so that they can be linked to
* Explain that anyone (even maintainers) are allowed to work on good first issues
* Expand section on Getting Started slightly
ACKs for top commit:
hebasto:
ACK facef3d413
fanquake:
ACK facef3d413
Tree-SHA512: 8998e273a76dbf4ca77e79374c14efe4dfcc5c6df6b7d801e1e1e436711dbe6f76b436f9cbc6cacb45a56827babdd6396f3bd376a9426ee7be3bb9b8a3b8e383
While emoji and other symbols in C++ identifers (as accepted by newer
compilers) are fun, they might create confusion during code review, for
example because some symbols look very similar. Forbid such extended
identifiers for now.
This is done by providing `-fno-extended-identifiers`. Thanks to sipa
for suggesting this compiler flag.
e2bab2aa16 multiprocess: add multiprocess travis configuration (Russell Yanofsky)
603fd6a2e7 depends: add MULTIPROCESS depends option (Russell Yanofsky)
5d1377b52b build: multiprocess autotools changes (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This PR consists of build changes only. It adds an `--enable-multiprocess` autoconf option (off by default and marked experimental), that builds new `bitcoin-node` and `bitcoin-gui` binaries. These currently function the same as existing `bitcoind` and `bitcoin-qt` binaries, but are extended in #10102 with IPC features to execute node, wallet, and gui functions in separate processes.
In addition to adding the `--enable-multiprocess` config flag, it also adds a depends package and autoconf rules to build with the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library, and it adds new travis configuration to exercise the build code and run functional tests with the new binaries.
The changes in this PR were originally part of #10102 but were moved into #16367 to be able to develop and review the multiprocess build changes independently of the code changes. #16367 was briefly merged and then reverted in #18588. Only change since #16367 has been dropping the `native_boost.mk` depends package which was pointed out to be no longer necessary in https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-596484337 and https://github.com/bitcoin/bitcoin/pull/18588#pullrequestreview-391765649
ACKs for top commit:
practicalswift:
ACK e2bab2aa16
Sjors:
tACK e2bab2aa16 on macOS 10.15.4
hebasto:
ACK e2bab2aa16, tested on Linux Mint 19.3 (x86_64):
Tree-SHA512: b5a76eab5abf63d9d8b6d628cbdff4cc1888eef15cafa0a5d56369e2f9d02595fed623f4b74b2cf2830c42c05a774f0943e700f9c768a82d9d348cad199e135c
Instruct the linker to set the major & minor subsystem versions in the PE
header to 6 & 1 (NT 6.1 which corresponds to Windows 7). Similar to
macOS, the binary will now refuse to run on unsupported versions of
Windows.
a30b0a24e9 build: enable -Werror=gnu (Vasil Dimov)
Pull request description:
Stop the build if a warning is emitted due to `-Wgnu` and
`--enable-werror` has been used. As usual - this would help notice such
a warning that is about to be introduced in new code.
This is a followup to
https://github.com/bitcoin/bitcoin/pull/18088 build: ensure we aren't using GNU extensions
ACKs for top commit:
practicalswift:
ACK a30b0a24e9
Empact:
ACK a30b0a24e9
Tree-SHA512: f81b71cf3ee4db88b6f664c571075e0d30800a604f067f44273f256695a1dea533779db2ac859dd0a4cd8b66289c3e45f4aff1cfadfa160a1c354237167b05e2
df6bde031b test: remove glibc fdelt sanity check (fanquake)
8bf1540cc2 build: remove fdelt_chk backwards compatibility code (fanquake)
Pull request description:
ae30d40e50
The return type of [`fdelt_chk`](https://sourceware.org/git/?p=glibc.git;a=blob;f=debug/fdelt_chk.c;h=f62ce7349707cb68f55831c1c591fd7387a90258;hb=HEAD) changed from `unsigned long int` to `long int` in glibc 2.16. See [this commit](https://sourceware.org/git/?p=glibc.git;a=commit;h=ceb9e56b3d1f8c1922e0526c2e841373843460e2). Now that we require [glibc >=2.17](https://github.com/bitcoin/bitcoin/pull/17538) we can remove our back-compat code.
ab7bce584a
While looking at the above changes, I noticed that our glibc fdelt sanity check doesn't seem to be checking anything. `fdelt_warn()` also isn't something we'd want to actually "trigger" at runtime, as doing so would cause `bitcoind` to abort.
The comments:
> // trigger: Call FD_SET to trigger __fdelt_chk. FORTIFY_SOURCE must be defined
> // as >0 and optimizations must be set to at least -O2.
suggest calling FD_SET to check the invocation of `fdelt_chk` (this is [aliased with fdelt_warn in glibc](https://sourceware.org/git/?p=glibc.git;a=blob;f=debug/fdelt_chk.c;h=f62ce7349707cb68f55831c1c591fd7387a90258;hb=HEAD)). However just calling `FD_SET()` will not necessarily cause the compiler to insert a call to `fd_warn()`.
Whether or not GCC (recent Clang should work, but may use different heuristics) inserts a call to `fdelt_warn()` depends on if the compiler can determine if the value passed in is a compile time constant (using [`__builtin_constant_p`](https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html)) and whether the value is < 0 or >= `FD_SETSIZE`. The glibc implementation is [here](https://sourceware.org/git/?p=glibc.git;a=blob;f=misc/bits/select2.h;h=7e17430ed94dd1679af10afa3d74795f9c97c0e8;hb=HEAD). This means our check should never cause a call to be inserted.
Compiling master without `--glibc-back-compat` (if you do pass `--glibc-back-compat` the outcome is still the same; however the abort will only happen with >=`FD_SETSIZE` as that is what our [fdelt_warn()](https://github.com/bitcoin/bitcoin/blob/master/src/compat/glibc_compat.cpp#L24) checks for), there are no calls to `fdelt_warn()` inserted by the compiler:
```bash
objdump -dC bitcoind | grep sanity_fdelt
...
0000000000399d20 <sanity_test_fdelt()>:
399d20: 48 81 ec 98 00 00 00 sub $0x98,%rsp
399d27: b9 10 00 00 00 mov $0x10,%ecx
399d2c: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax
399d33: 00 00
399d35: 48 89 84 24 88 00 00 mov %rax,0x88(%rsp)
399d3c: 00
399d3d: 31 c0 xor %eax,%eax
399d3f: 48 89 e7 mov %rsp,%rdi
399d42: fc cld
399d43: f3 48 ab rep stos %rax,%es:(%rdi)
399d46: 48 8b 84 24 88 00 00 mov 0x88(%rsp),%rax
399d4d: 00
399d4e: 64 48 33 04 25 28 00 xor %fs:0x28,%rax
399d55: 00 00
399d57: 75 0d jne 399d66 <sanity_test_fdelt()+0x46>
399d59: b8 01 00 00 00 mov $0x1,%eax
399d5e: 48 81 c4 98 00 00 00 add $0x98,%rsp
399d65: c3 retq
399d66: e8 85 df c8 ff callq 27cf0 <__stack_chk_fail@plt>
399d6b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
```
If you modify the sanity test to pass `-1` or `FD_SETSIZE` to `FD_SET`, you'll see calls to `fdelt_warn` inserted, and the runtime behaviour is an abort as expected.
```diff
diff --git a/src/compat/glibc_sanity_fdelt.cpp b/src/compat/glibc_sanity_fdelt.cpp
index 87140d0c7..16974bfa0 100644
--- a/src/compat/glibc_sanity_fdelt.cpp
+++ b/src/compat/glibc_sanity_fdelt.cpp
@@ -20,7 +20,7 @@ bool sanity_test_fdelt()
{
fd_set fds;
FD_ZERO(&fds);
- FD_SET(0, &fds);
+ FD_SET(FD_SETSIZE, &fds);
return FD_ISSET(0, &fds);
}
#endif
```
```bash
0000000000399d20 <sanity_test_fdelt()>:
399d20: 48 81 ec 98 00 00 00 sub $0x98,%rsp
399d27: b9 10 00 00 00 mov $0x10,%ecx
399d2c: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax
399d33: 00 00
399d35: 48 89 84 24 88 00 00 mov %rax,0x88(%rsp)
399d3c: 00
399d3d: 31 c0 xor %eax,%eax
399d3f: 48 89 e7 mov %rsp,%rdi
399d42: fc cld
399d43: f3 48 ab rep stos %rax,%es:(%rdi)
399d46: 48 c7 c7 ff ff ff ff mov $0xffffffffffffffff,%rdi
399d4d: e8 3e ff ff ff callq 399c90 <__fdelt_warn>
399d52: 0f b6 04 24 movzbl (%rsp),%eax
399d56: 83 e0 01 and $0x1,%eax
399d59: 48 8b 94 24 88 00 00 mov 0x88(%rsp),%rdx
399d60: 00
399d61: 64 48 33 14 25 28 00 xor %fs:0x28,%rdx
399d68: 00 00
399d6a: 75 08 jne 399d74 <sanity_test_fdelt()+0x54>
399d6c: 48 81 c4 98 00 00 00 add $0x98,%rsp
399d73: c3 retq
399d74: e8 77 df c8 ff callq 27cf0 <__stack_chk_fail@plt>
399d79: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
```
```bash
src/bitcoind
*** buffer overflow detected ***: src/bitcoind terminated
Aborted
```
I think the test should should be removed and replaced (if possible) with additional checks in security-check.py. I was thinking about adding a version of [this script](https://github.com/fanquake/core-review/blob/master/fortify.py) as part of the output, but that needs more thought. I'll address this in a follow up.
ACKs for top commit:
laanwj:
ACK df6bde031b
Tree-SHA512: d8b3af4f4eb2d6c767ca6e72ece51d0ab9042e1bbdfcbbdb7ad713414df21489ba3217662b531b8bfdac0265d2ce5431abfae6e861b6187d182ff26c6e59b32d
0c63f80854 build: Suppress -Wdeprecated-copy warnings (Hennadii Stepanov)
Pull request description:
Tomorrow, on Apr 23 the Ubuntu 20.04 release is expected. It packaged with Qt 5.12 LTS that has a nasty peculiarity to cause modern compilers, including Clang 10.0 and GCC 9.3, to emit spammy `-Wdeprecated-copy` warnings (#15822, #18419).
This PR suppress such warnings _temporarily_, until the [upstream is fixed](https://codereview.qt-project.org/c/qt/qtbase/+/272258).
Here are some affected systems (with system packages):
- Ubuntu 20.04 LTS + Qt 5.12.8 LTS + { Clang 10.0 | GCC 9.3 }
- Fedora 32 + Qt 5.13.2 + Clang 10.0
Reference: [QTBUG-75210](https://bugreports.qt.io/browse/QTBUG-75210)
Also see **fanquake**'s [comment](https://github.com/bitcoin/bitcoin/pull/18738#issuecomment-622956100).
ACKs for top commit:
MarcoFalke:
ACK 0c63f80854 seems fine to disable this warning for the 0.21.0 release temporarily and then enable it for 0.22.0, when boost is removed.
fanquake:
ACK 0c63f80854 - I think it's ok to suppress these for now, given that `-Wdeprecated-copy` is enabled (via `-Wextra`) in GCC 9 and Clang 10. The Qt output is pretty noisy, and there's a few warnings from Boost as well.
Tree-SHA512: 7064a3272bc9eae00b73a16c421ac58be148f374cbef87320e8f092f52761f6e98166eff60346b70867f8a69a9698a79455dc16b42d92f8fbe7c56519571ac08
839add193b build: Enable -Wsuggest-override (Hennadii Stepanov)
de5e91c303 refactor: Add BerkeleyDatabaseVersion() function (Hennadii Stepanov)
Pull request description:
From GCC [docs](https://gcc.gnu.org/onlinedocs/gcc-9.2.0/gcc/Warning-Options.html):
> `-Wsuggest-override`
> Warn about overriding virtual functions that are not marked with the override keyword.
~This PR is based on #16722 (the first commit).~ See: https://github.com/bitcoin/bitcoin/pull/16722#issuecomment-584111086
ACKs for top commit:
fanquake:
ACK 839add193b
vasild:
ACK 839add193
practicalswift:
ACK 839add193b assuming Travis is happy: patch looks correct
Tree-SHA512: 1e8cc085da30d41536deff9b181962c1882314ab252c2ad958294087ae1e5a0dfa4886bdbe36f21cf6ae71df776a8420f349f007d4b5b49fd79ba98ce308965a
autoconf and automake changes to support multiprocess gui/node/wallet execution.
This adds a new --enable-multiprocess flag, and build configuration code to
detect libraries needed for multiprocess support. The --enable-multiprocess
flag builds new bitcoin-node and bitcoin-gui executables, which are updated in
https://github.com/bitcoin/bitcoin/pull/10102 to communicate across processes.
But for now they are functionally equivalent to existing bitcoind and
bitcoin-qt executables.
89fea68ffd build: don't pass -w when building for Windows (fanquake)
Pull request description:
This has been around since the introduction of autotools. However at
this point I'm not sure we'd ever want to suppress all warnings when
performing a build, and given that CXX FLAGS will have been overriden
when cross-compiling for Windows (using depends), this would rarely,
if-ever be used anyways.
From https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html:
> -w
>
> Inhibit all warning messages.
ACKs for top commit:
hebasto:
ACK 89fea68ffd
Tree-SHA512: 2b5bdef7fff5c87b28199f5822cab3cdf600c90c01a40db5cd85053eef5dcb5816e2e97ff61a30ff94b4f0c6cb7be22beaef34d82235bdf05ff9da865d40b381
This has been around since the introduction of autotools. However at
this point I'm not sure we'd every want to suppress all warnings when
performing a build, and given that CXX FLAGS will have been overriden
when cross-compiling for Windows (using depends), this would rarely,
if-ever be used anyways.
From https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html:
-w
Inhibit all warning messages.
As is, this sanity check doesn't seem to be testing fdelt_chk, because
passing a value of "0" to FD_SET wont cause the compiler to insert any
calls to fdelt_chk().
The documentation is a little misleading. If we actually triggered fdelt_chk
at runtime, bitcoind would abort. I think this check would be better replaced
(if possible) by additional checks in security-check.py.
The compiler may insert a call to fdelt_warn() (aliased with fdelt_chk
in glibc) at compile time if it can determine that an invalid value is
being passed to FD_SET.
These checks are essentially; value < 0 or value >= FD_SETSIZE along
with a check for wether the value is a compile time constant.
If the compiler can determine an invalid value is being passed, a call
to fdelt_warn will be inserted. Passing 0 should never cause a call to
be inserted.
You can check this after compiling:
```bash
objdump -dC bitcoind | grep sanity_fdelt
...
0000000000399d20 <sanity_test_fdelt()>:
399d20: 48 81 ec 98 00 00 00 sub $0x98,%rsp
399d27: b9 10 00 00 00 mov $0x10,%ecx
399d2c: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax
399d33: 00 00
399d35: 48 89 84 24 88 00 00 mov %rax,0x88(%rsp)
399d3c: 00
399d3d: 31 c0 xor %eax,%eax
399d3f: 48 89 e7 mov %rsp,%rdi
399d42: fc cld
399d43: f3 48 ab rep stos %rax,%es:(%rdi)
399d46: 48 8b 84 24 88 00 00 mov 0x88(%rsp),%rax
399d4d: 00
399d4e: 64 48 33 04 25 28 00 xor %fs:0x28,%rax
399d55: 00 00
399d57: 75 0d jne 399d66 <sanity_test_fdelt()+0x46>
399d59: b8 01 00 00 00 mov $0x1,%eax
399d5e: 48 81 c4 98 00 00 00 add $0x98,%rsp
399d65: c3 retq
399d66: e8 85 df c8 ff callq 27cf0 <__stack_chk_fail@plt>
399d6b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
```
To test, you could modify this test to pass -1 to FD_SET, and check
that a call to fdelt_warn() is inserted, and that running bitcoind
fails. i.e:
```bash
0000000000399d20 <sanity_test_fdelt()>:
399d20: 48 81 ec 98 00 00 00 sub $0x98,%rsp
399d27: b9 10 00 00 00 mov $0x10,%ecx
399d2c: 64 48 8b 04 25 28 00 mov %fs:0x28,%rax
399d33: 00 00
399d35: 48 89 84 24 88 00 00 mov %rax,0x88(%rsp)
399d3c: 00
399d3d: 31 c0 xor %eax,%eax
399d3f: 48 89 e7 mov %rsp,%rdi
399d42: fc cld
399d43: f3 48 ab rep stos %rax,%es:(%rdi)
399d46: 48 c7 c7 ff ff ff ff mov $0xffffffffffffffff,%rdi
399d4d: e8 3e ff ff ff callq 399c90 <__fdelt_warn>
399d52: 0f b6 04 24 movzbl (%rsp),%eax
399d56: 83 e0 01 and $0x1,%eax
399d59: 48 8b 94 24 88 00 00 mov 0x88(%rsp),%rdx
399d60: 00
399d61: 64 48 33 14 25 28 00 xor %fs:0x28,%rdx
399d68: 00 00
399d6a: 75 08 jne 399d74 <sanity_test_fdelt()+0x54>
399d6c: 48 81 c4 98 00 00 00 add $0x98,%rsp
399d73: c3 retq
399d74: e8 77 df c8 ff callq 27cf0 <__stack_chk_fail@plt>
399d79: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
```
```bash
./src/bitcoind
*** buffer overflow detected ***: src/bitcoind terminated
Aborted
```
a029805f57 build: remove -Qunused-arguments workaround for clang + ccache (fanquake)
Pull request description:
This was added in 386efb7695 to address spammy Clang warnings when building with ccache.
The issue was addressed in [ccache 3.2](https://bugzilla.samba.org/show_bug.cgi?id=8118), and from a look at most major distros, it's only Debian Jessie that has a version of ccache older than that ([3.1](https://packages.debian.org/jessie/ccache)).
Therefore I think it's acceptable to drop this workaround, and re-enable warnings for unused driver arguments (when compiling using Clang and ccache).
ACKs for top commit:
hebasto:
ACK a029805f57.
vasild:
utACK a029805f57
Tree-SHA512: f887b9bd12f9c1c8d209943b86e8dafe33cfd1572912f2cafabe08ffe403973e48f0f7289280a8c6db9263c57aad43fbd4bb72f42db762eb090f3b1ef0538f43
03da4c7781 build: make linker checks more robust (Cory Fields)
Pull request description:
Check for a flag to turn linker warnings into errors. When flags are passed to
linkers via the compiler driver using a -Wl,-foo flag, linker warnings may be
swallowed rather than bubbling up.
This is one of [Corys commits](b9acd3d33e) that I've modified to also add `-Wl,-fatal_warnings`
for darwin.
ACKs for top commit:
vasild:
re-ACK 03da4c778
Tree-SHA512: 212031d619ed88e52aaae30cf3b711681d72c4d670884406403605d1d86c784c84cb07e2e0d6c30926e659db8f14f8dabd5af3de5291637f8080d6dfee358248
71f183a49b build: warn on potentially uninitialized reads (Vasil Dimov)
Pull request description:
* Enable `conditional-uninitialized` warning class to show potentially uninitialized
reads.
* Fix the sole such warning in Bitcoin Core in `GetRdRand()`: `r1` would be
set to `0` on `rdrand` failure, so initializing it to `0` is a non-functional
change.
ACKs for top commit:
practicalswift:
ACK 71f183a49b
laanwj:
ACK 71f183a49b
Tree-SHA512: 2c1d8caacd86424b16a9d92e5df19e0bedb51ae111eecad7e3bfa46447bc88e5fff1f32dacf6c4a28257ebb3d87e79f80f074ce2c523ce08b1a0c0a67ab44204
Check for a flag to turn linker warnings into errors. When flags are passed to
linkers via the compiler driver using a -Wl,-foo flag, linker warnings may be
swallowed rather than bubbling up.
Co-authored-by: fanquake <fanquake@gmail.com>
Stop the build if a warning is emitted due to `-Wgnu` and
`--enable-werror` has been used. As usual - this would help notice such
a warning that is about to be introduced in new code.
This is a followup to
https://github.com/bitcoin/bitcoin/pull/18088
build: ensure we aren't using GNU extensions
Enable -Wconditional-uninitialized to warn on potentially uninitialized
reads.
Fix the sole such warning in Bitcoin Core in GetRdRand(): r1 would be
set to 0 on rdrand failure, so initializing it to 0 is a non-functional
change.
From "Intel 64 and IA-32 ArchitecturesSoftware Developer's Manual" [1],
page 1711: "CF=1 indicates that the data in the destination is valid.
Otherwise CF=0 and the data in the destination operand will be returned
as zeros for the specified width."
[1] https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf
c31cbe7cfe Add C++17 test to Travis (Pieter Wuille)
7829685e27 Add configure option for c++17 (Pieter Wuille)
0fbde488b2 Support conversion between Spans of compatible types (Pieter Wuille)
7cbfebbf3d Update ax_cxx_compile_stdcxx.m4 (Pieter Wuille)
Pull request description:
This adds a `--enable-c++17` option to the configure script, fixes the only C++17 incompatibility (with a commit taken from #18468), and adds a Travis test for it.
This is all off by default, and release builds remain C++11.
It implements the first step of the plan in https://github.com/bitcoin/bitcoin/issues/16684.
ACKs for top commit:
elichai:
tACK c31cbe7cfe
practicalswift:
Tested ACK c31cbe7cfe
hebasto:
ACK c31cbe7cfe, tested on Linux Mint 19.3 both C++11 and C++17 modes. Compiled and passed tests locally.
Tree-SHA512: a4b00776dbceef9c12abbb404c6bcd48f7916ce24c8c7a14116355f64e817578b7fcddbedd5ce435322319d1e4de43429b68553f4d96d970c308fe3e3e59b9d1
182dbdf0f4 util: Detect posix_fallocate() instead of assuming (Vasil Dimov)
Pull request description:
Don't assume that `posix_fallocate()` is available on Linux and not
available on other operating systems. At least FreeBSD has it and we
are not using it.
Properly check whether `posix_fallocate()` is present and use it if it
is.
ACKs for top commit:
laanwj:
ACK 182dbdf0f4
Tree-SHA512: f9ed4bd661f33ff6b2b1150591e860b3c1f44e12b87c35e870d06a7013c4e841ed2bf17b41ad6b18fe471b0b23a4b5e42cf1400637180888e0bc56c254fe0766
b155fcda51 doc: fix typo in configure.ac (fanquake)
20a30922fb doc: note why we can't use thread_local with glibc back compat (fanquake)
Pull request description:
Given that we went through a [gitian build](https://github.com/bitcoin/bitcoin/pull/18681) to remember why this is the case, we might as well make a note of it in configure.ac.
[From #18681](https://github.com/bitcoin/bitcoin/pull/18681#issuecomment-615526634):
Looking at the Linux build log, this has failed with:
```bash
Checking glibc back compat...
bitcoind: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
bitcoind: failed IMPORTED_SYMBOLS
bitcoin-cli: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
bitcoin-cli: failed IMPORTED_SYMBOLS
bitcoin-tx: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
bitcoin-tx: failed IMPORTED_SYMBOLS
bitcoin-wallet: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
bitcoin-wallet: failed IMPORTED_SYMBOLS
test/test_bitcoin: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
test/test_bitcoin: failed IMPORTED_SYMBOLS
bench/bench_bitcoin: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
bench/bench_bitcoin: failed IMPORTED_SYMBOLS
qt/bitcoin-qt: symbol __cxa_thread_atexit_impl from unsupported version GLIBC_2.18
```
`__cxa_thread_atexit_impl` is used for [thread_local variable destruction](https://sourceware.org/glibc/wiki/Destructor%20support%20for%20thread_local%20variables):
> To implement this support, glibc defines __cxa_thread_atexit_impl exclusively for use by libstdc++ (which has the __cxa_thread_atexit to wrap around it), that registers destructors for thread_local variables in a list. Upon thread or process exit, the destructors are called in reverse order in which they were added.
As suggested, this only became available in glibc 2.18. From the [2.18 release notes](https://sourceware.org/legacy-ml/libc-alpha/2013-08/msg00160.html):
> * Add support for calling C++11 thread_local object destructors on thread
and program exit. This needs compiler support for offloading C++11
destructor calls to glibc.
ACKs for top commit:
hebasto:
ACK b155fcda51
Tree-SHA512: 5b9567e4a70598a4b0b91956f44ae0d93091db17c84cbf9817dac6cfa992c97d3438a8b1bb66644c74891f2149e44984daed445d22de93ca8858c5b0eabefb40
Don't assume that `posix_fallocate()` is available on Linux and not
available on other operating systems. At least FreeBSD has it and we
are not using it.
Properly check whether `posix_fallocate()` is present and use it if it
is.
b919efadff depends: Use default macos clang compiler (Russell Yanofsky)
d54f64c6c7 Add multiprocess travis configuration (Russell Yanofsky)
787f40668d Set LD_LIBRARY_PATH consistently in travis tests (Russell Yanofsky)
d630646662 libmultiprocess depends build (Russell Yanofsky)
e6e44eedd5 Multiprocess build changes (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
This splits autotools, depends build, and travis changes out of #10102, so code changes and build system changes can be reviewed separately.
ACKs for top commit:
hebasto:
re-ACK b919efadff, rebased only since my [previous](https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-605514556) review.
Tree-SHA512: ebc5e403cc99a0d9629ed7fe1595e01d57e6d1255cbf03968a3196ff6f528f734c78060fdc065724ee1f923bcc5aa2b29470fcb36a7f15957eb57c76d58178a4
01a3392b1b Drop bitcoin-wallet dependency on libevent (Russell Yanofsky)
0660119ac3 Drop unintended bitcoin-tx dependency on libevent (Russell Yanofsky)
Pull request description:
This fixes compile errors trying to build bitcoin-tx and bitcoin-wallet without libevent, which were reported by Luke Dashjr in https://github.com/bitcoin/bitcoin/issues/18465
The fix avoiding `bitcoin-tx` dependency on libevent just adds a conditional build rule. This is implemented in the first commit (more details in commit description).
The fix avoiding `bitcoin-wallet` dependency on libevent requires minor code changes, because `bitcoin-wallet` (unlike `bitcoin-tx`) links against code that calls `urlDecode` / `evhttp_uridecode`. This fix is implemented in the second commit (again details in the commit description).
ACKs for top commit:
jonasschnelli:
utACK 01a3392b1b.
Tree-SHA512: d2245e912ab494cccceeb427a1eca8e55b01a0006ff93eebcfb5461ae7cecd1083ac2de443d9db036b18bdc6f0fb615546caaa20c585046f66d234937f74870a
Now that 0.20 branch has been split off, master is 0.20.99 (pre-0.21).
Also clean out release notes.
Tree-SHA512: bba6133ae9708f75206c8934901b9f9909a233330f4dfefb3c24175bf8e11631cdc89a5d24a22421a73083f7eb743e977db8020b87dfbd3c1e6043929a19a285
This was added in 386efb7695 to
address spammy Clang warnings when building with ccache.
The issue was addressed in ccache 3.2
(https://bugzilla.samba.org/show_bug.cgi?id=8118, Nov 2014),
and from a look at all major distros, it's only Debian Jessie
that has a version of ccache older than that (3.1).
Therefore I think it's acceptable to drop this workaround, and
re-enable warnings for unused driver arguments (when compiling
using Clang).
autotools and automake changes to support multiprocess execution.
This adds a new --enable-multiprocess flag, and build configuration code to
detect libraries needed for multiprocess support. The --enable-multiprocess
flag builds new bitcoin-node and bitcoin-gui executables, which are updated in
https://github.com/bitcoin/bitcoin/pull/10102 to communicate across processes.
But for now they are functionally equivalent to existing bitcoind and
bitcoin-qt executables.
Don't include util/url.cpp to libbitcoin_util.a when libevent isn't available.
This fixes a compile error trying to build bitcoin-tx without libevent reported
by Luke Dashjr in https://github.com/bitcoin/bitcoin/issues/18465Fixes#18465
faf7d4fa86 build: Add cov_fuzz target (MarcoFalke)
fac71e364e build: link fuzz/test_runner.py for out-of-tree builds (MarcoFalke)
faf2c5aca0 build: Remove unused USE_COVERAGE (MarcoFalke)
Pull request description:
Only libFuzzer is supported right now, so clang is required. Thus, this needs a workaround such as https://github.com/bitcoin/bitcoin/issues/12602#issuecomment-562788247
Can be tested with:
```
mkdir build && cd build
../configure --enable-fuzz --with-sanitizers=fuzzer --enable-lcov --enable-lcov-branch-coverage CC=clang CXX=clang++
make $MAKEJOBS
make cov_fuzz
ACKs for top commit:
practicalswift:
ACK faf7d4fa86
Tree-SHA512: 6828f8f81d95f6781713d0b09d7eba2ffdb50217e09ca839db61791a4ed70024859c7a0cb01d9eede79166d574dd57ece01f9d9fe2610d4a72a4ca4a4ce0b838
sysctl() on *BSD takes a "const int *name", whereas sysctl() on macOS
it takes an "int *name". So our configure check and sysctl() detection on
macOS currently fails:
```bash
/usr/include/sys/sysctl.h:759:9: note: candidate function not viable:
no known conversion from 'const int [2]' to 'int *' for 1st argument
int sysctl(int *, u_int, void *, size_t *, void *, size_t);
```
This change removes the name argument from the sysctl() detection check,
meaning we will detect correctly on macOS and *BSD.
For consistency we also switch to using the more generic, non-const
version of the name parameter in the rest of our usage.
This change adds to the BITCOIN_QT_CONFIGURE script ability to use
pkg-config for MinGW. All of the non-pkg-config paths are removed as
needless.
If depends is built with DEBUG=1 the configure script fails to pickup
Qt:
- for macOS host (similar, but not the same as issue 16391)
- for Windows host (regression)
fae86c38bc util: Remove unused MilliSleep (MarcoFalke)
fa9af06d91 scripted-diff: Replace MilliSleep with UninterruptibleSleep (MarcoFalke)
fa4620be78 util: Add UnintrruptibleSleep (MarcoFalke)
Pull request description:
We don't use the interruptible feature of boost's sleep anywhere, so replace it with the sleep in `std::thread`
ACKs for top commit:
ajtowns:
ACK fae86c38bc quick code review
practicalswift:
ACK fae86c38bc -- patch looks correct
sipa:
Concept and code review ACK fae86c38bc
fanquake:
ACK fae86c38bc - note that an instance of `DHAVE_WORKING_BOOST_SLEEP_FOR` was missed in the [linter](https://github.com/bitcoin/bitcoin/blob/master/test/lint/extended-lint-cppcheck.sh#L69), but that can be cleaned up later.
Tree-SHA512: 7c0f8eb197664b9f7d9fe6c472c77d384f11c797c913afc31de4b532e3b4fd9ea6dd174f92062ff9d1ec39b25e0900ca7c597435add87f0f2477d9557204848c
d76894987d logging: enable thread_local usage on macOS (fanquake)
Pull request description:
Now that we're building against a newer SDK (`10.14`), we should be able to enable `thread_local` usage on macOS. Have tested building and running locally, as well as cross-compiling and running the binaries on a macOS 10.14 system.
#### master 8a56f79d49
```bash
src/bitcoind -logthreadnames=1
2020-02-06T04:38:33Z [] Bitcoin Core version v0.19.99.0-8a56f79d4 (release build)
2020-02-06T04:38:33Z [] Assuming ancestors of block 00000000000000000005f8920febd3925f8272a6a71237563d78c2edfdd09ddf have valid signatures.
2020-02-06T04:38:33Z [] Setting nMinimumChainWork=000000000000000000000000000000000000000008ea3cf107ae0dec57f03fe8
2020-02-06T04:38:33Z [] Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2020-02-06T04:38:33Z [] Using RdSeed as additional entropy source
```
#### this PR d76894987d
```bash
checking for thread_local support... yes
...
src/bitcoind -logthreadnames=1
2020-02-06T04:17:49Z [net] net thread start
2020-02-06T04:17:49Z [opencon] opencon thread start
2020-02-06T04:17:49Z [dnsseed] dnsseed thread start
2020-02-06T04:17:49Z [init] init message: Done loading
2020-02-06T04:17:49Z [msghand] msghand thread start
2020-02-06T04:17:49Z [addcon] addcon thread start
...
2020-02-06T04:17:54Z [init] tor: Thread interrupt
2020-02-06T04:17:54Z [init] Shutdown: In progress...
```
From the [Xcode 8 release notes](https://developer.apple.com/library/archive/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html#//apple_ref/doc/uid/TP40001051-CH1-SW78)
> C++ now supports the thread_local keyword, which declares thread-local storage (TLS) and supports C++ classes with non-trivial constructors and destructors. (9001553)
ACKs for top commit:
jonasschnelli:
Tested ACK d76894987d
nijynot:
ACK d768949
hebasto:
ACK d76894987d
Tree-SHA512: 48f3e4104b80bd7b6aedcef10bb1957b073530130f33af7c5cb59e876ac3f5480e53d7af1c0b226d809fe9eef1add3d6c3fb6de4af174966202c6030060ea823
677fb8e923 test: Add ubsan surpression for crc32c (Wladimir J. van der Laan)
8e68bb1dde build: Disable msvc warning 4722 for leveldb build (Aaron Clauson)
be23949765 build: MSVC changes for leveldb update (Aaron Clauson)
9ebdf04757 build: CRC32C build system integration (Wladimir J. van der Laan)
402252a808 build: Add LCOV exception for crc32c (Wladimir J. van der Laan)
3a037d0067 test: Add crc32c exception to various linters and generation scripts (Wladimir J. van der Laan)
84ff1b2076 test: Add crc32c to subtree check linter (Wladimir J. van der Laan)
7cf13a5134 doc: Add crc32c subtree to developer notes (Wladimir J. van der Laan)
24d02a9ac0 build: Update build system for new leveldb (Wladimir J. van der Laan)
2e1819311a Squashed 'src/crc32c/' content from commit 224988680f7673cd7c769963d4035cb315aa3388 (Wladimir J. van der Laan)
66480821b3 Squashed 'src/leveldb/' changes from f545dfabff4c2e9836efed094dba99a34fbc6b88..f8ae182c1e5176d12e816fb2217ae33a5472fdd7 (Wladimir J. van der Laan)
Pull request description:
This updates leveldb to currently newest upstream commit 0c40829872:
- CRC32C hardware acceleration is now an external library [crc32c](https://github.com/google/crc32c). This adds acceleration on ARM, and should be faster on x86 because of using prefetch. It also makes it easy to support similar instruction sets on other platforms in the future.
- Thread handling uses C++11, instead of platform specific code.
- Native windows environment was added. No need to maintain our own hacky one, anymore.
- Upstream now builds using CMake. This doesn't mean we need to use that (phew), but internal configuration changed to a a series of checks, instead of OS profiles. This means the blanket error "Cannot build leveldb for $host. Please file a bug report' is removed.
All changes: a53934a3ae...0c40829872
Pretty much all our changes have been subsumed by upstream, so we figured it was cleaner to start over with a new branch from upstream with the still-relevant patches applied: https://github.com/bitcoin-core/leveldb/tree/bitcoin-fork-new
There's quite some testing to be done (see below). See https://github.com/bitcoin-core/leveldb/issues/25 and https://github.com/bitcoin-core/leveldb/pull/26 for more history and context.
TODO:
- [x] Subtree `crc32c`
- [x] Make linters happy about crc32 subtree
- [x] Integrate `crc32c` library into build system
- [x] MSVC build system
ACKs for top commit:
sipa:
ACK 677fb8e923
Tree-SHA512: 37ee92a750e053e924bc4626b12bb3fd81faa9f8c5ebaa343931fee810c45ba05aa6051fdea82535fa351bf2be7297801b98af9469865fc5ead771650a5d6240
This flag was used when building 32-bit Windows executables, which we no-longer
do, and is not accepted by the linker for any of the hosts we currently build
for. i.e:
```bash
checking whether the linker accepts -Wl,--large-address-aware... no
```
--large-address-aware
If given, the appropriate bit in the "Characteristics" field of the COFF
header is set to indicate that this executable supports virtual addresses
greater than 2 gigabytes. This should be used in conjunction with the /3GB
or /USERVA=value megabytes switch in the "[operating systems]" section of
the BOOT .INI. Otherwise, this bit has no effect. [This option is specific
to PE targeted ports of the linker]
You can check that the appropriate bit in the COFF header of our current
Windows binaries is still be set using dumpbin. i.e:
```powershell
dumpbin /headers .\bitcoind.exe
FILE HEADER VALUES
<snip>
26 characteristics
Executable
Line numbers stripped
Application can handle large (>2GB) addresses
```
f7453dcc03 build: remove linking librt for backwards compatibility (fanquake)
Pull request description:
Now that we require glibc 2.17+, see #17538, we can remove linking librt
for backwards compatibility purposes. The `clock_*` functions from librt
were merged into glibc as part of the [2.17 release](https://sourceware.org/ml/libc-announce/2012/msg00001.html):
* The `clock_*` suite of functions (declared in <time.h>) is now available
directly in the main C library. Previously it was necessary to link with
-lrt to use these functions. This change has the effect that a
single-threaded program that uses a function such as `clock_gettime' (and
is not linked with -lrt) will no longer implicitly load the pthreads
library at runtime and so will not suffer the overheads associated with
multi-thread support in other code such as the C++ runtime library.
Note that `librt` is already unused by the RISC-V and AARCH64 binaries as their librts don't export any `clock_*` functions. As an example, you can find a diff of the arm32 vs arm64 librt symbols [here](https://gist.github.com/fanquake/b08cb1f0d14df3133395d7796ebf030c).
Below is the library usage for the `v0.19.0.1` release (can delete these tables pre-merge).
#### RISC-V
```bash
riscv/bin/bitcoin-cli: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-riscv64-lp64d.so.1']
riscv/bin/bitcoin-qt: ['libpthread.so.0', 'libfontconfig.so.1', 'libfreetype.so.6', 'libxcb.so.1', 'libdl.so.2', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-riscv64-lp64d.so.1', 'libatomic.so.1']
riscv/bin/bitcoin-wallet: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-riscv64-lp64d.so.1', 'libatomic.so.1']
riscv/bin/bitcoind: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-riscv64-lp64d.so.1', 'libatomic.so.1']
riscv/bin/bitcoin-tx: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-riscv64-lp64d.so.1']
riscv/bin/test_bitcoin: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-riscv64-lp64d.so.1', 'libatomic.so.1']
```
#### AARCH64
```bash
aarch64/bin/bitcoin-cli: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-aarch64.so.1']
aarch64/bin/bitcoin-qt: ['libpthread.so.0', 'libfontconfig.so.1', 'libfreetype.so.6', 'libxcb.so.1', 'libdl.so.2', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-aarch64.so.1']
aarch64/bin/bitcoin-wallet: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-aarch64.so.1']
aarch64/bin/bitcoind: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-aarch64.so.1']
aarch64/bin/bitcoin-tx: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-aarch64.so.1']
aarch64/bin/test_bitcoin: ['libpthread.so.0', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-aarch64.so.1']
```
#### ARM LINUX GNUEABIHF
```bash
arm32/bin/bitcoin-cli: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-armhf.so.3']
arm32/bin/bitcoin-qt: ['libpthread.so.0', 'librt.so.1', 'libfontconfig.so.1', 'libfreetype.so.6', 'libxcb.so.1', 'libdl.so.2', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-armhf.so.3']
arm32/bin/bitcoin-wallet: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-armhf.so.3']
arm32/bin/bitcoind: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-armhf.so.3']
arm32/bin/bitcoin-tx: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-armhf.so.3']
arm32/bin/test_bitcoin: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-armhf.so.3']
```
#### LINUX X86_64
```bash
x86_64/bin/bitcoin-cli: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
x86_64/bin/bitcoin-qt: ['libpthread.so.0', 'librt.so.1', 'libfontconfig.so.1', 'libfreetype.so.6', 'libxcb.so.1', 'libdl.so.2', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
x86_64/bin/bitcoin-wallet: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
x86_64/bin/bitcoind: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
x86_64/bin/bitcoin-tx: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
x86_64/bin/test_bitcoin: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux-x86-64.so.2']
```
#### LINUX i686
```bash
i686/bin/bitcoin-cli: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux.so.2']
i686/bin/bitcoin-qt: ['libpthread.so.0', 'librt.so.1', 'libfontconfig.so.1', 'libfreetype.so.6', 'libxcb.so.1', 'libdl.so.2', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux.so.2']
i686/bin/bitcoin-wallet: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux.so.2']
i686/bin/bitcoind: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux.so.2']
i686/bin/bitcoin-tx: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux.so.2']
i686/bin/test_bitcoin: ['libpthread.so.0', 'librt.so.1', 'libm.so.6', 'libgcc_s.so.1', 'libc.so.6', 'ld-linux.so.2']
```
ACKs for top commit:
laanwj:
ACK f7453dcc03
Tree-SHA512: b418260edcda88583abfa386a592ebfb977d111e8e2ba887a30bf830b0b10dba429b9cfd615fad453ff0bb824225914ccb91433064b158ae1fbb9d20fc0b9937
Now that we require glibc 2.17+, #17538, we can remove linking in librt
for backwards compatibility purposes. The clock_* functions from librt
were merged into glibc as part of the 2.17 release.
* The `clock_*' suite of functions (declared in <time.h>) is now available
directly in the main C library. Previously it was necessary to link with
-lrt to use these functions. This change has the effect that a
single-threaded program that uses a function such as `clock_gettime' (and
is not linked with -lrt) will no longer implicitly load the pthreads
library at runtime and so will not suffer the overheads associated with
multi-thread support in other code such as the C++ runtime library.
https://sourceware.org/ml/libc-announce/2012/msg00001.html
Note that librt is already not linked by the RISC-V and AARCH64 binaries.
c78b123982 build: add -bind_at_load to hardened LDFLAGS (fanquake)
Pull request description:
This performs the same function as `-Wl,-z,now`, except for ld on macOS.
You can check the binaries using `otool -l`, and looking for the `LC_DYLD_INFO_ONLY` section; `lazy_bind_off` and `lazy_bind_size` should both be 0.
This seems to be the case with our current release binaries. However we can make the check, and applying the flag explicit in configure.
man ld:
```bash
-bind_at_load
Sets a bit in the mach header of the resulting binary which tells dyld
to bind all symbols when the binary is loaded, rather than lazily.
```
TODO:
- [ ] Follow up with `MH_BINDATLOAD` flag.
ACKs for top commit:
theuni:
ACK c78b123982.
Tree-SHA512: 12259558b84f7e3d75d6fcde63b517685e42b18fcf8e8cfcf347483c5ba089d3b4b6d330e7b7f61f83a328fe4d141b771e8e52ddee9cac6da87dfc073ab1183d
This performs the same function as -Wl,-z,now, except for ld on macOS.
You can check the binaries using otool -l, looking for the
LC_DYLD_INFO_ONLY section. lazy_bind_off and lazy_bind_size should both
be 0.
man ld:
-bind_at_load
Sets a bit in the mach header of the resulting binary which tells dyld
to bind all symbols when the binary is loaded, rather than lazily.
bd44711e1b build: pass -dead_strip_dylibs to ld on macOS (fanquake)
Pull request description:
This strips some unused dylibs from bitcoin-qt.
```diff
otool -L src/qt/bitcoin-qt
/usr/lib/libSystem.B.dylib
- /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
-/System/Library/Frameworks/Security.framework/Versions/A/Security
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
-/System/Library/Frameworks/AGL.framework/Versions/A/AGL
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/usr/lib/libc++.1.dylib
/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
/usr/lib/libobjc.A.dylib
```
`AGL` - ObjC wrapper for OpenGL.
`DiskArbitration` - mount/unmount notifications and events.
`Security` - low level security operations, authentication services.
From `man ld`:
```
Remove dylibs that are unreachable by the entry point or exported symbols.
That is, suppresses the generation of load command commands for dylibs
which supplied no symbols during the link. This option should not be
used when linking against a dylib which is required at runtime for
some indirect reason such as the dylib has an important initializer.
```
ACKs for top commit:
theuni:
ACK bd44711e1b.
Tree-SHA512: 9592ce2966d28cb6c58e01efd401f56a4baa5dc5be5313f4fe8454632b578608be65a23c8602772049cd4655a9cb020fdd40d6622a244c301920d8c3db43f99a
This strips some unused dylibs from bitcoin-qt.
From man ld:
Remove dylibs that are unreachable by the entry point or exported symbols.
That is, suppresses the generation of load command commands for dylibs
which supplied no symbols during the link. This option should not be
used when linking against a dylib which is required at runtime for
some indirect reason such as the dylib has an important initializer.
d1c02775aa Report amount of data gathered from environment (Pieter Wuille)
64e1e022ce Use thread-safe atomic in perfmon seeder (Pieter Wuille)
d61f2bb076 Run background seeding periodically instead of unpredictably (Pieter Wuille)
483b94292e Add information gathered through getauxval() (Pieter Wuille)
11793ea22e Feed CPUID data into RNG (Pieter Wuille)
a81c494b4c Use sysctl for seeding on MacOS/BSD (Pieter Wuille)
2554c1b81b Gather additional entropy from the environment (Pieter Wuille)
c2a262a78c Seed randomness with process id / thread id / various clocks (Pieter Wuille)
723c796667 [MOVEONLY] Move cpuid code from random & sha256 to compat/cpuid (Pieter Wuille)
cea3902015 [MOVEONLY] Move perfmon data gathering to new randomenv module (Pieter Wuille)
b51bae1a5a doc: minor corrections in random.cpp (fanquake)
Pull request description:
This introduces a new `randomenv` module that queries varies non-cryptographic (and non-RNG) sources of entropy available on the system; things like user IDs, system configuration, time, statistics, CPUID data.
The idea is that these provide a fallback in scenarios where system entropy is somehow broken (note that if system entropy *fails* we will abort regardless; this is only meant to function as a last resort against undetected failure). It includes some data sources OpenSSL currently uses, and more.
The separation between random and randomenv is a bit arbitrary, but I felt that all this "non-essential" functionality deserved to be separated from the core random module.
ACKs for top commit:
TheBlueMatt:
utACK d1c02775aa. Certainly no longer measuring the time elapsed between a 1ms sleep (which got removed in the latest change) is a fair tradeoff for adding about 2 million other actually-higher-entropy bits :).
laanwj:
ACK d1c02775aa
Tree-SHA512: d290a8db6538a164348118ee02079e4f4c8551749ea78fa44b2aad57f5df2ccbc2a12dc7d80d8f3e916d68cdd8e204faf9e1bcbec15f9054eba6b22f17c66ae3
f9af3ced1c Android: add all arch support (Block Mechanic)
d419ca7e32 depends: export dynamic JNI symbols from static qtforandroid.a (Igor Cota)
ed30684d03 Qt: patch androidjnimain.cpp to make sure JNI is initialised when statically compiled (Igor Cota)
e4c319e8a1 builds: remove superfluous config_opts_aarch64_android (Igor Cota)
24ffef0c27 Patch libevent when building for Android (fix arc4random_addrandom) (Igor Cota)
f1e40b3e71 Update bitcoin_qt.m4 (BlockMechanic)
b4057d8261 Define TARGET_OS when host is android (Igor Cota)
80b475f159 Fix Android zlib cross compilation issue (https://stackoverflow.com/questions/21396988/zlib-build-not-configuring-properly-with-cross-compiler-ignores-ar) (Igor Cota)
45f8219015 Add full Android build example command and instructions on getting SDK/NDK (Igor Cota)
b68f2a68c2 Add config opts and patch for aarch64_android build of Qt (Igor Cota)
9c4cb0166e Add ranlib to android.mk hosts file (fix OSX Android NDK build) (Igor Cota)
c2a749c9c1 Add example Android host-platform-triplet and options (Igor Cota)
0b0cff3c61 Add support for building Android dependencies (Igor Cota)
Pull request description:
This allows one to build the dependencies with the Android SDK and goes towards fixing #11844. It has been tested to work with:
`make HOST=aarch64-linux-android ANDROID_API_LEVEL=28 ANDROID_TOOLCHAIN_BIN=/home/user/Android/Sdk/ndk-bundle/toolchains/llvm/prebuilt/linux-x86_64/bin NO_QT=1 NO_WALLET=1`
ACKs for top commit:
Sjors:
ACK f9af3ce. I'm OK with merging and then improving later.
Tree-SHA512: cb805115ebe5c9e33db2bf3eab8628808fe3f50052053d8877d8b8e4406d6fea1ed9e5c4dff85d777fb99c81be6ffb9d95a0e6d32344e728e5e0da6c653e2ce7
44f7a8d7a7 Disable _FORTIFY_SOURCE when enable-debug (Andrew Chow)
Pull request description:
The `_FORTIFY_SOURCE` macro is enabled by default when hardening is enabled, but it requires optimization in order to be used. Since we disable all optimization with `--enable-debug`, this macro doesn't actually do anything and instead just causes a lot of warnings to be printed. This PR explicitly disables `_FORTIFY_SOURCE` so that these useless warnings aren't printed.
ACKs for top commit:
laanwj:
Thanks. ACK 44f7a8d7a7
Tree-SHA512: e9302aef794dfd9ca9d0d032179ecc51d3212a9a0204454419f410011343b27c32e6be05f385051b5b594c607b91b8e0e588f644584d6684429a649a413077d9
Since enable-debug disables optimization entirely, _FORTIFY_SOURCE
does nothing as it requires some kind of optimization enabled. It
instead produces a bunch of useless warnings. So explicitly disable
it when enable-debug so that those warnings are not produced.
Since libtool 1.5.2, on Linux libtool no longer sets RPATH for any
directories in the dynamic linker search path, so there is no longer an
issue.
This commit reverts a98356fee8.
as per doc/release-process.md.
Note: On branch-off, these same changes should be made on the release branch, with also these additional changes to both files:
- set `CLIENT_VERSION_REVISION` to `0`
- set `CLIENT_VERSION_IS_RELEASE` to `true`
b4fd0ca9be Include cstring for sanity_test_fdelt if required (Ben Woosley)
7fb886b1b1 [moveonly] Split glibc sanity_test_fdelt out (Ben Woosley)
Pull request description:
SmartOS FD_ZERO is implemented in a way that requires
an external declaration of memcpy. We can not simply
include cstring in the existing file because
sanity_test_memcpy is attempting to replace memcpy.
Instead split glibc_sanity into fdelt and memcpy files,
and include <cstring> in glibc_sanity/fdelt.cpp.
Fixes#13581, see also #13619
ACKs for top commit:
laanwj:
Code review an lightly tested (but not on SmartOS) ACK b4fd0ca9be
Tree-SHA512: 231306da291ad9eca8ba91bea1e9c27b6c2e96e484d1602e1c2cf27761202f9287ce0bc19fefd000943d2b449d0e5929cd39e2f7e09cf930d89fa520228ccbec
Enable users to see if the prop tests are enabled during the build. This can be particularly helpful as property-based tests are silently auto-enabled by default if librapidcheck is found.
Minor fixes to the docs and help grammar for this option.
d6ac25bdd9 autoconf: Sane --enable-debug defaults. (Carl Dong)
Pull request description:
```
Don't optimize even if variables adhere to as-if rule. This is a
somewhat sane default for debugging.
```
-----
Fixes: #14830
This is more of a "do something dumb and have people correct you" kind of PR. The end goal is to have a configure flag that will allow for debugging without annoying "optimized out" messages, for developer experiences' sake. This is the minimal diff, but people have suggested `--enable-debug-slow` in the past.
ACKs for top commit:
jonasschnelli:
Tested ACK d6ac25bdd9
Tree-SHA512: 7a5576ad1d33850aff1445ccb71b133f654b455da2d1daed2ed1b82ea773965790a62895aeeab74b23a25513ab96dddb670f9dbc593dd0b8c030694206a99ccf
