fac0c3d4bf doc: Add release notes for two pull requests (MarcoFalke)
fa7b57e5f5 refactor: Replace ParseHashStr with FromHex (MarcoFalke)
fa90777245 rest: Reject truncated hex txid early in getutxos parsing (MarcoFalke)
fab6ddbee6 refactor: Expose FromHex in transaction_identifier (MarcoFalke)
fad2991ba0 refactor: Implement strict uint256::FromHex() (MarcoFalke)
fa103db2bb scripted-diff: Rename SetHex to SetHexDeprecated (MarcoFalke)
fafe4b8051 test: refactor: Replace SetHex with uint256 constructor directly (MarcoFalke)
Pull request description:
In `rest_getutxos` truncated txids such as `aa` or `ff` are accepted. This is brittle at best.
Fix it by rejecting any truncated (or overlarge) input.
----
Review note: This also starts a major refactor to rework hex parsing in Bitcoin Core, meaning that a few refactor commits are included as well. They are explained individually in the commit message and the work will be continued in the future.
ACKs for top commit:
stickies-v:
re-ACK fac0c3d4bf - only doc and test updates to address review comments, thanks!
hodlinator:
ACK fac0c3d4bf
Tree-SHA512: 473feb3fcf6118443435d1dd321006135b0b54689bfbbcb1697bb5811a449bef51f475c715de6911ff3c4ea3bdb75f601861ff93347bc4414d6b9e5298105dd7
faa3598772 ci: Add missing qttools5-dev install to Asan task (MarcoFalke)
Pull request description:
This is required, according to the docs:
```
$ git grep --line-number 'qtbase5-dev qttools5-dev qttools5-dev-tools' doc
doc/build-unix.md:84: sudo apt-get install qtbase5-dev qttools5-dev qttools5-dev-tools
```
Also, needed for cmake.
ACKs for top commit:
hebasto:
ACK faa3598772.
Tree-SHA512: c986908f757d70d958267c1e902b5d7d94589360db61ddf7b9b398cd635b2172e83510c0c77fd6032810166342a286c0f95225b6c6639acd869e1e51c3348ea7
25bf86a225 [test]: ensure `estimatesmartfee` default mode is `economical` (ismaelsadeeq)
41a2545046 [fees]: change `estimatesmartfee` default mode to `economical` (ismaelsadeeq)
Pull request description:
Fixes#30009
This PR changes the `estimatesmartfee` default mode to `economical`.
This was also suggested on IRC https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2024-04-26#1021609
- `conservative` mode: This is the `estimatesmartfee` RPC mode which considers a longer history of blocks. It potentially returns a higher fee rate and is more likely to be sufficient for the desired target, but it is not as responsive to short-term drops in the prevailing fee market.
- `economical` mode: This is the `estimatesmartfee` RPC mode where estimates are potentially lower and more responsive to short-term drops in the prevailing fee market.
Since users are likely to use the default mode, this change will reduce overestimation for many users. The conservative mode remains available for those who wish to opt-in.
For an in-depth analysis of how significantly the `conservative` mode overestimates, see
https://delvingbitcoin.org/t/bitcoind-policy-estimator-modes-analysis/964.
ACKs for top commit:
instagibbs:
reACK 25bf86a225
glozow:
ACK 25bf86a225
willcl-ark:
ACK 25bf86a225
Tree-SHA512: 78ebda667eb9c8f87dcc2f0e6c14968bd1de30358dc77a13611b186fb8427ad97d9f537bad6e32e0a1aa477ccd8c64fee4d41e19308ef3cb184ff1664e6ba8a6
e3edaccd9d ci: add _LIBCPP_REMOVE_TRANSITIVE_INCLUDES to TSAN job (fanquake)
6e786165ca refactor: fix missing includes (fanquake)
Pull request description:
Add `_LIBCPP_REMOVE_TRANSITIVE_INCLUDES` to one of the libc++ CI jobs, to catch missing includes, that are otherwise hidden by transitive includes inside libc++. A more appropriate place for this might be the tidy job, but that does not use libc++.
See https://libcxx.llvm.org/DesignDocs/HeaderRemovalPolicy.html for more information.
ACKs for top commit:
maflcko:
re-ACK e3edaccd9d
Tree-SHA512: 3fb2e9bbbf4bb1570633d52939875ee674d934b645a4037a309643f84ab69edf0fb5b6cfcbd02fa7d92052a64fa63f31979a58fede23593c4df7c33a8cb2953a
This is a safe replacement of the previous SetHex, which now returns an
optional to indicate success or failure.
The code is similar to the ParseHashStr helper, which will be removed in
a later commit.
These cause compile failures with _LIBCPP_REMOVE_TRANSITIVE_INCLUDES.
i.e:
```bash
In file included from init.cpp:8:
./init.h:46:54: error: no template named 'atomic' in namespace 'std'
46 | bool AppInitBasicSetup(const ArgsManager& args, std::atomic<int>& exit_status);
| ~~~~~^
1 error generated.
```
See: https://libcxx.llvm.org/DesignDocs/HeaderRemovalPolicy.html.
1bc9f64bee contrib: assume binary existence in sec/sym checks (fanquake)
51d8f435c9 contrib: simplify ELF test-security-check (fanquake)
1810e20677 contrib: simplify PE test-security-check (fanquake)
6c9746ff92 contrib: simplify MACHO test-security-check (fanquake)
Pull request description:
The current `test-security-check` script is hard to understand, and change (i.e https://github.com/bitcoin/bitcoin/pull/29987/files#diff-52aa0cda44721f089e53b128cb1232a876006ef257b211655456b17dfb2ec712); tests are also not done in isolation (when-possible). Fix that, and add missing checks. Simplifies future toolchain/security/hardening changes.
ACKs for top commit:
hebasto:
ACK 1bc9f64bee (assuming my Guix hashes match; I'll provide them shortly).
TheCharlatan:
ACK 1bc9f64bee
Tree-SHA512: 1885d0ce63a94ffa61345327f919da20b63de6dd4148d6db3ee8bad4485253a36e8ab0dbee48cecc02ea35d139edfed75453af45fc364bcbef6fe16b6823bc7a
c85accecaf [refactor] delete EraseTxNoLock, just use EraseTx (glozow)
6ff84069a5 remove obsoleted TxOrphanage::m_mutex (glozow)
61745c7451 lock m_recent_confirmed_transactions using m_tx_download_mutex (glozow)
723ea0f9a5 remove obsoleted hashRecentRejectsChainTip (glozow)
18a4355250 update recent_rejects filters on ActiveTipChange (glozow)
36f170d879 add ValidationInterface::ActiveTipChange (glozow)
3eb1307df0 guard TxRequest and rejection caches with new mutex (glozow)
Pull request description:
See #27463 for full project tracking.
This contains the first few commits of #30110, which require some thinking about thread safety in review.
- Introduce a new `m_tx_download_mutex` which guards the transaction download data structures including `m_txrequest`, the rolling bloom filters, and `m_orphanage`. Later this should become the mutex guarding `TxDownloadManager`.
- `m_txrequest` doesn't need to be guarded using `cs_main` anymore
- `m_recent_confirmed_transactions` doesn't need its own lock anymore
- `m_orphanage` doesn't need its own lock anymore
- Adds a new `ValidationInterface` event, `ActiveTipChanged`, which is a synchronous callback whenever the tip of the active chainstate changes.
- Flush `m_recent_rejects` and `m_recent_rejects_reconsiderable` on `ActiveTipChanged` just once instead of checking the tip every time `AlreadyHaveTx` is called. This should speed up calls to that function (no longer comparing a block hash each time) and removes the need to lock `cs_main` every time it is called.
Motivation:
- These data structures need synchronization. While we are holding `m_tx_download_mutex`, these should hold:
- a tx hash in `m_txrequest` is not also in `m_orphanage`
- a tx hash in `m_txrequest` is not also in `m_recent_rejects` or `m_recent_confirmed_transactions`
- In the future, orphan resolution tracking should also be synchronized. If a tx has an entry in the orphan resolution tracker, it is also in `m_orphanage`, and not in `m_txrequest`, etc.
- Currently, `cs_main` is used to e.g. sync accesses to `m_txrequest`. We should not broaden the scope of things it locks.
- Currently, we need to know the current chainstate every time we call `AlreadyHaveTx` so we can decide whether we should update it. Every call compares the current tip hash with `hashRecentRejectsChainTip`. It is more efficient to have a validation interface callback that updates the rejection filters whenever the chain tip changes.
ACKs for top commit:
instagibbs:
reACK c85accecaf
dergoegge:
Code review ACK c85accecaf
theStack:
Light code-review ACK c85accecaf
hebasto:
ACK c85accecaf, I have reviewed the code and it looks OK.
Tree-SHA512: c3bd524b5de1cafc9a10770dadb484cc479d6d4c687d80dd0f176d339fd95f73b85cb44cb3b6b464d38a52e20feda00aa2a1da5a73339e31831687e4bd0aa0c5
a517029646 depends: switch to building expat with CMake (fanquake)
Pull request description:
Switch to building Expat with CMake, instead of Autotools.
ACKs for top commit:
hebasto:
re-ACK a517029646.
Tree-SHA512: ca040545dd83fb81a8b209aa24cae6e22eaeff04f44bdabc4454adf6ea63d34f4ae27bd5980c65db2d2542e23eb2712102719023c262ab63a933c90b5999c11e
SetHex is fragile, because it accepts any non-hex input or any length of
input, without error feedback. This can lead to issues when the input is
truncated or otherwise corrupted.
Document the problem by renaming the method.
In the future, the fragile method should be removed from the public
interface.
-BEGIN VERIFY SCRIPT-
sed -i 's/SetHex/SetHexDeprecated/g' $( git grep -l SetHex ./src )
-END VERIFY SCRIPT-
09ce3501fa fix: Make TxidFromString() respect string_view length (Hodlinator)
01e314ce0a refactor: Change base_blob::SetHex() to take std::string_view (Hodlinator)
2f5577dc2e test: uint256 - Garbage suffixes and zero padding (Hodlinator)
f11f816800 refactor: Make uint256_tests no longer use deprecated BOOST_CHECK() (Hodlinator)
f0eeee2dc1 test: Add test for TxidFromString() behavior (Ryan Ofsky)
Pull request description:
### Problem
Prior to this, `TxidFromString()` was passing `string_view::data()` into `uint256S()` which meant it would only receive the a naked `char*` pointer and potentially scan past the `string_view::length()` until it found a null terminator (or some other non-hex character).
Appears to have been a fully dormant bug as callers were either passing a string literal or `std::string` directly to `TxidFromFromString()`, meaning a null terminator always existed at `pointer[length()]`. Bug existed since original merge of `TxidFromString()`.
### Solution
Make `uint256S()` (and `base_blob::SetHex()`) take and operate on `std::string_view` instead of `const char*` and have `TxidFromString()` pass that in.
(PR was prompted by comment in https://github.com/bitcoin/bitcoin/pull/30377#issuecomment-2208857200 (referring to https://github.com/bitcoin/bitcoin/pull/28922#discussion_r1404437378)).
ACKs for top commit:
maflcko:
re-ACK 09ce3501fa🕓
paplorinc:
ACK 09ce3501fa
ryanofsky:
Code review ACK 09ce3501fa. I think the current code changes are about as small as you could make to fix the bug without introducing a string copy, and the surrounding test improvements are all very nice and welcome.
Tree-SHA512: c2c10551785fb6688d1e2492ba42a8eee4c19abbe8461bb0774d56a70c23cd6b0718d2641632890bee880c06202dee148126447dd2264eaed4f5fee7e1bcb581
29eafd5733 rpc: doc: use "output script" terminology consistently in "asm"/"hex" results (Sebastian Falbesoner)
Pull request description:
The wording "public key script" was likely chosen as a human-readable form of the technical term `scriptPubKey`, but it doesn't seem to be really widespread. Replace it by the more (probably most?) common term "output script" instead. Note that the argument for the `decodescript` RPC is not necessarily an output script (it could e.g. be also a redeem script), so in this case we just stay generic and use "script".
See also the draft BIP "Terminology for Transaction Components" (https://github.com/murchandamus/bips/blob/2022-04-tx-terminology/bip-tx-terminology.mediawiki) from murchandamus which suggests to use "output script" as well.
Affects the help text of the following RPCs:
- decodepsbt
- decoderawtransaction
- decodescript
- getblock (if verbosity=3)
- getrawtransaction (if verbosity=2,3)
- gettxout
ACKs for top commit:
maflcko:
ACK 29eafd5733
achow101:
ACK 29eafd5733
BrandonOdiwuor:
ACK 29eafd5733
tdb3:
ACK 29eafd5733
Tree-SHA512: 62eb92d42bc44e36dc3090df7b248a123868a74af253d2046de02086e688bf6ff98307b927ba2fee3d599f85e073aeb8eca90ed15105ca63b648b6796cfa340b
d63ef73800 test: Add loadtxoutset test with tip on snapshot block (Fabian Jahr)
c2f86d4bcb test: Remove already resolved assumeutxo todo comments (Fabian Jahr)
Pull request description:
The first commit removes three Todos that have been addressed previously (see commit message for details).
The second message resolves another todo by adding the missing test case. This is a special case of "the tip has more work than the snapshot" where the tip is the same block as the snapshot base block.
Related to #28648.
ACKs for top commit:
jrakibi:
ACK [d63ef73](d63ef73800)
achow101:
ACK d63ef73800
maflcko:
ACK d63ef73800
alfonsoromanz:
Re ACK d63ef73800
Tree-SHA512: 8d5a25fc0b26531db3a9740132694138f2103b7b42eeb1d4a64095bfc901c1372e23601c0855c7def84c8a4e185d10611e4e830c4e479f1b663ae6ed53abb130
Prior to this, passing string_view::data() into uint256S() meant the latter would only receive the a naked char* pointer and potentially scan past the string_view::length() until it found a null terminator (or some other non-hex character).
Appears to have been a fully dormant bug as callers were either passing a string literal or std::string directly to TxidFromFromString(), meaning null terminator always existed at pointer[length()]. Bug existed since original merge of TxidFromString(), discussed in https://github.com/bitcoin/bitcoin/pull/28922#discussion_r1404437378.
6a5e9e40e1 doc: use proper doxygen formatting for CTxMemPool::cs (Vasil Dimov)
Pull request description:
Having `@par title` followed by an empty line renders improperly in Doxygen - it results in a paragraph with a title but without a body.
https://www.doxygen.nl/manual/commands.html#cmdpar
This also results in a compiler warning (or error) with Clang 19:
```
./txmempool.h:368:34: error: empty paragraph passed to '@par' command [-Werror,-Wdocumentation]
368 | * @par Consistency guarantees
| ~~~~~~~~~~~~~~~~~~~~~~~~~~^
1 error generated.
```
ACKs for top commit:
maflcko:
review ACK 6a5e9e40e1
tdb3:
ACK 6a5e9e40e1
Tree-SHA512: 2c4c9e5fd4bd44754800a9bcfff74df101afc060b84451c45aa098e4ceb05a47f28a36f8473b31222552fad6339b752a148e6b1c7d41c2003f515b3eb4060902
Having `@par title` followed by an empty line renders improperly in
Doxygen - it results in a paragraph with a title but without a body.
https://www.doxygen.nl/manual/commands.html#cmdpar
This also results in a compiler warning (or error) with Clang 19:
```
./txmempool.h:368:34: error: empty paragraph passed to '@par' command [-Werror,-Wdocumentation]
368 | * @par Consistency guarantees
| ~~~~~~~~~~~~~~~~~~~~~~~~~~^
1 error generated.
```
fa33a63bd9 fuzz: Speed up PickValue in txorphan (MarcoFalke)
Pull request description:
`PickValue` will advance a begin iterator on the `outpoints` set, which is expensive, because it only has a `++` operator. As it is called in a loop of `num_in` (~`outpoints.size()`), the runtime is `O(outpoints.size() ^ 2)`.
Fix it by making the runtime linear.
ACKs for top commit:
glozow:
ACK fa33a63bd9, thanks for taking the suggestion
dergoegge:
utACK fa33a63bd9
Tree-SHA512: 33f440d97c6834d907d43a8d29e4fb2c995f0d244460bd079af100f13d3607a53e44a0db52f4eb5c487d98df0ff4f2f6d987bf94b922ae9f4506f1295ad6214c
8c935e625e depends: Fix CMake-generated `libevent*.pc` files (Hennadii Stepanov)
Pull request description:
Broken out of #30454. This is a backport of the merged upstream PR: https://github.com/libevent/libevent/pull/1622.
Note that after #29835 we might end up dropping pkg-config and using the installed CMake files directly, but that depends on whether or not enough distros actually ship those files.
Either way, having fixed up .pc files won't hurt.
ACKs for top commit:
hebasto:
ACK 8c935e625e.
fanquake:
ACK 8c935e625e
Tree-SHA512: 259c2ad78fb9e90370a7205dc71c40acda1a872f6509435133bc1c4c2c3de57366e80679aa083e13ed85e7966883dc470c0147ee171a2ed0171a18cd5ffc99b3
fa7bee13bf lint: Use git clone --depth=1 (MarcoFalke)
fadb7c2a91 lint: Add missing docker.io prefix to ci/lint_imagefile (MarcoFalke)
Pull request description:
Currently, the `ci/lint_imagefile` may pick the wrong (non-native) architecture due to the missing prefix.
For example, assuming the user has previously pulled an s390x image:
```
$ podman run --rm 'docker.io/s390x/debian:bookworm' dpkg --print-architecture
exec /usr/bin/dpkg: exec format error
```
Now, `debian:bookworm` will refer to the same image:
```
$ podman run --rm 'debian:bookworm' dpkg --print-architecture
exec /usr/bin/dpkg: exec format error
```
However, `docker.io/debian:bookworm` works fine:
```
$ podman run --rm 'docker.io/debian:bookworm' dpkg --print-architecture
arm64
```
(Also includes a nit-fix from https://github.com/bitcoin/bitcoin/pull/30499#discussion_r1686470495)
ACKs for top commit:
paplorinc:
utACK fa7bee13bf
hebasto:
ACK fa7bee13bf.
Tree-SHA512: 4b6d562c14c67bef984ad25f6a3a1ef7f1059dc2859c603c45083b36bcacafa3248fc74176e2e4626fdc39507e9353f458ddbc4077f805c03e970df46af02224
0388ad0d65 depends: switch zmq to CMake (Cory Fields)
fefb3bbe5b depends: add zeromq no librt patch (fanquake)
a522ef1542 depends: add zeromq cmake minimum patch (fanquake)
cbbc229adf depends: add zeromq windows usage patch (fanquake)
2de68d6d38 depends: add zeromq builtin sha1 patch (fanquake)
0c8605253a depends: add zeromq mktemp macos patch (fanquake)
Pull request description:
This picks up a change, which is a switch to building zeromq with CMake. It includes a number of patches, some which have already been upstreamed (see each patch for details).
ACKs for top commit:
hebasto:
ACK 0388ad0d65.
Tree-SHA512: 5567e432b4e4e0446c41d502bd61810a80b329dea2399b5d9d9f6e79acc450d1c6ba861c8238ba895de98338cfc5dc44ad2bf86ee8c222ecb3fbf47d6eb60da4
fa8d73e86e lint: Use consistent out-of-tree build for python and test_runner (MarcoFalke)
fa0f859885 doc: Clarify intent of ./ci/lint_run_all.sh (MarcoFalke)
fa9ad59f87 lint: Use $CI_RETRY_EXE when building ./ci/lint_imagefile (MarcoFalke)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/30496
Seems odd to sometimes do an out-of-tree build (via `./ci/lint_imagefile`, see `test/lint/README.md`) and sometimes not (via Cirrus CI, see `./ci/lint_run_all.sh`).
Fix it by doing an out-of-tree build consistently in the same location.
Also, fix `$CI_RETRY_EXE`, while touching this.
ACKs for top commit:
josibake:
utACK fa8d73e86e
willcl-ark:
utACK fa8d73e86e
paplorinc:
utACK fa8d73e86e
Tree-SHA512: 4181ca14299a798850f5e05f180f3305a3378081ca8dabf6ab2da6115997cc17f6ef0f10db9b2b31618e59231083e5c4a971432d27b4d77903e655be21155abb
Previous code was confusing and brittle. For example, the full import
"source ./ci/test/00_setup_env.sh" and $PATH overwrite was not needed.
Fix it by simply copying the exe to /ci_retry and use that in
$CI_RETRY_EXE.
This is also a fix, because previously ci/lint_imagefile did use an
empty $CI_RETRY_EXE.
a8e3af1a82 qa: Do not assume running `feature_asmap.py` from source directory (Hennadii Stepanov)
9bf7ca6cad qa: Consider `cache` and `config.ini` relative to invocation directory (Hennadii Stepanov)
a0473442d1 scripted-diff: Add `__file__` argument to `BitcoinTestFramework.init()` (Hennadii Stepanov)
Pull request description:
This PR includes changes split from https://github.com/bitcoin/bitcoin/pull/30454. They improve the functional test framework, allowing users to [run individual functional tests](https://github.com/hebasto/bitcoin/issues/146) from the build directory in the new CMake-based build system.
This functionality is not available for out-of-source builds using the current Autotools-based build system, which always requires write permissions for the source directory. Nevertheless, this PR can be tested as suggested in https://github.com/bitcoin/bitcoin/pull/30463#issuecomment-2232618421:
1. Make an out-of-source build:
```
$ ./autogen.sh
$ mkdir ../build && cd ../build
$ ../bitcoin/configure
$ make
```
2. Create a symlink in the build directory to a functional test:
```
$ ln --symbolic ../../../bitcoin/test/functional/wallet_disable.py ./test/functional/
```
3. Run this symlink:
```
$ ./test/functional/wallet_disable.py
```
The last command fails on the master branch:
```
Traceback (most recent call last):
File "/home/hebasto/git/build/./test/functional/wallet_disable.py", line 31, in <module>
DisableWalletTest().main()
^^^^^^^^^^^^^^^^^^^
File "/home/hebasto/git/bitcoin/test/functional/test_framework/test_framework.py", line 106, in __init__
self.parse_args()
File "/home/hebasto/git/bitcoin/test/functional/test_framework/test_framework.py", line 210, in parse_args
config.read_file(open(self.options.configfile))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/hebasto/git/bitcoin/test/config.ini'
```
and succeeds with this PR.
ACKs for top commit:
maflcko:
tested ACK a8e3af1a82🎨
glozow:
ACK a8e3af1a82, tested with the steps in op
stickies-v:
ACK a8e3af1a82
Tree-SHA512: 899e4efc09edec13ea3f5b47825d03173fb21d3569c360deda7fa6a56b99b4d24e09ad4f0883bad1ee926b1c706e47ba07c6a6160c63c07c82b3cf4ae5816e91
7703884ab1 Fix MSVC warning C4273 "inconsistent dll linkage" (Hennadii Stepanov)
Pull request description:
Broken out of https://github.com/bitcoin/bitcoin/pull/30454.
When using CMake, the user can select the MSVC runtime library to be:
1) Statically-linked (with the corresponding `x64-windows-static` vcpkg triplet) or
2) Dynamically-linked (with the corresponding `x64-windows` vcpkg triplet)
In the latter case, the compiler emits the [C4273](https://learn.microsoft.com/en-us/cpp/error-messages/compiler-warnings/compiler-warning-level-1-c4273) warning.
As the "Necessary on some platforms" comment does not apply to MSVC, skip the declaration for MSVC.
The MSVC build system in the master branch supports the statically-linked runtime only: ed739d14b5/build_msvc/common.init.vcxproj.in (L65)
ACKs for top commit:
sipa:
utACK 7703884ab1
sipsorcery:
utACK 7703884ab1.
theuni:
utACK 7703884ab1
Tree-SHA512: a42e1a0d48973217462e703c418f3e9ef9cb5236267c1bf32912aacaf68976cdd2b9229168523f7c2a99ee3f2fb1bf8add4f342796bdb1e4063ca026b761db51
d318c4ef56 depends: bump libmultiprocess for CMake fixes (Cory Fields)
Pull request description:
Broken out of #30454 . Bumped [even further](4883197abc (r1684802528)) after https://github.com/chaincodelabs/libmultiprocess/pull/98 was merged upstream.
hebasto Presumably this approach works now with the CMake branch?
ACKs for top commit:
ryanofsky:
Code review ACK d318c4ef56.
Tree-SHA512: 4b5491f73c0063d09e8339829cd831b1f4c441dd7b55a22037c9337c80cce19bb00a3e5cf925efa77d6d4e89ab45482f40f5799bc14948f8cabcbad3c3549430
