caac999ff0 refactor: remove dependence on AddrManTest (josibake)
f961c477b5 refactor: check Good() in tried_collisions test (josibake)
207f1c825c refactor: make AddrMan::Good return bool (josibake)
Pull request description:
Previously, the `addrman_tried_collisions` test behaved in the following way:
1. add an address to addrman
2. attempt to move the new address to the tried table (using `AddrMan.Good()`)
3. verify that `num_addrs` matched `size()` to check for collisions in the new table
`AddrMan.size()`, however, returns the number of unique address in addrman, regardless of whether they are in new or tried. This means the test would still pass for addresses where a collision did occur in the tried table. After 3 collisions in the tried table, there would eventually be a collision in the new table when trying to add a new address, which was then detected by checking `num_addrs - collisions == size()`.
While the collision in the new table was caused by a collision in the tried table, the test is misleading as it's not directly testing for collisions in the tried table and misses 3 collisions before identifying a collision in the new table.
### solution
To more directly test the tried table, I refactored `AddrMan::Good()` to return a boolean after successfully adding an address to the tried table. This makes the test much cleaner by first adding an address to new, calling `Good` to move it to the tried table, and checking if it was successful or not. It is worth noting there are other reasons, aside from collisions, which will cause `Good` to return false. That being said, this is an improvement over the previous testing methodology.
Additionally, having `Good()` return a boolean is useful outside of testing as it allows the caller to handle the case where `Good` is unable to move the entry to the tried table (e.g a063647413/src/rpc/net.cpp (L945)).
### followup
As a follow up to this PR, I plan to look at the following places `Good()` is called and see if it makes sense to handle the case where it is unable to add an entry to tried:
* a063647413/src/rpc/net.cpp (L945)
* a063647413/src/net.cpp (L2067)
* a063647413/src/net_processing.cpp (L2708)
ACKs for top commit:
jnewbery:
utACK caac999ff0
mzumsande:
Code review ACK caac999ff0
Tree-SHA512: f328896b1f095e8d2581fcdbddce46fc0491731a0440c6fff01081fa5696cfb896dbbe1d183eda2c100f19aa111e1f8b096ef93582197edc6b791de563a58f99
eaf6be0114 [net processing] Do not request transaction relay from feeler connections (John Newbery)
0220b834b1 [test] Add testing for outbound feeler connections (John Newbery)
Pull request description:
Feelers are short-lived connections used to test the viability of peers. The bitcoind node will periodically open feeler connections to addresses in its addrman, wait for a `version` message from the peer, and then close the connection.
Currently, we set `fRelay` to `1` in the `version` message for feeler connections, indicating that we want the peer to relay transactions to us. However, we close the connection immediately on receipt of the `version` message, and so never process any incoming transaction announcements. This PR changes that behaviour to instead set `fRelay` to `0` indicating that we do not wish to receive transaction announcements from the peer.
This PR also extends the `addconnection` RPC to allow creating outbound feeler connections from the node to the test framework, and a test to verify that the node sets `fRelay` to `0` in the `version` message to feeler connections.
ACKs for top commit:
naumenkogs:
ACK eaf6be0114
MarcoFalke:
review ACK eaf6be0114🏃
Tree-SHA512: 1c56837dbd0a396fe404a5e39f7459864d15f666664d6b35ad109628b13158e077e417e586bf48946a23bd5cbe63716cb4bf22cdf8781b74dfce6047b87b465a
Rather than try to infer a collision by checking `AddrMan::size`,
check whether or not moving to the tried table was successful by
checking the output from `AddrMan::Good`
If AddrMan::Good is unable to add an entry
to tried (for a number of reasons), return false.
This makes it much easier and cleaner to directly
test for tried collisions. It also allows anyone
calling Good() to handle the case where adding an
address to tried is unsuccessful.
Update docs to doxygen style.
fa19bab90a fuzz: Rework FillNode (MarcoFalke)
fae6e31df7 refactor: Set fSuccessfullyConnected in FillNode (MarcoFalke)
fa3583f856 fuzz: Avoid negative NodeId in ConsumeNode (MarcoFalke)
Pull request description:
Currently `FillNode` is a bit clumsy because it directly modifies memory of `CNode`. This gets in the way of moving that memory to `Peer`. Also, it isn't particularly consistent. See for example https://github.com/bitcoin/bitcoin/pull/21160#discussion_r739206139 .
Fix all issues by sending a `version`/`verack` in `FillNode` and let net_processing figure out the internal details.
ACKs for top commit:
jnewbery:
Strong concept ACK and light code review ACK fa19bab90a
Tree-SHA512: 33261d857c3fa6d5d39d742624009a29178ad5a15eb3fd062da741affa5a4854fd45ed20d59a6bba2fb068cf7b39cad6f95b2910be7cb6afdc27cd7917955b67
9544ab60ce doc: Improve ZMQ documentation (node01)
Pull request description:
This PR intends to clarify:
. when ZMQ notifications occur
. the message structure of each topic
Closes https://github.com/bitcoin/bitcoin/issues/23452#issue-1046579585
ACKs for top commit:
theStack:
ACK 9544ab60ce
Tree-SHA512: 3b1314ad5eb8c359ffabd91ce9b47cf6cf8efa69be083a3bb5d865833ac67d7a88a8e012c4a4f59f2cd2a0e7f491e828897e85e9d01bae72ee83c4a6ad459f6e
1ef2c03452 Add multisig tutorial (lsilva01)
Pull request description:
This PR adds a mutisig tutorial, as requested in https://github.com/bitcoin/bitcoin/issues/21278
Although there is already a brief explanation and a functional test about the multisig implemented in https://github.com/bitcoin/bitcoin/pull/22067, this tutorial proposes to use the signet (instead of regtest), bringing the reader closer to a real environment and explaining some functions in more detail.
I'm not sure if this format should be in this repository or on some wiki page. But as there is an open issue regarding this matter, that is my suggestion.
ACKs for top commit:
Sjors:
re-utACK 1ef2c03452
prayank23:
ACK 1ef2c03452
Tree-SHA512: 2c3f17a8c50e554f802029dceb28ab90a77021f135b8cbd77dca3879ba1f1a0eac6bda0afb90d1ff6b8116fb0628471687d3fb77bb255ef5d8b9590b775cbce9
fa26c55644 wallet: Replace Assume with Assert where needed in coinselection (MarcoFalke)
Pull request description:
`Assume` should only be used when a failed check is recoverable. The checks here don't recover and would run into UB, so use `Assert` instead.
ACKs for top commit:
theStack:
Code-review ACK fa26c55644
Tree-SHA512: 0cf9435f9ec44794022ce0274cba602aec95102ab73f4c8a93dae54ef4c0a594f6a81640477039719ddfb6f23b05f8ece3e4886ef7f8a725efff45685ac49d92
fa24a3df87 rpc: Quote user supplied strings in error messages (MarcoFalke)
Pull request description:
I can't see a downside doing this and this fixes a fuzzing crash
Background:
This is a follow-up to commit 926fc2a0d4, which introduced the "starts_with-hack". Maybe an alternative to the hack would be to assign a unique error code to internal bugs? However, I think this can be done in an separate pull request and the changes here make sense even on their own.
ACKs for top commit:
fanquake:
ACK fa24a3df87 - to fix the fuzzers.
Tree-SHA512: d998626406a64396a037a6d1fce22fce3dadb7567c2f9638e450ebe8fb8ae77d134e15dd02555326732208f698d77b0028bc62be9ceee9c43282b61fe95fccbd
e9440aeb5c build: use __SIZEOF_INT128__ for checking __int128 availability (fanquake)
Pull request description:
We already use this in the blockfilter code,
bf66e258a8/src/blockfilter.cpp (L34-L36)
so not sure we need to maintain two different ways of testing
for the same functionality. Consolidate on testing for `__SIZEOF_INT128__`,
which we already use, is supported by the compilers we care about, and is
also used by libsecp256k1.
ACKs for top commit:
sipa:
utACK e9440aeb5c
Zero-1729:
crACK e9440aeb5c
Tree-SHA512: 8aeef1734486a863b5091123bb5f9ba8868b1e2b4b35114586e3eb5862a38d4a1518ed069f37f41cb5e5ce2f6c87d95671996366d5ee990e0c90f268a8978ba3
41b9f7d062 test: Use byte unit 'M' for -maxuploadtarget functional test (Douglas Chimento)
Pull request description:
ACKs for top commit:
shaavan:
ACK 41b9f7d062
stratospher:
ACK 41b9f7d.
Tree-SHA512: 25b46347c671e8d6fd8878e7fee40e773bb03641e53e41e8a79a286fe4a0cf71c0c0986d6d7418fcb656c07f7216cc50a7ee4366f9213c32b01ae74326031f80
50209a42ad validation, doc: remove TODO comment (Jon Atack)
8e37fa8393 validation, log: improve logging in FlushSnapshotToDisk() (Jon Atack)
271252c0bd validation, log: extract FlushSnapshotToDisk() function (Jon Atack)
Pull request description:
Use the `LOG_TIME_MILLIS_WITH_CATEGORY_MSG_ONCE` macro to improve the logging of ChainstateManager snapshot persistance, log task start and completion separately and no longer manually track the duration, as suggested by Marco Falke in https://github.com/bitcoin/bitcoin/pull/22872#discussion_r715571280.
Extract the flushing into one function, which clarifies the logic, extends the improved logging to both flushing call sites, and allows logging the prefix `FlushSnapshotToDisk`, which is similar to `FlushStateToDisk`.
before
```
[snapshot] flushing coins cache (0 MB)... done (0.00ms)
[snapshot] flushing snapshot chainstate to disk
```
after
```
FlushSnapshotToDisk: flushing coins cache (0 MB) started
...
FlushSnapshotToDisk: completed (0.00ms)
FlushSnapshotToDisk: saving snapshot chainstate (0 MB) started
...
FlushSnapshotToDisk: completed (0.00ms)
```
The logging can be observed in the output of
```
./src/test/test_bitcoin -t validation_chainstate_tests -- DEBUG_LOG_OUT
```
Top commit has no ACKs.
Tree-SHA512: 5d954cd8c7455f8625152a43663a237f04717bb834aed62925a56e17c711fca6ccfc03783970b6b0bde44f64617d804b423a7048287c06ee816db36247acf272
We already use this in the blockfilter code, so not sure we need to maintain two
different ways of testing for the same functionality. Consolidate on testing
for __SIZEOF_INT128__, which we already use, is supported by the compilers we
care about, and is also used by libsecp256k1.
9600ea0145 test: Add edge case of pruning up to index height (Martin Zumsande)
698c524698 index: Fix backwards search for bestblock (Martin Zumsande)
Pull request description:
This PR attempts to fix an intermittent Init issue encountered during the stress testing of #23289, which relates to the pruning-compatible filter reconstruction logic introduced in #15946.
The problem would occur when the node starts with `-txindex=1` but `ThreadSync` is interrupted after it sets `m_best_block_index` to Genesis, and before it gets do any further work.
In that case, during the next restart of the node, an Init error would be thrown because `BaseIndex::Init()` tries to backtrack from the tip to the last block which has been successfully indexed (here: Genesis), but the backtracking logic didn't work properly in this case:
The loop
`while (block_to_test && block->pprev && (block->pprev->nStatus & BLOCK_HAVE_DATA))` checks if a predecessor exists **before** performing the check `block_to_test == block` and then possbily setting `prune_violation = false`
If `block_to_test` and `block` are the Genesis block this check will not be reached because `block->pprev` does not exist.
To reproduce this bug on regtest:
1) start a node with a fresh datadir using `-txindex=1` (or any other index)
2) stop and restart without any index
3) mine a block
3) stop and restart again with the index enabled
->InitError `Error: txindex best block of the index goes beyond pruned data. (...)`
Fix this by requiring that we have the data for the block of the current iteration `block` (instead of requiring it for the predecessor `block->pprev`)
That way, the check for `block_to_test == block` is also reached when `block_to_test` is the Genesis block.
No longer requiring the data of `block->pprev` also means that we can now prune up to `m_best_block_index` height without requiring a reindex (one block more than before). I added this edge case to `feature_blockfilterindex_prune.py`, the new version should fail on master.
ACKs for top commit:
ryanofsky:
Partial code review ACK 9600ea0145 for the code change, not the test changes. (Test changes are indirect and little over my head.) It seems obvious that previous code `prune_violation = true, while (block->pprev)` would incorrectly detect a prune violation at the genesis block, and the fix here make sense and looks correct.
Tree-SHA512: c717f372cee8fd49718b1b18bfe237aa6ba3ff4468588c10e1272d7a2ef3981d10af4e57de51dec295e2ca72d441bc6c2812f7990011a94d7f818775e3ff1a38
a9256dc340 rpc: output all hash preimages in 'decodepsbt' (Antoine Poinsot)
4d6b5321a5 psbt: implement hash preimages fields (Antoine Poinsot)
Pull request description:
This implements the [bip174 input fields for hash preimages](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki).
One motivation is that we will need those once we implement signing support for Miniscript descriptors.
ACKs for top commit:
achow101:
ACK a9256dc340
Sjors:
re-tACK a9256dc340
w0xlt:
reACK a9256dc
Tree-SHA512: 145d39f7de86256d4174d063dcee217ea6f9ec7a138bbd5205941d17ca99dcccef0ace05f6d0d6a77dd863e3877b05e0752f2bc36ecd8c508e2c8adae2e03ae1
ef81c51f62 build: Use config_opts instead of passing a compiler flag directly (Hennadii Stepanov)
3b8f4e347b build, qt: Drop unused CROSS_COMPILE variable for Android builds (Hennadii Stepanov)
e5a81e65d7 build, refactor: Re-use `qt_lib_suffix` variable (Hennadii Stepanov)
Pull request description:
This PR suggests small cleanups as follow ups of bitcoin/bitcoin#23489.
ACKs for top commit:
fanquake:
ACK ef81c51f62
Tree-SHA512: c834c4f76218c7d567223c49bb86404b02facf61bc9be9a66a984ddd2b2f5e6150ba97071c993ea1faf4da10c87e81df292b3628d4407afccbe9df379ac452d7
Since Qt 5.14.0 the CROSS_COMPILE variable always being overridden by
the Qt build system internally.
See upstream commit: c28b881c98fadcd3415370fad2525b558f6b03e4
It would make for sense for the TODO to be done in PR 17487
(or noted in the review feedback for a follow-up),
no need to continue maintaining the TODO in the codebase.
d5cab1a96d Add createmultisig and addmultisigaddress warnings release note (Samuel Dobson)
e46fc935aa Add warnings field to addmultisigaddress to warn about uncompressed keys (Samuel Dobson)
d1a9742623 Add warnings field to createmultisig to warn about uncompressed keys (Samuel Dobson)
Pull request description:
Fixes#21368
Currently, if there are any uncompressed keys when calling `AddAndGetMultisigDestination`, it will just default to a legacy address regardless of the chosen `address_type`. Rather than keeping this silent behaviour which may be confusing to users, we explicitly add a `warnings` field which will warn the user why their address format is different.
ACKs for top commit:
achow101:
ACK d5cab1a96d
Tree-SHA512: c2ac7f7689251bd4fcd8c26506f053921fbaf34c7a26a74e82ebc7f82cc0bd25407fd7954bf98365dcafa51fa45dcdbee6214320580ca69509690c3555e71cc0
0bfb9208df test: fix test failures in test/functional/p2p_timeouts.py (Jon Atack)
Pull request description:
Fixes #23739.
Top commit has no ACKs.
Tree-SHA512: 0173be87d673dd34fe8ebe77789f85a469fde75a81d5d5bf09b5586c7a1092658bcb056ff5ea4b21ba22c99aba06592815734d68f928c131bfa824f3c1b3c2e6
2f97c1180b doc: Remove TODO 'exclude peers with download permission' (Douglas Chimento)
Pull request description:
Following from PR https://github.com/bitcoin/bitcoin/pull/23109
The [TODO](https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp#L2872) is no longer necessary.
Removing it to prevent future confusion.
Top commit has no ACKs.
Tree-SHA512: c2f4c3eae951d13d623e1b4bd9315804ec33473e501367f89edae80fa446674edc71549e145f058fe7126b8588790dc895d6ea3dfb4347ceeca61f5e5f2f95cc
fa72dd314f fuzz: Move ISO8601 to one place (MarcoFalke)
Pull request description:
Seems confusing to split this to two places.
Also fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=42178
ACKs for top commit:
fanquake:
ACK fa72dd314f
Tree-SHA512: 637b0671078848ea417fdf66b92715602040fad34d4ca5f7b843a519a1cfeebe5d992a79a399deba39926905125681d66ab0dc05f66f79a26f3bf555e12fb0ba
81521173ba Merge global xpubs in joinpsbts and combinepsbts (Andrew Chow)
d8043ddf64 Add global xpub test vectors from BIP (Andrew Chow)
35670df866 Add global_xpubs to decodepsbt (Andrew Chow)
903848562e Implement serializations for PSBT_GLOBAL_XPUB (Andrew Chow)
c5c63b8e4f Implement operator< for KeyOriginInfo and CExtPubKey (Andrew Chow)
d3dbb16168 Separate individual HD Keypath serialization into separate functions (Andrew Chow)
a69332fd89 Store version bytes and be able to serialize them in CExtPubKey (Andrew Chow)
5fdaf6a2ad moveonly: Move (Un)Serialize(To/From)Vector, (De)SerializeHDKeypaths to psbt module (Andrew Chow)
94065cc6c5 Test for proprietary field (Andrew Chow)
a4cf810174 Output proprietary type info in decodepsbt (Andrew Chow)
aebe758e54 Implement PSBT proprietary type (Andrew Chow)
10ba0b593d Output psbt version in decodepsbt (Andrew Chow)
df84fa99c5 Add GetVersion helper to PSBT (Andrew Chow)
c3eb416b88 Implement PSBT versions (Andrew Chow)
3235847473 Types are compact size uints (Andrew Chow)
Pull request description:
Implements the changes to BIP 174 proposed in https://github.com/bitcoin/bips/pull/849 and https://github.com/bitcoin/bips/pull/784
Implements `PSBT_GLOBAL_VERSION`, `PSBT_GLOBAL_PROPRIETARY`, `PSBT_IN_PROPRIETARY`, `PSBT_OUT_PROPRIETARY`, and `PSBT_GLOBAL_XPUB`. The `PSBT_GLOBAL_XPUB` changes are merged in from #16463.
Also includes the test vectors added to BIP 174 for these fields.
A number of additional changes to keypath and xpub serialization are made to support `PSBT_GLOBAL_XPUB`.
ACKs for top commit:
laanwj:
Code review ACK 81521173ba
Tree-SHA512: bd71c3f26030fc23824e76a30d3d346a753e1db224ecee163d6813348feb52d3f4cf4e739a4699e2cff381197ce2a7ea4a92a054f2c3e1db579e91e92a0945e0
Use the `LOG_TIME_MILLIS_WITH_CATEGORY_MSG_ONCE` macro to improve the
logging of snapshot persistance and no longer manually track the duration.
before
[snapshot] flushing coins cache (0 MB)... done (0.00ms)
[snapshot] flushing snapshot chainstate to disk (0 MB)... done (0.00ms)
after
FlushSnapshotToDisk: flushing coins cache (0 MB) started
FlushSnapshotToDisk: completed (0.00ms)
FlushSnapshotToDisk: saving snapshot chainstate (0 MB) started
FlushSnapshotToDisk: completed (0.00ms)
The logging can be observed in the output of
./src/test/test_bitcoin -t validation_chainstate_tests -- DEBUG_LOG_OUT
7f15eff2dd style-only: Remove redundant scope in *Chainstate (Carl Dong)
89bec827fd Collapse the 2 cs_main locks in LoadChainstate (Carl Dong)
3b1584b794 Remove all #include // for * comments (Carl Dong)
9a5a5a3d08 test/setup: Use LoadChainstate (Carl Dong)
c541da0d62 node/chainstate: Add options for in-memory DBs (Carl Dong)
ceb9790341 node/caches: Remove intermediate variables (Carl Dong)
ac4bf138b8 node/caches: Extract cache calculation logic (Carl Dong)
15f2e33bb3 validation: VerifyDB only needs Consensus::Params (Carl Dong)
4da9c076d1 node/chainstate: Decouple from ShutdownRequested (Carl Dong)
05441c2dc5 node/chainstate: Decouple from GetTime (Carl Dong)
2414ebc18b init: Delay RPC block notif until warmup finished (Carl Dong)
8d466a8504 Move -checkblocks LogPrintf to AppInitMain (Carl Dong)
aad8d59789 node/chainstate: Reduce coupling of LogPrintf (Carl Dong)
b345979a2b node/chainstate: Decouple from concept of uiInterface (Carl Dong)
ca7c0b934d Split off VerifyLoadedChainstate (Carl Dong)
adf4912d77 node/chainstate: Remove do/while loop (Carl Dong)
975235ca0a Move init logistics message for BAD_GENESIS_BLOCK to init.cpp (Carl Dong)
8715658983 Move mempool nullptr Assert out of LoadChainstate (Carl Dong)
9162a4f93e node/chainstate: Decouple from concept of NodeContext (Carl Dong)
c7a5c46e6f node/chainstate: Decouple from ArgsManager (Carl Dong)
ae9121f958 node/chainstate: Decouple from stringy errors (Carl Dong)
cbac28b72f node/chainstate: Decouple from GetTimeMillis (Carl Dong)
cb64af9635 node: Extract chainstate loading sequence (Carl Dong)
Pull request description:
This PR:
1. Coalesce the Chainstate loading sequence between `AppInitMain` and `*TestingSetup` (which makes it more tested)
2. Makes the Chainstate loading sequence reusable in preparation for future work extracting out our consensus engine.
Code-wise, this PR:
1. Extracts `AppInitMain`'s Chainstate loading sequence into a `::LoadChainstateSequence` function
2. Makes this `::LoadChainstateSequence` function reusable by
1. Decoupling it from various concepts (`ArgsManager`, `uiInterface`, etc)
2. Making it report errors using an `enum` rather than by setting a `bilingual_str`
3. Makes `*TestingSetup` use this new `::LoadChainstateSequence`
Reviewers: Aside from commentary, I've also included `git diff` flags of interest in the commit messages which I hope will aid review!
ACKs for top commit:
ryanofsky:
Code review ACK 7f15eff2dd. Thanks for updates!
MarcoFalke:
review ACK 7f15eff2dd💳
Tree-SHA512: fb9a6cbd1c511a52b477c62a5e68e53a8be5dec2fff0e44a279966afb91efbab44bf1fe7c6b1519f8464ecc25f42dd4bae8e1efbf55ee91fc90fa0b92e3a83e2
CExtPubKey does not store the version bytes for the extended public key.
We store these so that a CExtPubKey can be serialized and deserialized with
the same version bytes.
SerializeToVector, UnserializeFromVector, DeserializeHDKeypaths, and SerializeHDKeypaths
were in sign.h where PSBT was originally implemented. Since all of the PSBT serialization
has moved to its own file, these functions should follow.
