faf8be7c32 test: Disable known broken USDT test (MarcoFalke)
Pull request description:
The failure is known and running into more failures doesn't help anyone. Not disabling the test would be a waste of CPU and developer time.
https://github.com/bitcoin/bitcoin/issues/27380
Top commit has no ACKs.
Tree-SHA512: d0469153b00d6b30e10a21bcd52d508fcf9f796ff2468f59aff75020a82c718bcae85caf4b58397dea6fd9e210b501353fd51567f979c6b57d3b1bb23d318216
8f6f0d81ee guix: backport glibc patch to prevent redundant librt link (fanquake)
e14473299c contrib: remove librt from release deps (fanquake)
e64e17830a build: remove check for gettimeofday & librt (fanquake)
Pull request description:
Our release binaries currently have a runtime dependency on `librt`. However this is redundant, and only the case due to a bug in glibc. The `clock_*` suit of funcs were absorbed into libc long ago, however an issue with compatibility code meant that librt would still be linked against / used redundantly:
> But the forwarders were not marked as compatibility symbols.
> As a result, on older architectures, historic configure checks such as
> AC_CHECK_LIB(rt, clock_gettime)
> still cause linking against librt, even though this is completely
> unnecessary. It also creates a needless porting hazard because
> architectures behave differently when it comes to symbol availability.
This PR drops our configure check for librt (which is redundant, and could be PR'd standalone), and backports [the relevant patch](https://sourceware.org/git/?p=glibc.git;a=commit;h=f289e656ec8221756519a601042bc9fbe1b310fb) into our glibc, so we can drop librt from our runtime dependencies.
Guix Build:
```bash
67078bddd5dc32801b8c916c3bc12f1404da572312f0158a89b9603c1f753969 guix-build-8f6f0d81ee3a/output/aarch64-linux-gnu/SHA256SUMS.part
794dd00009860fd67d7e51463ee1c5ea9677dfff1c739dd0b91cf73136deb655 guix-build-8f6f0d81ee3a/output/aarch64-linux-gnu/bitcoin-8f6f0d81ee3a-aarch64-linux-gnu-debug.tar.gz
eb9cf3f472ffbc37446fe4d80fe81dc62cf1c28c4d57dd8a7b7176e65487aeeb guix-build-8f6f0d81ee3a/output/aarch64-linux-gnu/bitcoin-8f6f0d81ee3a-aarch64-linux-gnu.tar.gz
e775a9e9b23be44b5c7e7121e88124746836d5bdeda1cd9ba693080d9f3a52a8 guix-build-8f6f0d81ee3a/output/arm-linux-gnueabihf/SHA256SUMS.part
8289f0770333d800e414747026c0fb105d95f389f6c8d901c1041cc65272fb02 guix-build-8f6f0d81ee3a/output/arm-linux-gnueabihf/bitcoin-8f6f0d81ee3a-arm-linux-gnueabihf-debug.tar.gz
e40256c5fb1b9a137845a50fc051f92c3e4cc013b0875a71c62af32f7024af9d guix-build-8f6f0d81ee3a/output/arm-linux-gnueabihf/bitcoin-8f6f0d81ee3a-arm-linux-gnueabihf.tar.gz
c8db222e54e78b27a8a5d3a373a9bbafa51ed29a1fda5c19e8b0eac819b002f2 guix-build-8f6f0d81ee3a/output/arm64-apple-darwin/SHA256SUMS.part
52d4063af628467605fcf533205705b38237a0cc60cafbec224ca8cf4a644738 guix-build-8f6f0d81ee3a/output/arm64-apple-darwin/bitcoin-8f6f0d81ee3a-arm64-apple-darwin-unsigned.dmg
103d80180a9f38e7c903d0b6581e4bb5130c640fac1fd5019eee7fa90e303c1d guix-build-8f6f0d81ee3a/output/arm64-apple-darwin/bitcoin-8f6f0d81ee3a-arm64-apple-darwin-unsigned.tar.gz
a8f0a89c4d4b1d05e6ea968dde3b13368999dfc1c3ea765e81fd3c4db46197b3 guix-build-8f6f0d81ee3a/output/arm64-apple-darwin/bitcoin-8f6f0d81ee3a-arm64-apple-darwin.tar.gz
726d2671bbed2355c083b8516faa5d8e0422fab6cb38a135f68ee011f9e09af5 guix-build-8f6f0d81ee3a/output/dist-archive/bitcoin-8f6f0d81ee3a.tar.gz
955fff1c9998bb04bcf1afe9b467590960206e9c512b3446ecdd701e251bb419 guix-build-8f6f0d81ee3a/output/powerpc64-linux-gnu/SHA256SUMS.part
e95cdeda727d641c002755c4a3e3b69049a35f1bff4867ac14320585d65595c4 guix-build-8f6f0d81ee3a/output/powerpc64-linux-gnu/bitcoin-8f6f0d81ee3a-powerpc64-linux-gnu-debug.tar.gz
21bda341cd8af44bc731cf7e3637322a92032e7a956acdde25ea6e59989c67b9 guix-build-8f6f0d81ee3a/output/powerpc64-linux-gnu/bitcoin-8f6f0d81ee3a-powerpc64-linux-gnu.tar.gz
6f90c38998696f61c373c3546bcc03e6b5ecfbe3b9fec9a7c75d601b3175b698 guix-build-8f6f0d81ee3a/output/powerpc64le-linux-gnu/SHA256SUMS.part
7166c2354b8777464bf8c5c3d7e4a171d00b5e0617635fa8b12c4d47ad619e84 guix-build-8f6f0d81ee3a/output/powerpc64le-linux-gnu/bitcoin-8f6f0d81ee3a-powerpc64le-linux-gnu-debug.tar.gz
8c879a3ae9fefc1071d0b6ea3b0cf858295386860b10079b472b526abfdcd2b5 guix-build-8f6f0d81ee3a/output/powerpc64le-linux-gnu/bitcoin-8f6f0d81ee3a-powerpc64le-linux-gnu.tar.gz
7dc7153d3c180308d873cb20320e8a6221cec81d8018da85683870168380eef7 guix-build-8f6f0d81ee3a/output/riscv64-linux-gnu/SHA256SUMS.part
c37b79e33b9a318d3acee9114cdf057ee518abaa09736bd63e015d924d2c3ffb guix-build-8f6f0d81ee3a/output/riscv64-linux-gnu/bitcoin-8f6f0d81ee3a-riscv64-linux-gnu-debug.tar.gz
d25abfb09d12e74bffd7f42e95eba211317acefa4718dbea27055d905f5b6999 guix-build-8f6f0d81ee3a/output/riscv64-linux-gnu/bitcoin-8f6f0d81ee3a-riscv64-linux-gnu.tar.gz
5ffc5c97012d8ae85cb56e635760029b774ea4f57a64e41cd4bdade4ed93e619 guix-build-8f6f0d81ee3a/output/x86_64-apple-darwin/SHA256SUMS.part
ecf96275016e82af2c1a4842578feac286de9db8b7f5e4266cf877cb29da1da8 guix-build-8f6f0d81ee3a/output/x86_64-apple-darwin/bitcoin-8f6f0d81ee3a-x86_64-apple-darwin-unsigned.dmg
50bee378ed88471dc326730564ca24cea2625ce1477b82881cda572f0a8913cc guix-build-8f6f0d81ee3a/output/x86_64-apple-darwin/bitcoin-8f6f0d81ee3a-x86_64-apple-darwin-unsigned.tar.gz
f4215a018f18e3639c50f10909af3ceff6982abf8b292fd88fa5d690b06d704a guix-build-8f6f0d81ee3a/output/x86_64-apple-darwin/bitcoin-8f6f0d81ee3a-x86_64-apple-darwin.tar.gz
ee5278c8afc7ead80853aff69c1bbd624ef078428076f0e92b0ad35931036b3f guix-build-8f6f0d81ee3a/output/x86_64-linux-gnu/SHA256SUMS.part
daed3889107ffe8b3ec2c59abff93d4b92a4dff382457485d29489a0e9421965 guix-build-8f6f0d81ee3a/output/x86_64-linux-gnu/bitcoin-8f6f0d81ee3a-x86_64-linux-gnu-debug.tar.gz
f1acd6b1d296f2de5ff838fe3fb82035f2774485b06678ecdd461e631ebbe092 guix-build-8f6f0d81ee3a/output/x86_64-linux-gnu/bitcoin-8f6f0d81ee3a-x86_64-linux-gnu.tar.gz
3e9f9f92e4de995c9029f17962c33e317f7000df9c1afa2a447b65ac98c27f4b guix-build-8f6f0d81ee3a/output/x86_64-w64-mingw32/SHA256SUMS.part
4b50a73917450770c793bfc787a6785c7389ce02bd25368db9a1445da07bb7b1 guix-build-8f6f0d81ee3a/output/x86_64-w64-mingw32/bitcoin-8f6f0d81ee3a-win64-debug.zip
832ddec19b8c5698cc3497f93fc59f0f72b0d7a3f313d46c2c1c52b5badf19fd guix-build-8f6f0d81ee3a/output/x86_64-w64-mingw32/bitcoin-8f6f0d81ee3a-win64-setup-unsigned.exe
d9bc2dabd0cff8e9ee6ccb309bee34a6faa1298771c0cc9bff8f948d34ec047e guix-build-8f6f0d81ee3a/output/x86_64-w64-mingw32/bitcoin-8f6f0d81ee3a-win64-unsigned.tar.gz
55cc5607d3fdf113fde463d87c5dd895c305ba0313e56bba1b0875a8a78c65a7 guix-build-8f6f0d81ee3a/output/x86_64-w64-mingw32/bitcoin-8f6f0d81ee3a-win64.zip
```
ACKs for top commit:
hebasto:
ACK 8f6f0d81ee
Tree-SHA512: f6fd4b9ed37ad93c7a5df4ca17f1ae5b8705f5dc4a377c8e01c6376b1818980534a233a08f2a20c4ff851a25f660ebb89c7416b93f6f039747194661b00c75ed
fa6dfaaf45 scripted-diff: Use new FUZZ_TARGET macro everywhere (MarcoFalke)
fa36ad8b09 fuzz: Accept options in FUZZ_TARGET macro (MarcoFalke)
Pull request description:
The `FUZZ_TARGET` macros have many issues:
* The developer will have to pick the right macro to pass the wanted option.
* Adding a new option requires doubling the number of existing macros in the worst case.
Fix all issues by using only a single macro.
This refactor does not change behavior.
ACKs for top commit:
dergoegge:
ACK fa6dfaaf45
Tree-SHA512: 49a34553867a1734ce89e616b2d7c29b784a67cd8990db6573f0c7b18957636ef0c81d3d0d444a04c12cdc98bc4c4aa7a2ec94e6232dc363620a746e28416444
05ef059a33 doc: update windows -fstack-clash-protection doc (fanquake)
Pull request description:
Now that changes have been made in GCC, to fix the build failures.
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458.
ACKs for top commit:
TheCharlatan:
ACK 05ef059a33
hebasto:
ACK 05ef059a33, I've verified that the fix commit is present in all branches starting from `gcc-11`.
Tree-SHA512: 96b79d65b46e6b9d939c8e6079e984da86987503210106d5155dbe5a6fd82d56d9983694656e27156b01bab795c766b85fc60c799813bc676bba5f3b73f9be22
4da243ba02 qt: show own outputs on PSBT signing window (Hernan Marino)
Pull request description:
This fixes https://github.com/bitcoin-core/gui/issues/732 .
It allows you to identify your own addresses in the outputs of a transaction in the PSBT signing window. This enables easy identification of change outputs, and prevents certain attacks where someone (co-signers of a multisig, or others ) might trick you into signing a transaction while they are stealing the change, since prior to this modification there was no easy way of knowing this.
The identification of the output is similar to the way this is done in the transaction details window.
A sample output is :
ACKs for top commit:
achow101:
ACK 4da243ba02
jarolrod:
ACK 4da243ba02
Tree-SHA512: fa9901d2acc84472c11afcd0a59a859db598cdf5cea755b492178d3e7434b70d9bd8f554928938a2ff9920c8f397fef814ce14b416556c30fba0c3c1f62cd722
31eca93a9e kernel: Remove StartShutdown calls from validation code (Ryan Ofsky)
Pull request description:
This change drops the last kernel dependency on shutdown.cpp. It also adds new hooks for libbitcoinkernel applications to be able to interrupt kernel operations when the chain tip changes.
This change is mostly a refactoring, but does slightly change `-stopatheight` behavior (see release note and commit message)
ACKs for top commit:
TheCharlatan:
ACK 31eca93a9e
furszy:
Concept and light review ACK 31eca93a
hebasto:
ACK 31eca93a9e, I have reviewed the code and it looks OK.
MarcoFalke:
lgtm ACK 31eca93a9e 🕷
Tree-SHA512: e26928436bcde658e842b1f92e9c24b1ce91031fb63b41aafccf3130bfff532b75338a269a2bb7558bff2973913f17b97a00fec3e7e0588e2ce44de097142047
fa4ccf1511 ci: Add missing -O2 to valgrind tasks (MarcoFalke)
Pull request description:
Currently the tasks have nothing (`-O0`) set, which makes them slow.
Fix this by falling back to the default (`-O2`).
ACKs for top commit:
recursive-rat4:
utACK fa4ccf1511
dergoegge:
utACK fa4ccf1511
Tree-SHA512: 44d803000d883cfa534f2c76d793d7d7f840e114fc377d20fc36d008b471d69ec9f0170358ed1f3567d49e3ff63682244062c954cd0b963df31ca39c08d2d5b9
35a2175ad8 fuzz: addrman, add coverage for `network` field in `Select()`, `Size()` and `GetAddr()` (brunoerg)
Pull request description:
This PR adds fuzz coverage for `network` field in `Select()`, `Size()` and `GetAddr()`, there was only call to them without passing a network.
https://marcofalke.github.io/b-c-cov/fuzz.coverage/src/addrman.cpp.gcov.html
ACKs for top commit:
amitiuttarwar:
for the record, ACK 35a2175ad8 - only small changes from the version (previously) proposed in 27213
achow101:
ACK 35a2175ad8
mzumsande:
Code Review ACK 35a2175ad8, haven't tested this yet, but I will let the fuzzer run for a while now.
Tree-SHA512: dddb8322298d6c373c8e68d57538470b11825a9a310a355828c351d5c0b19ff6779d024a800e3ea90126d0c050e86f71fd22cd23d1a306c784cef0f82c45e3ca
e7cf8657e1 test: add unit test for local address advertising (Martin Zumsande)
f4754b9dfb net: restrict self-advertisements with privacy networks (Martin Zumsande)
e4d541c7cf net, refactor: pass reference for peer address in GetReachabilityFrom (Martin Zumsande)
62d73f5370 net, refactor: pass CNode instead of CNetAddr to GetLocalAddress (Martin Zumsande)
Pull request description:
The current logic for self-advertisements works such that we detect as many local addresses as we can, and then, using the scoring matrix from `CNetAddr::GetReachabilityFrom()`, self-advertise with the address that fits best to our peer.
It is in general not hard for our peers to distinguish our self-advertisements from other addrs we send them, because we self-advertise every ~24h and because the first addr we send over a connection is likely our self-advertisement.
`GetReachabilityFrom()` currently only takes into account actual reachability, but not whether we'd _want_ to announce our identity for one network to peers from other networks, which is not straightforward in connection with privacy networks.
While the general approach is to prefer self-advertising with the address for the network our peer is on, there are several special situations in which we don't have one, and as a result could allow self-advertise other local addresses, for example:
A) We run i2p and clearnet, use `-i2pacceptincoming=0` (so we have no local i2p address), and we have a local ipv4 address. In this case, we'd advertise the ipv4 address to our outbound i2p peers.
B) Our `-discover` logic cannot detect any local clearnet addresses in our network environment, but we are actually reachable over clearnet. If we ran bitcoind clearnet-only, we'd always advertise the address our peer sees us with instead, and could get inbound peers this way. Now, if we also have an onion service running (but aren't using tor as a proxy for clearnet connections), we could advertise our onion address to clearnet peers, so that they would be able to connect our clearnet and onion identities.
This PR tries to avoid these situations by
1.) never advertising our local Tor or I2P address to peers from other networks.
2.) never advertising local addresses from non-anonymity networks to peers from Tor or I2P
Note that this affects only our own self-advertisements, the rules to forward other people's addrs are not changed.
[Edit] after Initial [discussion](https://github.com/bitcoin/bitcoin/pull/27411#issuecomment-1497176155): CJDNS is not being treated like Tor and I2P at least for now, because it has different privacy properties and for the practical reason that it has still very few bitcoin nodes.
ACKs for top commit:
achow101:
ACK e7cf8657e1
vasild:
ACK e7cf8657e1
luke-jr:
utACK e7cf8657e1
Tree-SHA512: 3db8415dea6f82223d11a23bd6cbb3b8cf68831321280e926034a1f110cbe22562570013925f6fa20d8f08e41d0202fd69c733d9f16217318a660d2a1a21b795
0bf87476f5 test: add ChaCha20 test triggering 32-bit block counter overflow (Sebastian Falbesoner)
7f2a985147 tests: improve ChaCha20 unit tests (Pieter Wuille)
511a8d406e crypto: Implement RFC8439-compatible variant of ChaCha20 (Pieter Wuille)
Pull request description:
Based on and replaces part of #25361, part of the BIP324 project (#27634). See also #19225 for background.
There are two variants of ChaCha20 in use. The currently implemented one uses a 64-bit nonce and a 64-bit block counter, while the one used in RFC8439 (and thus BIP324) uses a 96-bit nonce and 32-bit block counter. This PR changes the logic to use the 96-bit nonce variant, though in a way that's compatible with >256 GiB output (by automatically incrementing the first 32-bit part of the nonce when the block counter overflows).
For those who reviewed the original PR, the biggest change is here that the 96-bit nonce is passed as a Nonce96 type (pair of 32-bit + 64-bit integer) rather than a 12-byte array.
ACKs for top commit:
achow101:
ACK 0bf87476f5
theStack:
Code-review ACK 0bf87476f5
Tree-SHA512: 62e4cbd5388b8d50ef1a0dc99b6f4ad36c7b4419032035f8e622dda63a62311dd923032217e20054bcd836865d4be5c074f9e5538ca158f94f08eab75c5519c1
5cf44275c8 test: refactor: deduplicate legacy ECDSA signing for tx inputs (Sebastian Falbesoner)
Pull request description:
There are several instances in functional tests and the framework (MiniWallet, feature_block.py, p2p_segwit.py) where we create a legacy ECDSA signature for a certain transaction's input by doing the following steps:
1. calculate the `LegacySignatureHash` with the desired sighash type
2. create the actual digital signature by calling `ECKey.sign_ecdsa` on the signature message hash calculated above
3. put the DER-encoded result as CScript data push into tx input's scriptSig
Create a new helper `sign_input_legacy` which hides those details and takes only the necessary parameters (tx, input index, relevant scriptPubKey, private key, sighash type [SIGHASH_ALL by default]). For further convenience, the signature is prepended to already existing data-pushes in scriptSig, in order to avoid rehashing the transaction after calling the new signing function.
ACKs for top commit:
dimitaracev:
ACK `5cf4427`
achow101:
ACK 5cf44275c8
pinheadmz:
ACK 5cf44275c8
Tree-SHA512: 8f0e4fb2c3e0f84fac5dbc4dda87973276242b0f628034272a7f3e45434c1e17dd1b26a37edfb302dcaf380dbfe98b0417391ace5e0ac9720155d8fba702031e
This change drops the last kernel dependency on shutdown.cpp. It also adds new
hooks for libbitcoinkernel applications to be able to interrupt kernel
operations when the chain tip changes.
This is a refactoring that does not affect behavior. (Looking at the code it
can appear like the new break statement in the ActivateBestChain function is a
change in behavior, but actually the previous StartShutdown call was indirectly
triggering a break before, because it was causing m_chainman.m_interrupt to be
true. The new code just makes the break more obvious.)
89ba8905f5 test: indexes, fix on error infinite loop (furszy)
Pull request description:
Coming from https://github.com/bitcoin/bitcoin/pull/28036#issuecomment-1623813703, I thought that we were going to fix it there but seems that got merged without it for some reason.
As index sync failures trigger a shutdown request without notifying `BaseIndex::BlockUntilSyncedToCurrentChain` in any way, we also need to check whether a shutdown was requested or not inside 'IndexWaitSynced'.
Otherwise, any error inside the index sync process will hang the test forever.
ACKs for top commit:
MarcoFalke:
lgtm ACK 89ba8905f5
jamesob:
ACK 89ba890
ryanofsky:
Code review ACK 89ba8905f5. Just comment update since last review
Tree-SHA512: 1f6daf34e51d3fbc802799bfa4ac0ef0d8f774db5f9e2f5d35df18a77679778475c94efc3da1fb723ebaf3583e4075e4a5cbe4a5104ad0c50e2b32076e247b29
462390c85f refactor: Move stopafterblockimport handling out of blockstorage (TheCharlatan)
Pull request description:
This has the benefit of moving this StartShutdown call out of the blockstorage file and thus out of the kernel's responsibility. The user can now decide if he wants to start shutdown / interrupt after a block import or not.
This also simplifies https://github.com/bitcoin/bitcoin/pull/28048, making it one fewer shutdown call to handle.
ACKs for top commit:
MarcoFalke:
lgtm ACK 462390c85f 🗝
ryanofsky:
Code review ACK 462390c85f. Just has been rebased and is a simpler change after #27607
Tree-SHA512: 84e58256b1c61f10e7ec5ecf32916f40a2ab1ea7cce703de0fa1c61ee2be94bd45ed32718bc99903b6eff3e6d3d5b506470bf567ddbb444a58232913918e8ab8
This has the benefit of moving the StartShutdown call out of the
blockstorage file and thus out of the kernel's responsibility. The user
can now decide if he wants to start shutdown / interrupt after a block
import or not.
faf902858d test: Check expected_stderr after stop (MarcoFalke)
Pull request description:
This fixes a bug where stderr wasn't checked for the shutdown sequence.
Fix that by waiting for the shutdown to finish and then check stderr.
ACKs for top commit:
theStack:
ACK faf902858d
Tree-SHA512: a70cd1e6cda84d542782e41e8b59741dbcd472c0d0575bcef5cbfd1418473ce94efe921481d557bae3fbbdd78f1c49c09c48872883c052d87c5c9a9a51492692
As index sync failures trigger a shutdown request without notifying
BaseIndex::BlockUntilSyncedToCurrentChain in any way, we also need
to check whether a shutdown was requested or not inside 'IndexWaitSynced'.
Otherwise, any error inside the index sync process will hang the test
forever.
ca91c244ef index: verify blocks data existence only once (furszy)
fcbdaeef4d init: don't start indexes sync thread prematurely (furszy)
2ec89f1970 refactor: simplify pruning violation check (furszy)
c82ef91eae make GetFirstStoredBlock assert that 'start_block' always has data (furszy)
430e7027a1 refactor: index, decouple 'Init' from 'Start' (furszy)
225e213110 refactor: init indexes, decouple 'Start()' from the creation step (furszy)
2ebc7e68cc doc: describe 'init load' thread actions (Martin Zumsande)
04575106b2 scripted-diff: rename 'loadblk' thread name to 'initload' (furszy)
ed4462cc78 init: start indexes sync earlier (furszy)
Pull request description:
Simplifies index startup code, eliminating the `g_indexes_ready_to_sync` variable,
deduplicating code and moving the prune violation check out of the `BaseIndex` class.
Also makes startup more efficient by running the prune violation check once for all indexes
instead of once for each index, and by delaying the prune violation check and moving it off
of the main thread so the node can start up faster and perform the block data availability
verification even when the '-reindex" or the "-reindex-chainstate" flags are enabled (which
hasn't being possible so far).
ACKs for top commit:
ryanofsky:
Code review ACK ca91c244ef. Just rebase and suggested changes since last review (Start return check, and code simplification)
TheCharlatan:
re-ACK ca91c244ef
Tree-SHA512: e9c98ce89aeb29e8d0f505f17b34aa54fe44efefbf017f4746e3b446ab4de25ade4f707254a0bbe4b99b69731b04a4067ce529eb7aa834ced196784b694cf7ce
3e8bf2e10c test: make assumeUTXO test capture the expected fatal error (furszy)
Pull request description:
The test is exercising the error, so it can capture it before the
test framework displays it on the console as an unforeseen
fatal error.
It is odd to observe a fatal error after executing the complete
test suite and seeing it pass successfully.
Reproduction Steps:
Run the unit test suite. A long AssumeUTXO fatal error will be
printed even when all tests pass successfully.
ACKs for top commit:
MarcoFalke:
lgtm ACK 3e8bf2e10c
theStack:
Tested ACK 3e8bf2e10c
TheCharlatan:
ACK 3e8bf2e10c
Tree-SHA512: 820a5a4db52085ed72cbe7eb433b8c0f2d283ac6f5d456bc2b3e3f0305301022b2729e32e5fd9002859e4491ae7ac6de568a4c20557c7b249b0e7694ab8bd177
At present, during init, we traverse the chain (once per index)
to confirm that all necessary blocks to sync each index up to
the current tip are present.
To make the process more efficient, we can fetch the oldest block
from the indexers and perform the chain data existence check from
that point only once.
This also moves the pruning violation check to the end of the
'loadinit' thread, which is where the reindex, block loading and
chain activation processes happen.
Making the node's startup process faster, allowing us to remove
the global g_indexes_ready_to_sync flag, and enabling the
execution of the pruning violation verification even when the
reindex or reindex-chainstate flags are enabled (which has being
skipped so far).
By moving the 'StartIndexes()' call into the 'initload'
thread, we can remove the threads active wait. Optimizing
the available resources.
The only difference with the current state is that now the
indexes threads will only be started when they can process
work and not before it.
By generalizing 'GetFirstStoredBlock' and implementing
'CheckBlockDataAvailability' we can dedup code and
avoid repeating work when multiple indexes are enabled.
E.g. get the oldest block across all indexes and
perform the pruning violation check from that point
up to the tip only once (this feature is being introduced
in a follow-up commit).
This commit shouldn't change behavior in any way.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
And transfer the responsibility of verifying whether 'start_block'
has data or not to the caller.
This is because the 'GetFirstStoredBlock' function responsibility
is to return the first block containing data. And the current
implementation can return 'start_block' when it has no data!. Which
is misleading at least.
Edge case behavior change:
Previously, if the block tip lacked data but all preceding blocks
contained data, there was no prune violation. And now, such
scenario will result in a prune violation.
Verify that our ChaCha20 implementation using the 96/32 split interface
is compatible with >256 GiB outputs by triggering a 32-bit block counter
overflow and checking that the keystream matches one created with an
alternative implementation using a 64/64 split interface with the
corresponding input data. The test case data was generated with the
following Python script using the PyCryptodome library (version 3.15.0):
----------------------------------------------------------------------------------------------
from Crypto.Cipher import ChaCha20
key = bytes(list(range(32))); nonce = 0xdeadbeef12345678; pos = 2**32 - 1
c = ChaCha20.new(key=key, nonce=nonce.to_bytes(8, 'little'))
c.seek(pos * 64); stream = c.encrypt(bytes([0])*128)
print(f"Key: {key.hex()}\nNonce: {hex(nonce)}\nPos: {hex(pos)}\nStream: {stream.hex()}")
----------------------------------------------------------------------------------------------
No behavior change.
The goal here is to group indexes, so we can perform the same
initialization and verification process equally for all of them.
The checks performed inside `StartIndexes` will be expanded
in the subsequent commits.
The thread does not only load blocks, it loads the mempool and,
in a future commit, will start the indexes as well.
Also, renamed the 'ThreadImport' function to 'ImportBlocks'
And the 'm_load_block' class member to 'm_thread_load'.
-BEGIN VERIFY SCRIPT-
sed -i "s/ThreadImport/ImportBlocks/g" $(git grep -l ThreadImport -- ':!/doc/')
sed -i "s/loadblk/initload/g" $(git grep -l loadblk -- ':!/doc/release-notes/')
sed -i "s/m_load_block/m_thread_load/g" $(git grep -l m_load_block)
-END VERIFY SCRIPT-
The mempool load can take a while, and it is not
needed for the indexes' synchronization.
Also, having the mempool load function call
inside 'blockstorage.cpp' wasn't structurally
correct.
There are two variants of ChaCha20 in use. The original one uses a 64-bit
nonce and a 64-bit block counter, while the one used in RFC8439 uses a
96-bit nonce and 32-bit block counter. This commit changes the interface
to use the 96/32 split (but automatically incrementing the first 32-bit
part of the nonce when the 32-bit block counter overflows, so to retain
compatibility with >256 GiB output).
Simultaneously, also merge the SetIV and Seek64 functions, as we almost
always call both anyway.
Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>
8b5397c00e wallet: bdb: include bdb header from our implementation files only (Cory Fields)
6e010626af wallet: bdb: don't use bdb define in header (Cory Fields)
004b184b02 wallet: bdb: move BerkeleyDatabase constructor to cpp file (Cory Fields)
b3582baa3a wallet: bdb: move SafeDbt to cpp file (Cory Fields)
e5e5aa1da2 wallet: bdb: move SpanFromDbt to below SafeDbt's implementation (Cory Fields)
4216f69250 wallet: bdb: move TxnBegin to cpp file since it uses a bdb function (Cory Fields)
43369f3706 wallet: bdb: drop default parameter (Cory Fields)
Pull request description:
Only `#include` upstream bdb headers from our cpp files.
It's generally good practice to avoid including 3rd party deps in headers as otherwise they tend to sneak into new compilation units. IMO this makes for a nice cleanup.
There's a good bit of code movement here, but each commit is small and _should_ be obviously correct.
Note: in the future, the buildsystem can add the bdb include path for `bdb.cpp` and `salvage.cpp` only, rather than all wallet sources.
ACKs for top commit:
achow101:
reACK 8b5397c00e
hebasto:
ACK 8b5397c00e
Tree-SHA512: 0ef6e8a9c4c6e2d1e5d6a3534495f91900e4175143911a5848258c56da54535b85fad67b6d573da5f7b96e7881299b5a8ca2327e708f305b317b9a3e85038d66
7ecc29a0b7 test: wallet, add coverage for addressbook migration (furszy)
a277f8357a wallet: migration bugfix, persist empty labels (furszy)
1b64f6498c wallet: migration bugfix, clone 'send' record label to all wallets (furszy)
Pull request description:
Addressing two specific bugs encountered during the wallet migration process, related to the address book, and improves the test coverage for it.
Bug 1: Non-Cloning of External 'Send' Records
The external 'send' records were not being correctly cloned to all wallets.
Bug 2: Persistence of Empty Labels
As address book entries without associated db label records can be treated as change (the `label` field inside the `CAddressBookData` class is optional, `nullopt` labels make `CAddressBookData ::IsChange()` return true), we must persist empty labels during the migration process.
The user might have called `setlabel` with an "" string for an external address and that must be retained during migration.
ACKs for top commit:
achow101:
ACK 7ecc29a0b7
Tree-SHA512: b8a8483a4178a37c49af11eb7ba8a82ca95e54a6cd799e155e33f9fbe7f37b259e28372c77d6944d46b6765f9eaca6b8ca8d1cdd9d223120a3653e4e41d0b6b7
