33e2b82a4f wallet, bench: Remove unused database options from WalletBenchLoading (Andrew Chow)
80ace042d8 tests: Modify records directly in wallet ckey loading test (Andrew Chow)
b3bb17d5d0 tests: Update DuplicateMockDatabase for MockableDatabase (Andrew Chow)
f0eecf5e40 scripted-diff: Replace CreateMockWalletDB with CreateMockableWalletDB (Andrew Chow)
075962bc25 wallet, tests: Include wallet/test/util.h (Andrew Chow)
14aa4cb1e4 wallet: Move DummyDatabase to salvage (Andrew Chow)
f67a385556 wallet, tests: Replace usage of dummy db with mockable db (Andrew Chow)
33c6245ac1 Introduce MockableDatabase for wallet unit tests (Andrew Chow)
Pull request description:
For the wallet's unit tests, we currently use either `DummyDatabase` or memory-only versions of either BDB or SQLite. The tests that use `DummyDatabase` just need a `WalletDatabase` so that the `CWallet` can be constructed, while the tests using the memory-only databases just need a backing data store. There is also a `FailDatabase` that is similar to `DummyDatabase` except it fails be default or can have a configured return value. Having all of these different database types can make it difficult to write tests, particularly tests that work when either BDB or SQLite is disabled.
This PR unifies all of these different unit test database classes into a single `MockableDatabase`. Like `DummyDatabase`, most functions do nothing and just return true. Like `FailDatabase`, the return value of some functions can be configured on the fly to test various failure cases. Like the memory-only databases, records can actually be written to the `MockableDatabase` and be retrieved later, but all of this is still held in memory. Using `MockableDatabase` completely removes the need for having BDB or SQLite backed wallets in the unit tests for the tests that are not actually testing specific database behaviors.
Because `MockableDatabase`s can be created by each unit test, we can also control what records are stored in the database. Records can be added and removed externally from the typical database modification functions. This will give us greater ability to test failure conditions, particularly those involving corrupted records.
Possible alternative to #26644
ACKs for top commit:
furszy:
ACK 33e2b82
TheCharlatan:
ACK 33e2b82a4f
Tree-SHA512: c2b09eff9728d063d2d4aea28a0f0e64e40b76483e75dc53f08667df23bd25834d52656cd4eafb02e552db0b9e619cfdb1b1c65b26b5436ee2c971d804768bcc
3ece0ebf62 build, doc: Adjust comment after PR27254 (Hennadii Stepanov)
Pull request description:
This is a follow up for https://github.com/bitcoin/bitcoin/pull/27254.
ACKs for top commit:
TheCharlatan:
ACK 3ece0ebf62
Tree-SHA512: 36c627524304f0ea964383488acb9b16d0b553cc9cf3bce7a12aec65a7905b13e4582b7b7ec6d1efa32cd3c623969bc00f805ff31d9e6eec86a614e75796c8ff
fa01c3c59c ci: Remove CI_EXEC bloat (MarcoFalke)
fa8a428c92 move-only: Move almost all CI_EXEC code to 06_script_b.sh (MarcoFalke)
Pull request description:
`CI_EXEC` has many issues:
* It is roughly equivalent to `bash -c "$*"`, meaning that the full command will be treated as a single string, ignoring tokens.
* It must be put in front of (almost) every command, making it easy to forget, hard to debug the resulting failure, and the code verbose.
Fix all issues by removing it almost completely.
ACKs for top commit:
TheCharlatan:
ACK fa01c3c59c
Tree-SHA512: 4a65d61f5c35ca945d31f270dba3e96305fd83333a7713f0452c67f02a78e1901113e9f18d21e1dc016403c0033eb32038a9308d0a0ded7ee6b970d18381a1c2
Test the case of a tx being conflicted by multiple
txs with different depths. The conflicted tx is also spent by
a child tx for which confirmation status is tied to the parent's.
After a reorg of conflicting txs, the conflicted status should be
undone properly.
Co-authored-by: furszy <mfurszy@protonmail.com>
In `blockDisconnected`, for each transaction in the block, look
for any wallet transactions spending the same inputs. If any of
these transactions were marked conflicted, they are now marked as
inactive.
Co-authored-by: ariard <antoine.riard@gmail.com>
9ae854da19 depends: no-longer nuke libc++abi.so* in native_clang package (fanquake)
Pull request description:
We weren't copying it over in any case.
ACKs for top commit:
hebasto:
ACK 9ae854da19
theuni:
Sure. utACK no-op 9ae854da19.
Tree-SHA512: 3cc7f18f27c1b498f930bc1a09283aa04ba673d3c1a5220d8462213f0a06b74bc34989f23404402850de518cba35ddab900a54f7f0fac112fc86664e4155f8cb
67aacc73ea build: cleanup comments after adding yet another libtool hack (Cory Fields)
283d95516a build: Fix shared lib linking for darwin with lld (Cory Fields)
Pull request description:
Solves one of the last remaining blockers for #21778. Fixes lld linking shared libs for macos via libtool.
lld fails one of libtool's earliest checks [because it happens to output a warning that contains a specific string](https://git.savannah.gnu.org/cgit/libtool.git/tree/m4/libtool.m4#n999):
> # If there is a non-empty error log, and "single_module"
> # appears in it, assume the flag caused a linker warning
And here is the test being run:
> x86_64-apple-darwin-ld: warning: Option `-single_module' is deprecated in ld64:
> x86_64-apple-darwin-ld: warning: Unnecessary option: this is already the default
Because the warning is printed the test fails. So libtool falls back to a very primitive and broken link-line for shared libs.
Arguably this should be worked-around in upstream lld by changing the warning string, as otherwise every libtool project will fail to link with it.
Like many other libtool hacks, the solution is to simply disable the check and hard-code the answer we know to be correct.
ACKs for top commit:
hebasto:
re-ACK 67aacc73ea
Tree-SHA512: 792e4d208a3a4921edb5f267f43ecd052b5b650df0db5cb2788ee1e4f3c4087413f354b22e407ff5fa2f99a22a16154ec6826d14c6654a57c00aae3b3e744bca
libtool gets a false-positive from the warning produced by lld -single_module
because it is already the default and unneeded.
Skip the check unconditionally for Darwin linkers.
308caf326d doc: remove version number from bips.md (fanquake)
Pull request description:
This always just needs "bumping" (see previous rc type pulls), and the version number is already whichever version of the code you acquired bips.md with.
ACKs for top commit:
MarcoFalke:
lgtm ACK 308caf326d
achow101:
ACK 308caf326d
theStack:
ACK 308caf326d
hebasto:
ACK 308caf326d
Tree-SHA512: fcb98e7cdc0c1f8960bfba86be09c2badb36b613060fae394a56e1561c69d28f433434f573c8b1ae1d71ae326277dea2a4841d5c08ad39f8e8848300743146e7
5b3406094f net_processing: Boost inv trickle rate (Anthony Towns)
228e9201ef txmempool: have CompareDepthAndScore sort missing txs first (Anthony Towns)
Pull request description:
Couple of performance improvements when draining the inventory-to-send queue:
* drop txs that have already been evicted from the mempool (or included in a block) immediately, rather than at the end of processing
* marginally increase outgoing trickle rate during spikes in tx volume
ACKs for top commit:
willcl-ark:
ACK 5b34060
instagibbs:
ACK 5b3406094f
darosior:
utACK 5b3406094f
glozow:
code review ACK 5b3406094f
dergoegge:
utACK 5b3406094f
Tree-SHA512: 155cd3b5d150ba3417c1cd126f2be734497742e85358a19c9d365f4f97c555ff9e846405bbeada13c3575b3713c3a7eb2f780879a828cbbf032ad9a6e5416b30
5ff63a09a9 refactor, blockstorage: Replace stopafterblockimport arg (TheCharlatan)
18e5ba7c80 refactor, blockstorage: Replace blocksdir arg (TheCharlatan)
02a0899527 refactor, BlockManager: Replace fastprune from arg with options (TheCharlatan)
a498d699e3 refactor/iwyu: Complete includes for blockmanager_args (TheCharlatan)
f0bb1021f0 refactor: Move functions to BlockManager methods (TheCharlatan)
cfbb212493 zmq: Pass lambda to zmq's ZMQPublishRawBlockNotifier (TheCharlatan)
8ed4ff8e05 refactor: Declare g_zmq_notification_interface as unique_ptr (TheCharlatan)
Pull request description:
The libbitcoin_kernel library should not rely on the `ArgsManager`, but rather use option structs that can be passed to the various classes it uses. This PR removes reliance on the `ArgsManager` from the `blockstorage.*` files. Like similar prior work, it uses the options struct in the `BlockManager` that can be populated with `ArgsManager` values.
Some related prior work: https://github.com/bitcoin/bitcoin/pull/26889https://github.com/bitcoin/bitcoin/pull/25862https://github.com/bitcoin/bitcoin/pull/25527https://github.com/bitcoin/bitcoin/pull/25487
Related PR removing blockstorage globals: https://github.com/bitcoin/bitcoin/pull/25781
ACKs for top commit:
ryanofsky:
Code review ACK 5ff63a09a9. Since last ACK just added std::move and fixed commit title. Sorry for the noise!
mzumsande:
Code Review ACK 5ff63a09a9
Tree-SHA512: 4bde8fd140a40b97eca923e9016d85dcea6fad6fd199731f158376294af59c3e8b163a0725aa47b4be3519b61828044e0a042deea005e0c28de21d8b6c3e1ea7
72efc26439 util: improve streams.h:FindByte() performance (Larry Ruane)
604df63f6c [bench] add streams findbyte (gzhao408)
Pull request description:
This PR is strictly a performance improvement; there is no functional change. The `CBufferedFile::FindByte()` method searches for the next occurrence of the given byte in the file. Currently, this is done by explicitly inspecting each byte in turn. This PR takes advantage of `std::find()` to do the same more efficiently, improving its CPU runtime by a factor of about 25 in typical use.
ACKs for top commit:
achow101:
re-ACK 72efc26439
stickies-v:
re-ACK 72efc26439
Tree-SHA512: ddf0bff335cc8aa34f911aa4e0558fa77ce35d963d602e4ab1c63090b4a386faf074548daf06ee829c7f2c760d06eed0125cf4c34e981c6129cea1804eb3b719
It's unnecessary to keep the data around, as it doesn't do anything. If
prioritisetransaction is called again, we'll make a new entry in
mapDeltas.
These entries are only deleted when the transaction is mined or conflicted
from a block (i.e. not in replacement or eviction), are persisted in
mempool.dat, and never expire. If node operators use the RPC to
regularly prioritise/de-prioritise transactions, these (meaningless)
map entries may hang around forever and take up valuable mempool memory.
Add a stop_after_block_import field to the BlockManager options. Use
this field instead of the global gArgs.
This should allow users of the BlockManager to not rely on the global
Args.
Add a blocks_dir field to the BlockManager options. Move functions
relying on the global gArgs to get the blocks_dir into the BlockManager
class.
This should eventually allow users of the BlockManager to not rely on
the global Args and instead pass in their own options.
Remove access to the global gArgs for the fastprune argument and
replace it by adding a field to the existing BlockManager Options
struct.
When running `clang-tidy-diff` on this commit, there is a diagnostic
error: `unknown type name 'uint64_t' [clang-diagnostic-error] uint64_t
prune_target{0};`, which is fixed by including cstdint.
This should eventually allow users of the BlockManager to not rely on
the global gArgs and instead pass in their own options.
This is a commit in preparation for the next few commits. The functions
are moved to methods to avoid their re-declaration for the purpose of
passing in BlockManager options.
The functions that were now moved into the BlockManager should no longer
use the params as an argument, but instead use the member variable.
In the moved ReadBlockFromDisk and UndoReadFromDisk, change
the function signature to accept a reference to a CBlockIndex instead of
a raw pointer. The pointer is expected to be non-null, so reflect that
in the type.
To allow for the move of functions to BlockManager methods all call
sites require an instantiated BlockManager, or a callback to one.
The lambda captures a reference to the chainman unique_ptr to retrieve
block data. An assert is added on the chainman to ensure that the lambda
is not used while the chainman is uninitialized.
This is done in preparation for the following commits where blockstorage
functions are made BlockManager methods.
fa266c4bbf Temporarily work around gcc-13 warning bug in interfaces_tests (MarcoFalke)
fa28850562 Fix clang-tidy performance-unnecessary-copy-initialization warnings (MarcoFalke)
faaa60a30e Remove unused find_value global function (MarcoFalke)
fa422aeec2 scripted-diff: Use UniValue::find_value method (MarcoFalke)
fa548ac872 Add UniValue::find_value method (MarcoFalke)
Pull request description:
The global function has issues:
* It causes gcc-13 warnings, see https://github.com/bitcoin/bitcoin/issues/26926
* There is no rationale for it being a global function, when it acts like a member function
* `performance-unnecessary-copy-initialization` clang-tidy isn't run on it
Fix all issues by making it a member function.
ACKs for top commit:
achow101:
ACK fa266c4bbf
hebasto:
re-ACK fa266c4bbf
Tree-SHA512: 6c4e25da3122cd3b91c376bef73ea94fb3beb7bf8ef5cb3853c5128d95bfbacbcbfb16cc843eb7b1a7ebd350c2b6311f8085eeacf9aeeab3366987037d209e44
Included a test that checks if we call submitblock with
block.vtx.empty() then it throws an rpc deserialization error, currently
we only test if !block.vtx->IsCoinBase() throws an rpc deserialization
error
[WARN] The commit is obviously broken and will not run the CI system. In
the rare case this is hit in a git bisect, just skip the commit.
The goal here was to make it trivial to review with the git option:
--color-moved=dimmed-zebra
It is required to move everything into one file because "exit 0" will
otherwise stop working as intended when the containing bash script is no
longer executed with "source ...".
If there is desire to split up 06_script_b.sh into logical chunks in the
future, it will also be easier after the following commit.
Ensures better memory safety for this global. This came up during
discussion of the following commit, but is not strictly required for its
implementation.
fa1dbd04ca ci: Remove CI_EXEC bloat in test/06_script_b.sh (MarcoFalke)
fae8de926a ci: Move CI container kill out of 06_script_b.sh (MarcoFalke)
fa7d75540e ci: Pass full env to CI pod to avoid missing a var (MarcoFalke)
Pull request description:
`CI_EXEC` has many issues:
* It is roughly equivalent to `bash -c "$*"`, meaning that the full command will be treated as a single string, ignoring tokens.
* It must be put in front of (almost) every command, making it easy to forget, hard to debug the resulting failure, and the code verbose.
Fix all issues in one script by removing it.
ACKs for top commit:
fanquake:
ACK fa1dbd04ca - this conflicts with #27125, but that is going to be rebased soon, and this could be merged in the interim. cc TheCharlatan
TheCharlatan:
ACK fa1dbd04ca
Tree-SHA512: e5ab5503a05a787f2bc6ca25e71ad3dc166aade57e25d9677e72b1ca4e5fb6045c058dfd55f47ac93f710538e62d57c12cd7eb9d1260c6f55f3c8091908dc70d
59ebee3fb4 add ryanofsky to trusted-keys (Ryan Ofsky)
Pull request description:
For maintaining interfaces and other areas of the codebase. Some previous discussion in IRC meeting https://gnusha.org/bitcoin-core-dev/2023-05-04.log
ACKs for top commit:
darosior:
ACK 59ebee3fb4 for adding ryanofsky as a maintainer.
hebasto:
ACK 59ebee3fb4
glozow:
ACK 59ebee3fb4
pinheadmz:
ACK 59ebee3fb4
stickies-v:
utACK 59ebee3fb4
brunoerg:
ACK 59ebee3fb4
theStack:
ACK 59ebee3fb4
mzumsande:
ACK 59ebee3fb4
theuni:
ACK 59ebee3fb4.
jarolrod:
ACK 59ebee3fb4
Tree-SHA512: e3d2815d8950e419316ee49ec70f01cb1939de61b3017b8140c0194b519b5b523a618d3ad2ab9fe3fd32543649c1465fdd6baf52ad68da48b680bd4898186ff4
52e52071e0 p2p: Avoid prematurely clearing download state for other peers (Suhas Daftuar)
Pull request description:
Avoid letting one peer send us data that clears out the download request (and related timers etc) from another peer.
The exception is if a block is definitely stored to disk, in which case we'll clear the download state (just as we do for compact blocks).
ACKs for top commit:
jamesob:
ACK 52e52071e0 ([`jamesob/ackr/27608.1.sdaftuar.p2p_avoid_prematurely_cl`](https://github.com/jamesob/bitcoin/tree/ackr/27608.1.sdaftuar.p2p_avoid_prematurely_cl))
instagibbs:
ACK 52e52071e0
fjahr:
Code review ACK 52e52071e0
mzumsande:
Code Review ACK 52e52071e0
Tree-SHA512: 3ee92507edc3303c16c70ca44ba6c28c104afe95196e4b9167032590ed23d4f569f654f8eb8758940bd6536bc9ca810d2a77d2739db386b927e8b3f3cf55cb16
If transactions are being added to the mempool at a rate faster than 7tx/s
(INVENTORY_BROADCAST_PER_SECOND) then peers' inventory_to_send queue can
become relatively large. If this happens, increase the number of txids
we include in an INV message (normally capped at 35) by 5 for each 1000
txids in the queue.
This will tend to clear a temporary excess out reasonably quickly; an
excess of 4000 invs to send will be cleared down to 1000 in about 30
minutes, while an excess of 20000 invs would be cleared down to 1000 in
about 60 minutes.
We use CompareDepthAndScore to choose an order of txs to inv. Rather
than sorting txs that have been evicted from the mempool at the end
of the list, sort them at the beginning so they are removed from
the queue immediately.
