mirrors/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2024-11-19 18:09:47 +01:00
Code Issues Projects Releases Wiki Activity
Bitcoin Core integration/staging tree
41,983 Commits 6 Branches 318 Tags 3.2 GiB
C++ 64%
Python 19.9%
C 12.4%
CMake 1.1%
Shell 0.9%
Other 1.6%
27a770b34b
Go to file
Ava Chow 27a770b34b
Merge bitcoin/bitcoin#28280: Don't empty dbcache on prune flushes: >30% faster IBD 
589db872e1 validation: don't erase coins cache on prune flushes (Andrew Toth)
0e8918755f Add linked-list test to CCoinsViewCache::SanityCheck (Pieter Wuille)
05cf4e1875 coins: move Sync logic to CoinsViewCacheCursor (Andrew Toth)
7825b8b9ae coins: pass linked list of flagged entries to BatchWrite (Andrew Toth)
a14edada8a test: add cache entry linked list tests (Andrew Toth)
24ce37cb86 coins: track flagged cache entries in linked list (Andrew Toth)
58b7ed156d coins: call ClearFlags in CCoinsCacheEntry destructor (Andrew Toth)
8bd3959fea refactor: require self and sentinel parameters for AddFlags (Andrew Toth)
75f36d241d refactor: add CoinsCachePair alias (Andrew Toth)
f08faeade2 refactor: move flags to private uint8_t and rename to m_flags (Andrew Toth)
4e4fb4cbab refactor: disallow setting flags in CCoinsCacheEntry constructors (Andrew Toth)
8737c0cefa refactor: encapsulate flags setting with AddFlags and ClearFlags (Andrew Toth)
9715d3bf1e refactor: encapsulate flags get access for all other checks (Andrew Toth)
df34a94e57 refactor: encapsulate flags access for dirty and fresh checks (Andrew Toth)

Pull request description:

  Since https://github.com/bitcoin/bitcoin/pull/17487 we no longer need to clear the coins cache when syncing to disk. A warm coins cache significantly speeds up block connection, and only needs to be fully flushed when nearing the `dbcache` limit.

  For frequent pruning flushes there's no need to empty the cache and kill connect block speed. However, simply using `Sync` in place of `Flush` actually slows down a pruned full IBD with a high `dbcache` value. This is because as the cache grows, sync takes longer since every coin in the cache is scanned to check if it's dirty. For frequent prune flushes and a large cache this constant scanning starts to really slow IBD down, and just emptying the cache on every prune becomes faster.

  To fix this, we can add two pointers to each cache entry and construct a doubly linked list of dirty entries. We can then only iterate through all dirty entries on each `Sync`, and simply clear the pointers after.

  With this approach a full IBD with `dbcache=16384` and `prune=550` was 32% faster than master. For default `dbcache=450` speedup was ~9%. All benchmarks were run with `stopatheight=800000`.

  |  | prune | dbcache | time | max RSS | speedup |
  |-----------:|----------:|------------:|--------:|-------------:|--------------:|
  | master | 550 | 16384 | 8:52:57 | 2,417,464k | - |
  | branch | 550 | 16384 | 6:01:00 | 16,216,736k | 32% |
  | branch | 550 | 450 | 8:05:08 | 2,818,072k | 8.8% |
  | master | 10000 | 5000 | 8:19:59 | 2,962,752k | - |
  | branch | 10000 | 5000| 5:56:39 | 6,179,764k | 28.8% |
  | master | 0 | 16384 | 4:51:53 | 14,726,408k | - |
  | branch | 0 | 16384 | 4:43:11 | 16,526,348k | 2.7% |
  | master | 0 | 450 | 7:08:07 | 3,005,892k | - |
  | branch | 0 | 450 | 6:57:24 | 3,013,556k |2.6%|

  While the 2 pointers add memory to each cache entry, it did not slow down IBD. For non-pruned IBD results were similar for this branch and master. When I performed the initial IBD, the full UTXO set could be held in memory when using the max `dbcache` value. For non-pruned IBD with max `dbcache` to tip ended up using 12% more memory, but it was also 2.7% faster somehow. For smaller `dbcache` values the `dbcache` limit is respected so does not consume more memory, and the potentially more frequent flushes were not significant enough to cause any slowdown.

  For reviewers, the commits in order do the following:
  First 4 commits encapsulate all accesses to `flags` on cache entries, and then the 5th makes `flags` private.
  Commits `refactor: add CoinsCachePair alias` to `coins: call ClearFlags in CCoinsCacheEntry destructor` create the linked list head nodes and cache entry self references and pass them into `AddFlags`.
  Commit `coins: track flagged cache entries in linked list` actually adds the entries into a linked list when they are flagged DIRTY or FRESH and removes them from the linked list when they are destroyed or the flags are cleared manually. However, the linked list is not yet used anywhere.
  Commit `test: add cache entry linked list tests` adds unit tests for the linked list.
  Commit `coins: pass linked list of flagged entries to BatchWrite` uses the linked list to iterate through DIRTY entries instead of using the entire coins cache.
  Commit `validation: don't erase coins cache on prune flushes` uses `Sync` instead of `Flush` for pruning flushes, so the cache is no longer cleared.

  Inspired by [this comment](https://github.com/bitcoin/bitcoin/pull/15265#issuecomment-457720636).

  Fixes https://github.com/bitcoin/bitcoin/issues/11315.

ACKs for top commit:
  paplorinc:
    ACK 589db872e1
  sipa:
    reACK 589db872e1
  achow101:
    ACK 589db872e1
  mzumsande:
    re-ACK 589db872e1

Tree-SHA512: 23b2bc01c83edacb5b39aa60bb0b766de9a74ce17f0c59bf13b97b4328a7b758ad9aff6581c3ca88e2973f7658380651530d497444f48d6e22ea0bfc51cc921d
2024-08-07 20:06:39 -04:00
.github Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
.tx qt: Bump Transifex slug for 28.x 2024-07-30 16:14:19 +01:00
build_msvc build: remove check for __declspec(dllexport) 2024-08-05 14:54:56 +01:00
build-aux/m4 qt, build: Drop QT_STATICPLUGIN macro 2024-08-01 14:01:07 +01:00
ci Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
contrib testnet: Introduce Testnet4 2024-08-06 01:38:10 +02:00
depends depends: fix ZMQ CMake getcachesize check 2024-08-05 14:17:58 +01:00
doc Merge bitcoin/bitcoin#29775: Testnet4 including PoW difficulty adjustment fix 2024-08-07 13:05:04 -04:00
share contrib: rpcauth.py - Add new option (-j/--json) to output text in json format 2024-04-25 08:32:28 -05:00
src Merge bitcoin/bitcoin#28280: Don't empty dbcache on prune flushes: >30% faster IBD 2024-08-07 20:06:39 -04:00
test Merge bitcoin/bitcoin#29775: Testnet4 including PoW difficulty adjustment fix 2024-08-07 13:05:04 -04:00
.cirrus.yml Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
.editorconfig
.gitattributes
.gitignore Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
.python-version Bump .python-version from 3.9.17 to 3.9.18 2023-10-24 18:51:24 +02:00
.style.yapf
autogen.sh build: make sure we can overwrite config.{guess,sub} 2023-06-13 14:58:43 +02:00
configure.ac Merge bitcoin/bitcoin#30590: build: Remove unused visibility checks 2024-08-06 17:44:43 +01:00
CONTRIBUTING.md Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
COPYING doc: upgrade Bitcoin Core license to 2024 2024-01-10 16:29:01 -06:00
INSTALL.md
Makefile.am Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
README.md Squashed 'src/secp256k1/' changes from 4af241b320..642c885b61 2024-08-02 11:32:47 +01:00
SECURITY.md Update security.md contact for achow101 2023-12-14 18:14:54 -05:00

README.md

Bitcoin Core integration/staging tree

https://bitcoincore.org

For an immediately usable, binary version of the Bitcoin Core software, see https://bitcoincore.org/en/download/.

What is Bitcoin Core?

Bitcoin Core connects to the Bitcoin peer-to-peer network to download and fully validate blocks and transactions. It also includes a wallet and graphical user interface, which can be optionally built.

Further information about Bitcoin Core is available in the doc folder.

License

Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/licenses/MIT.

Development Process

The master branch is regularly built (see doc/build-*.md for instructions) and tested, but it is not guaranteed to be completely stable. Tags are created regularly from release branches to indicate new official, stable release versions of Bitcoin Core.

The https://github.com/bitcoin-core/gui repository is used exclusively for the development of the GUI. Its master branch is identical in all monotree repositories. Release branches and tags do not exist, so please do not fork that repository unless it is for development reasons.

The contribution workflow is described in CONTRIBUTING.md and useful hints for developers can be found in doc/developer-notes.md.

Testing

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.

Automated Testing

Developers are strongly encouraged to write unit tests for new code, and to submit new unit tests for old code. Unit tests can be compiled and run (assuming they weren't disabled in configure) with: make check. Further details on running and extending unit tests can be found in /src/test/README.md.

There are also regression and integration tests, written in Python. These tests can be run (if the test dependencies are installed) with: test/functional/test_runner.py

The CI (Continuous Integration) systems make sure that every pull request is built for Windows, Linux, and macOS, and that unit/sanity tests are run automatically.

Manual Quality Assurance (QA) Testing

Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.

Translations

Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.

Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.

Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.