mirrors/bitcoin
1
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-02-21 14:34:49 +01:00
Code Issues Projects Releases Wiki Activity
Bitcoin Core integration/staging tree
40981 commits 6 branches 321 tags 5.8 GiB
C++ 56.4%
C 22.6%
Python 17.7%
CMake 1%
Shell 0.8%
Other 1.3%
Find a file
Ava Chow d6069cb8d6
Merge bitcoin/bitcoin#28233: validation: don't clear cache on periodic flush: >2x block connection speed 
4a6d1d1e3b validation: don't clear cache on periodic flush (Andrew Toth)

Pull request description:

  Since https://github.com/bitcoin/bitcoin/pull/17487 we no longer need to clear the coins cache when syncing to disk. A warm coins cache significantly speeds up block connection, and only needs to be fully flushed when nearing the `dbcache` limit.

  Periodic flushes occur every 24 hours, which empties the cache and causes block connection to slow down. By keeping the cache through periodic flushes a node can run for several days with an increasingly hotter cache and connect blocks much more quickly. Now not only can setting a higher `dbcache` value be beneficial for IBD, it can also be beneficial for connecting blocks faster.

  To benchmark in real world usage, I spun up 6 identical `t2.small` AWS EC2 instances, all running in the same region in the same VPC. I configured 2 instances to run master, 2 instances to run the change in this PR, and 2 instances to run the change in this PR but with `dbcache=1000`. All instances had `prune=5000` and a 20 GB `gp2` `EBS` volume. A 7th EC2 instance in the same VPC ran master and connected only to some trusted nodes in the outside network. Each of the 6 nodes under test only connected directly to this 7th instance. I manually pruned as much as possible and uploaded the same `blocks`, `chainstate` and `mempool.dat` to all instances. I started all 6 peers simultaneously at block height `835245` and ran them for over a week until block `836534`.

  The results were much faster block connection times for this branch compared to master, and much faster for this branch with `dbcache=1000` compared to default `dbcache`.

  |  branch |speed |
  |-----------:|----------:|
  | master 1 | 1995.49ms/blk |
  | master 2 | 2129.78ms/blk |
  | branch default dbcache 1 | 1189.65ms/blk |
  | branch default dbcache 2 | 1037.74ms/blk |
  | branch dbcache=1000 1 | 393.69ms/blk |
  | branch dbcache=1000 2 | 427.77ms/blk |

  The log files of all 6 instances are [here](https://gist.github.com/andrewtoth/03c95033e7581d5dbc5be028639a1a91).
  There is a lot of noise with the exact times of blocks being connected, so I plotted the rolling 20 block connect time averages. The large dots are the times where the cache is emptied. For the red master nodes, this happens every 24 hours. The blue branch nodes with default `dbcache` only filled up and emptied the caches once, which is seen in the middle. The green branch nodes with 1000 `dbcache` never emptied the cache. It is very clear from the chart that whenever the cache is emptied, connect block speed degrades significantly.

  ![plot](https://github.com/bitcoin/bitcoin/assets/237213/802cb28d-1ad4-47c3-a886-c5366b423eca)

  Also note that this still clears the cache for pruning flushes. Having frequent pruning flushes with a large cache that doesn't clear is less performant than the status quo https://github.com/bitcoin/bitcoin/pull/15265#issuecomment-458657451. See https://github.com/bitcoin/bitcoin/pull/28280.

ACKs for top commit:
  sipa:
    utACK 4a6d1d1e3b
  achow101:
    ACK 4a6d1d1e3b
  brunoerg:
    crACK 4a6d1d1e3b

Tree-SHA512: 05dbc677bc309bbcf89c52a6c5e853e2816b0ef0b5ee3719b30696df315a0427e244bb82da9ad828ec0e7ea8764552f8affe14c0184b52adf1909f5d8c1b4f9e
2024-05-13 16:31:19 -04:00
.github ci, msvc: Add "Run fuzz binaries" step 2024-04-18 10:27:55 +01:00
.tx qt: Bump Transifex slug for 27.x 2024-02-07 09:24:32 +00:00
build-aux/m4 Add lint check for bitcoin-config.h include IWYU pragma 2024-05-01 08:33:43 +02:00
build_msvc Merge bitcoin/bitcoin#29494: build: Assume HAVE_CONFIG_H, Add IWYU pragma keep to bitcoin-config.h includes 2024-05-07 14:14:03 -04:00
ci ci: Exclude feature_init for now in valgrind task 2024-05-07 08:53:18 +02:00
contrib Merge bitcoin/bitcoin#29739: build: swap cctools otool for llvm-objdump 2024-05-11 18:34:42 +08:00
depends build: swap otool for (llvm-)objdump 2024-05-08 16:36:41 +08:00
doc Merge bitcoin/bitcoin#30025: doc: fix broken relative md links 2024-05-08 11:54:46 +08:00
share contrib: rpcauth.py - Add new option (-j/--json) to output text in json format 2024-04-25 08:32:28 -05:00
src Merge bitcoin/bitcoin#28233: validation: don't clear cache on periodic flush: >2x block connection speed 2024-05-13 16:31:19 -04:00
test Merge bitcoin/bitcoin#29948: test: add missing comparison of node1's mempool in MempoolPackagesTest 2024-05-10 12:44:42 -04:00
.cirrus.yml ci: Skip git install if it is already installed 2024-02-16 16:06:45 +01:00
.editorconfig ci: Drop AppVeyor CI integration 2021-09-07 06:12:53 +03:00
.gitattributes Separate protocol versioning from clientversion 2014-10-29 00:24:40 -04:00
.gitignore Merge bitcoin/bitcoin#29733: build, macos: Drop unused osx_volname target 2024-04-02 14:57:22 +01:00
.python-version Bump .python-version from 3.9.17 to 3.9.18 2023-10-24 18:51:24 +02:00
.style.yapf Update .style.yapf 2023-06-01 23:35:10 +05:30
autogen.sh build: make sure we can overwrite config.{guess,sub} 2023-06-13 14:58:43 +02:00
configure.ac build: swap otool for (llvm-)objdump 2024-05-08 16:36:41 +08:00
CONTRIBUTING.md Merge bitcoin/bitcoin#29645: doc: update release-process.md 2024-04-30 20:17:04 -04:00
COPYING doc: upgrade Bitcoin Core license to 2024 2024-01-10 16:29:01 -06:00
INSTALL.md doc: Added hyperlink for doc/build 2021-09-09 19:53:12 +05:30
Makefile.am Merge bitcoin/bitcoin#29739: build: swap cctools otool for llvm-objdump 2024-05-11 18:34:42 +08:00
README.md doc: Explain Bitcoin Core in README.md 2022-05-10 07:49:09 +02:00
SECURITY.md Update security.md contact for achow101 2023-12-14 18:14:54 -05:00

README.md

Bitcoin Core integration/staging tree

https://bitcoincore.org

For an immediately usable, binary version of the Bitcoin Core software, see https://bitcoincore.org/en/download/.

What is Bitcoin Core?

Bitcoin Core connects to the Bitcoin peer-to-peer network to download and fully validate blocks and transactions. It also includes a wallet and graphical user interface, which can be optionally built.

Further information about Bitcoin Core is available in the doc folder.

License

Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/licenses/MIT.

Development Process

The master branch is regularly built (see doc/build-*.md for instructions) and tested, but it is not guaranteed to be completely stable. Tags are created regularly from release branches to indicate new official, stable release versions of Bitcoin Core.

The https://github.com/bitcoin-core/gui repository is used exclusively for the development of the GUI. Its master branch is identical in all monotree repositories. Release branches and tags do not exist, so please do not fork that repository unless it is for development reasons.

The contribution workflow is described in CONTRIBUTING.md and useful hints for developers can be found in doc/developer-notes.md.

Testing

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.

Automated Testing

Developers are strongly encouraged to write unit tests for new code, and to submit new unit tests for old code. Unit tests can be compiled and run (assuming they weren't disabled in configure) with: make check. Further details on running and extending unit tests can be found in /src/test/README.md.

There are also regression and integration tests, written in Python. These tests can be run (if the test dependencies are installed) with: test/functional/test_runner.py

The CI (Continuous Integration) systems make sure that every pull request is built for Windows, Linux, and macOS, and that unit/sanity tests are run automatically.

Manual Quality Assurance (QA) Testing

Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.

Translations

Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.

Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.

Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.