Commit Graph

5557 Commits

Author SHA1 Message Date
Jon Atack
1316119ce7 script: update ignored-words.txt 2023-06-29 16:14:07 -06:00
Jon Atack
92408224a4 test: fix PEP484 no implicit optional argument types errors
Fix warnings for these files when ./test/lint/lint-python.py is run using
mypy 0.991 (released 11/2022) and later:

$ test/lint/lint-python.py
test/functional/test_framework/coverage.py:23: error: Incompatible default for argument "coverage_logfile" (default has type "None", argument has type "str")  [assignment]
test/functional/test_framework/coverage.py:23: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
test/functional/test_framework/util.py:318: error: Incompatible default for argument "timeout" (default has type "None", argument has type "int")  [assignment]
test/functional/test_framework/util.py:318: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
test/functional/test_framework/util.py:318: error: Incompatible default for argument "coveragedir" (default has type "None", argument has type "str")  [assignment]
test/functional/interface_rest.py:67: error: Incompatible default for argument "query_params" (default has type "None", argument has type "dict[str, Any]")  [assignment]
test/functional/interface_rest.py:67: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True

Verified using https://github.com/hauntsaninja/no_implicit_optional

For details, see:

https://mypy-lang.blogspot.com/2022/11/mypy-0990-released.html
2023-06-29 16:14:07 -06:00
Jon Atack
f86a301433 script, test: add missing python type annotations
Fix warnings for these files when ./test/lint/lint-python.py is run using
mypy 0.991 (released 11/2022) and later:

"By default the bodies of untyped functions are not checked, consider using
--check-untyped-defs [annotation-unchecked]"

For details, see:

https://mypy-lang.blogspot.com/2022/11/mypy-0990-released.html
2023-06-29 16:13:51 -06:00
fanquake
3d51f7c9a8
Merge bitcoin/bitcoin#27932: test: Fuzz on macOS
fae7c50d20 test: Run fuzz tests on macOS (MarcoFalke)

Pull request description:

  Any reason not to?

ACKs for top commit:
  jamesob:
    Github ACK fae7c50d20
  dergoegge:
    utACK fae7c50d20

Tree-SHA512: e45122d73fafb17cea312258314b826cb0745e08daadd28465f687ec02d4c127d2f8cbe20179a9fff5712038850c02c968abb4838fa088b7555e28709317d3a3
2023-06-29 13:08:58 +01:00
fanquake
e8543629ae
Merge bitcoin/bitcoin#27884: test: Use TestNode datadir_path or chain_path where possible
aaaa3aefbd test: Use TestNode *_path properties where possible (MarcoFalke)
dddd89962b test: Allow pathlib.Path as RPC argument via authproxy (MarcoFalke)
fa41614a0a scripted-diff: Use wallets_path and chain_path where possible (MarcoFalke)
fa493fadfb test: Use wallet_dir lambda in wallet_multiwallet test where possible (MarcoFalke)

Pull request description:

  It seems inconsistent, fragile and verbose to:

  * Call `get_datadir_path` to recreate the path that already exists as field in TestNode
  * Call `os.path.join` with the hardcoded chain name or `self.chain` to recreate the TestNode `chain_path` property
  * Sometimes even use the hardcoded node dir name (`"node0"`)

  Fix all issues by using the TestNode properties.

ACKs for top commit:
  willcl-ark:
    re-ACK aaaa3aefbd
  theStack:
    Code-review ACK aaaa3aefbd 🌊

Tree-SHA512: e4720278085beb8164e1fe6c1aa18f601558a9263494ce69a83764c1487007de63ebb51d1b1151862dc4d5b49ded6162a5c1553cd30ea1c28627d447db4d8e72
2023-06-29 09:51:53 +01:00
Andrew Chow
626d346469
Merge bitcoin/bitcoin#26222: Introduce secp256k1 module with field and group classes to test framework
d4fb58ae8a test: EC: optimize scalar multiplication of G by using lookup table (Sebastian Falbesoner)
1830dd8820 test: add secp256k1 module with FE (field element) and GE (group element) classes (Pieter Wuille)

Pull request description:

  This PR rewrites a portion of `test_framework/key.py`, in a compatible way, by introducing classes that encapsulate field element and group element logic, in an attempt to be more readable and reusable.

  To maximize readability, the group element logic does not use Jacobian coordinates. Instead, group elements just store (affine) X and Y coordinates directly. To compensate for the performance loss this causes, field elements are represented as fractions. This undoes most, but not all, of the performance loss, and there is a few % slowdown (as measured in `feature_taproot.py`, which heavily uses this).

  The upside is that the implementation for group laws (point doubling, addition, subtraction, ...) is very close to the mathematical description of elliptic curves, and this extends to potential future extensions (e.g. ElligatorSwift as needed by #27479).

ACKs for top commit:
  achow101:
    ACK d4fb58ae8a
  theStack:
    re-ACK d4fb58ae8a
  stratospher:
    tested ACK d4fb58a. really liked how this PR makes the secp256k1 code in the tests more intuitive and easier to follow!

Tree-SHA512: 9e0d65d7de0d4fb35ad19a1c19da7f41e5e1db33631df898c6d18ea227258a8ba80c893dab862b0fa9b0fb2efd0406ad4a72229ee26d7d8d733dee1d56947f18
2023-06-28 16:27:55 -04:00
Andrew Chow
caff95a023
Merge bitcoin/bitcoin#27896: Remove the syscall sandbox
32e2ffc393 Remove the syscall sandbox (fanquake)

Pull request description:

  After initially being merged in #20487, it's no-longer clear that an internal syscall sandboxing mechanism is something that Bitcoin Core should have/maintain, especially when compared to better maintained/supported alterantives, i.e [firejail](https://github.com/netblue30/firejail).

  There is more related discussion in #24771.

  Note that given where it's used, the sandbox also gets dragged into the kernel.

  If it's removed, this should not require any sort of deprecation, as this was only ever an opt-in, experimental feature.

  Closes #24771.

ACKs for top commit:
  davidgumberg:
     crACK 32e2ffc393
  achow101:
    ACK 32e2ffc393
  dergoegge:
    ACK 32e2ffc393

Tree-SHA512: 8cf71c5623bb642cb515531d4a2545d806e503b9d57bfc15a996597632b06103d60d985fd7f843a3c1da6528bc38d0298d6b8bcf0be6f851795a8040d71faf16
2023-06-27 18:19:21 -04:00
fanquake
7ee41217b3
Merge bitcoin/bitcoin#27940: test: Add implicit-signed-integer-truncation:*/include/c++/ suppression
fae55f989e test: Add implicit-signed-integer-truncation:*/include/c++/ suppression (MarcoFalke)

Pull request description:

  Needed for aarch64. Steps to test on aarch64:

  ```
  lscpu | grep Arch
  FILE_ENV="./ci/test/00_setup_env_native_fuzz.sh" ./ci/test_run_all.sh
  ```

ACKs for top commit:
  fanquake:
    ACK fae55f989e  - reproduced the failure:

Tree-SHA512: b5058873118d285cc5d678a572cf4b890f8d68a24e1ac0987490f1b4123469a2b4456b08474f372e6aa49bb0d69e16f2c8277208b1cde3222a317f000beb5056
2023-06-27 15:52:33 +01:00
Sebastian Falbesoner
d4fb58ae8a test: EC: optimize scalar multiplication of G by using lookup table
On my machine, this speeds up the functional test feature_taproot.py by
a factor of >1.66x (runtime decrease from 1m16.587s to 45.334s).

Co-authored-by: Pieter Wuille <pieter@wuille.net>
2023-06-27 09:34:52 -04:00
Pieter Wuille
1830dd8820 test: add secp256k1 module with FE (field element) and GE (group element) classes
These are primarily designed for ease of understanding, not performance.
2023-06-27 09:34:48 -04:00
Andrew Chow
8847497161
Merge bitcoin/bitcoin#27631: test: avoid sporadic MINIMALDATA failure in feature_taproot.py (fixes #27595)
54877253c8 test: avoid sporadic MINIMALDATA failure in feature_taproot.py (fixes #27595) (Sebastian Falbesoner)

Pull request description:

  The functional test feature_taproot.py fails in some rare cases on the execution of the following `"branched_codesep"` spending script (can be reproduced via `$ ./test/functional/feature_taproot.py --randomseed 9048710178866422833` on master / 137a98c5a2):

  9d85c03620/test/functional/feature_taproot.py (L741)

  The problem occurs if the first data-push (having random content with a random length in the range [0, 510]) has a length of 1 and the single byte has value of [1...16] or [-1]; in this case, the data-push is not minimally encoded by test framework's CScript class (i.e. doesn't use the special op-codes OP_1...OP_16 or OP_1NEGATE) and the script interpreter throws an SCRIPT_ERR_MINIMALDATA error:

  ```
  test_framework.authproxy.JSONRPCException: non-mandatory-script-verify-flag (Data push larger than necessary) (-26)
  ```

  Background: the functional test framework's CScript class translates passed bytes/bytearrays always to data pushes using OP_PUSHx/OP_PUSHDATA{1,2,4} op-codes (see `CScript.__coerce_instance(...)`). E.g. the expression `CScript(bytes([1]))` yields `bytes([OP_PUSH1, 1])` instead of the minimal-encoded `bytes([OP_1])`.

  Fix this by adapting the random-size range to [2,...], i.e. never pass byte-arrays below length two to be pushed.

  Closes #27595.

ACKs for top commit:
  instagibbs:
    ACK 54877253c8
  sipa:
    utACK 54877253c8
  achow101:
    ACK 54877253c8

Tree-SHA512: 3ffad89b2c3985c20702242192e744c9b10188bff880efaf3c38424a00fa07bd4608d8c948678ff9cdbb4e1e5b06696c7f55407ee10bb05edbb3ee03aa599cdc
2023-06-23 18:54:06 -04:00
Andrew Chow
035ae61c5a
Merge bitcoin/bitcoin#27577: p2p: give seednodes time before falling back to fixed seeds
30778124b8 net: Give seednodes time before falling back to fixed seeds (Martin Zumsande)

Pull request description:

  `-seednode` is an alternative bootstrap mechanism - when choosing it, we make a `AddrFetch` connection to the specified peer, gather addresses from them, and then disconnect. Presumably, if users specify a seednode they prefer addresses from that node over fixed seeds.

  However, when disabling dns seeds and specifiying `-seednode`, `CConnman::ProcessAddrFetch()`  immediately removes the entry from `m_addr_fetches` (before the seednode could give us addresses) - and once `m_addr_fetches`  is empty, `ThreadOpenConnections` will add fixed seeds, resulting in a "race" between the fixed seeds and seednodes filling up AddrMan.

  This PR suggests to check for any provided `-seednode` arg instead of using the size of `m_addr_fetches`, thus delaying the querying of fixed seeds for 1 minute when specifying any seednode (as we already do for `addnode` peers).
  That way, we actually give the seednodes a chance for  to provide us with addresses before falling back to fixed seeds.

  This can be tested with `bitcoind -debug=net -dnsseed=0 -seednode=(...)` on a node without `peers.dat` and observing the debug log.

ACKs for top commit:
  ajtowns:
    utACK 30778124b8
  achow101:
    ACK 30778124b8
  dergoegge:
    Code review ACK 30778124b8
  sr-gi:
    ACK [3077812](30778124b8) with a tiny nit, feel free to ignore it

Tree-SHA512: 96446eb34c0805f10ee158a00a3001a07029e795ac40ad5638228d426e30e9bb836c64ac05d145f2f9ab23ec5a528f3a416e3d52ecfdfb0b813bd4b1ebab3c01
2023-06-23 17:39:58 -04:00
MarcoFalke
fae55f989e
test: Add implicit-signed-integer-truncation:*/include/c++/ suppression 2023-06-23 12:03:51 +02:00
fanquake
a7261da479
Merge bitcoin/bitcoin#27831: test: handle failed assert_equal() assertions in bcc callback functions
61f4b9b7ad Manage exceptions in bcc callback functions (virtu)

Pull request description:

  Address #27380 (and similar future issues) by handling failed `assert_equal()` assertions in bcc callback functions

  ### Problem

  Exceptions are not propagated in ctype callback functions used by bcc. This means an AssertionError exception raised by `assert_equal()` to signal a failed assertion is not getting caught and properly logged. Instead, the error is logged to stdout and execution of the callback stops.

  The current workaround to check whether all `assert_equal()` assertions in a callback succeeded is to increment a success counter after the assertions (which only gets incremented if none exception is raised and stops execution). Then, outside the callback, the success counter can be used to check whether a callback executed successfully.

  One issue with the described workaround is that when an exception occurs, there is no way of telling which of the `assert_equal()` statements caused the exception; moreover, there is no way of inspecting how the pieces of data that got compared in `assert_equal()` differed (often a crucial clue when debugging what went wrong).

  This problem is happening in #27380: Sporadically, in the `mempool:rejected` test, execution does not reach the end of the callback function and the success counter is not incremented. Thus, the test fails when comparing the counter to its expected value of one. Without knowing which of the asserts failed any why it failed, this issue is hard to debug.

  ### Solution

  Two fixes come to mind. The first involves having the callback function make event data accessible outside the callback and inspecting the event using `assert_equal()` outside the callback. This solution still requires a counter in the callback in order  to tell whether a callback was actually executed or if instead the call to perf_buffer_poll() timed out.

  The second fix entails wrapping all relevant `assert_equal()` statements inside callback functions into try-catch blocks and manually logging AssertionErrors. While not as elegant in terms of design, this approach can be more pragmatic for more complex tests (e.g., ones involving multiple events, events of different types, or the order of events).

  The solution proposed here is to select the most pragmatic fix on a case-by-case basis: Tests in `interface_usdt_net.py`, `interface_usdt_mempool.py` and `interface_usdt_validation.py` have been refactored to use the first approach, while the second approach was chosen for `interface_usdt_utxocache.py` (partly to provide a reference for the second approach, but mainly because the utxocache tests are the most intricate tests, and refactoring them to use the first approach would negatively impact their readability). Lastly, `interface_usdt_coinselection.py` was kept unchanged because it does not use `assert_equal()` statements inside callback functions.

ACKs for top commit:
  0xB10C:
    Reviewed the changes since my last review. ACK 61f4b9b7ad. I've tested that the combined log contains both exceptions by modifying `interface_usdt_utxocache.py`.
  willcl-ark:
    utACK 61f4b9b
  stickies-v:
    utACK 61f4b9b7a

Tree-SHA512: 85cdaabf370d4f09a9eab6af9ce7c796cd9d08cb91f38f021f71adda34c5f643331022dd09cadb95be2185dad6016c95cbb8942e41e4fbd566a49bf431c5141a
2023-06-22 16:08:15 +01:00
MarcoFalke
fae7c50d20
test: Run fuzz tests on macOS
Also, fix a few bugs:

* Error: RPC command "enumeratesigners" not found in RPC_COMMANDS_SAFE_FOR_FUZZING or RPC_COMMANDS_NOT_SAFE_FOR_FUZZING. Please update test/fuzz/rpc.cpp.
* in run_once: ...format(" ".join(result.args), ... TypeError: sequence item 2: expected str instance, PosixPath found
2023-06-22 13:54:17 +02:00
fanquake
2880bb588a
Merge bitcoin/bitcoin#27889: test: Kill BOOST_ASSERT and update the linter
28fff06afe test: Make linter to look for `BOOST_ASSERT` macros (Hennadii Stepanov)
47fe551e52 test: Kill `BOOST_ASSERT` (Hennadii Stepanov)

Pull request description:

  One of the goals of https://github.com/bitcoin/bitcoin/pull/27783 was to get rid of the `BOOST_ASSERT` macros instead of including the `boost/assert.hpp` headers. See https://github.com/bitcoin/bitcoin/pull/27783#discussion_r1210612717.

  It turns out that a couple of those macros sneaked into the codebase in https://github.com/bitcoin/bitcoin/pull/27790.

  This PR makes the linter guard against new instances of the `BOOST_ASSERT` macros and replaces the current ones.

ACKs for top commit:
  kevkevinpal:
    ACK [28fff06](28fff06afe)
  stickies-v:
    ACK 28fff06af
  TheCharlatan:
    ACK 28fff06afe

Tree-SHA512: 371f613592cf677afe0196d18c83943c6c8f1e998f57b4ff3ee58bfeff8636e4dac1357840d8611b4f7b197def94df10fe1a8ca3282b00b7b4eff4624552dda8
2023-06-22 12:33:35 +01:00
Martin Zumsande
30778124b8 net: Give seednodes time before falling back to fixed seeds
Before, we'd remove a seednode from the list right after connecting
to it, leading to a race with loading the fixed seed and connecting
to them.
2023-06-21 15:11:00 -04:00
fanquake
7d65e3372f
Merge bitcoin/bitcoin#27733: test: refactor: introduce generate_keypair helper with WIF support
1a572ce7d6 test: refactor: introduce `generate_keypair` helper with WIF support (Sebastian Falbesoner)

Pull request description:

  In functional tests it is a quite common scenario to generate fresh elliptic curve keypairs, which is currently a bit cumbersome as it involves multiple steps, e.g.:

      privkey = ECKey()
      privkey.generate()
      privkey_wif = bytes_to_wif(privkey.get_bytes())
      pubkey = privkey.get_pubkey().get_bytes()

  Simplify this by providing a new `generate_keypair` helper function that returns the private key either as `ECKey` object or as WIF-string (depending on the boolean `wif` parameter) and the public key as byte-string; these formats are what we mostly need (currently we don't use `ECPubKey` objects from generated keypairs anywhere).

  With this, most of the affected code blocks following the pattern above can be replaced by one-liners, e.g.:

      privkey, pubkey = generate_keypair(wif=True)

  Note that after this commit, the only direct uses of `ECKey` remain in situations where we want to set the private key explicitly, e.g. in MiniWallet (test/functional/test_framework/wallet.py) or the test for the signet miner script (test/functional/tool_signet_miner.py).

ACKs for top commit:
  instagibbs:
    ACK 1a572ce7d6
  kevkevinpal:
    reACK [1a572ce](1a572ce7d6)
  stratospher:
    ACK 1a572ce7. neat to have this since keypair generation is done in lots of places.

Tree-SHA512: ceb695ba7b34dc9f65357b55be03e67609e7e13a178083d405284eff4d8d3c5cea4fb0b6632658604a533f38ebfefc33e0c375995cc21ebc7843442ad764287b
2023-06-21 10:45:25 +01:00
fanquake
a596bdf3e9
Merge bitcoin/bitcoin#27919: ci: Run fuzz target even if input folder is empty
0000f55293 ci: Run fuzz target even if input folder is empty (MarcoFalke)

Pull request description:

  This should catch trivial integer sanitizer bugs if the author and all reviewers forget to look for them.

ACKs for top commit:
  brunoerg:
    reACK 0000f55293
  dergoegge:
    reACK 0000f55293

Tree-SHA512: f139b9d56f0cf1aae339c2890721c77c88d1fea77b73d492c1386ec99b4f393c5b664029919ff4a22e4e8a2929f085699a148c6acc2cc3e40df8a72fd39ff474
2023-06-21 10:08:53 +01:00
MarcoFalke
aaaa3aefbd
test: Use TestNode *_path properties where possible
Seems odd to place the burden on test writers to hardcode the chain or
datadir path for the nodes under test.
2023-06-21 08:49:18 +02:00
MarcoFalke
dddd89962b
test: Allow pathlib.Path as RPC argument via authproxy
Also, add datadir_path property to TestNode
2023-06-21 08:48:52 +02:00
MarcoFalke
fa41614a0a
scripted-diff: Use wallets_path and chain_path where possible
Instead of passing the datadir and chain name to os.path.join, just use
the existing properties, which are the same.

-BEGIN VERIFY SCRIPT-
 sed -i --regexp-extended 's|\.datadir, self\.chain, .wallets.|.wallets_path|g' $(git grep -l '\.datadir, self\.chain,')
 sed -i --regexp-extended 's|\.datadir, self\.chain,|.chain_path,|g'            $(git grep -l '\.datadir, self\.chain,')
-END VERIFY SCRIPT-
2023-06-21 08:48:34 +02:00
MarcoFalke
fa493fadfb
test: Use wallet_dir lambda in wallet_multiwallet test where possible
Seems odd to hardcode all parent directory names in the path for no good
reason.

Also, add wallet_path property to TestNode.

Also, rework wallet_backup.py test for scripted-diff in the next commit.
2023-06-21 08:47:54 +02:00
glozow
d1ae96755a
Merge bitcoin/bitcoin#27890: refactor: Make m_count_with_* in CTxMemPoolEntry int64_t, drop UBSAN supp
fa76f0d0ef refactor: Make m_count_with_* in CTxMemPoolEntry int64_t, drop UBSAN supp (MarcoFalke)

Pull request description:

  This is a refactor as long as no signed integer overflow appears. In normal operation and absent bugs, signed integer overflow should never happen in the touched code paths.

  The main benefit of this refactor is to drop the file-wide ubsan suppression `unsigned-integer-overflow:txmempool.cpp`.

  For now, this only changes the internal private representation and the publicly returned type remains `uint64_t`.

ACKs for top commit:
  glozow:
    ACK fa76f0d0ef
  ryanofsky:
    Code review ACK fa76f0d0ef

Tree-SHA512: a09e33a915d60c65d369d44ba1a45ce4a6a76e6dc2bea43216ba02b5eab0b74e214b2c7cc44360493f2c483d18d96e4636b7a75b23050976efc80e38de852c39
2023-06-20 21:38:28 +01:00
Ryan Ofsky
ee22ca59a2
Merge bitcoin/bitcoin#26740: wallet: Migrate wallets that are not in a wallet dir
a1e653828b test: Add test for migrating default wallet and plain file wallet (Andrew Chow)
bdbe3fd76b wallet: Generated migrated wallet's path from walletdir and name (Andrew Chow)

Pull request description:

  This PR fixes an assertion error that is hit during the setup of the new database during migration of a wallet that was not contained in a wallet dir. Also added a test for this case as well as one for migrating the default wallet.

ACKs for top commit:
  ryanofsky:
    Code review ACK a1e653828b
  furszy:
    ACK a1e65382

Tree-SHA512: 96b218c0de8567d8650ec96e1bf58b0f8ca4c4726f5efc6362453979b56b9d569baea0bb09befb3a5aed8d16d29bf75ed5cd8ffc432bbd4cbcad3ac5574bc479
2023-06-20 16:10:44 -04:00
Andrew Chow
e4bbfb2d49
Merge bitcoin/bitcoin#27632: Raise on invalid -debug and -loglevel config options
daa5a658c0 refactor: rename BCLog::BLOCKSTORE to BLOCKSTORAGE (Jon Atack)
cf622b214b doc: release note re raising on invalid -debug/debugexclude/loglevel (Jon Atack)
6cb1c66041 init: remove config option names from translated -loglevel strings (Jon Atack)
2547829272 test: -loglevel raises on invalid values (Jon Atack)
a9c295888b init: raise on invalid loglevel config option (Jon Atack)
b0c3995393 test: -debug and -debugexclude raise on invalid values (Jon Atack)
4c3c19d943 init: raise on invalid debug/debugexclude config options (Jon Atack)

Pull request description:

  and rename BCLog::BLOCKSTORE to BLOCKSTORAGE so the enum is the same as its value like the other BCLog enums.

  Per discussion in bitcoin-core-dev IRC today from https://bitcoin-irc.chaincode.com/bitcoin-core-dev/2023-05-11#921458.

ACKs for top commit:
  achow101:
    ACK daa5a658c0
  ryanofsky:
    Code review ACK daa5a658c0. Just translated string template cleanup since last review
  pinheadmz:
    re-ACK daa5a658c0

Tree-SHA512: 4c107a93d8e8ce4e2ee81d44aec672526ca354ec390b241221067f68204beac8b4ba7a65748bcfa124ff2245c4307fa9243ec4fe0b464d0fa69c787fb322c3cc
2023-06-20 13:55:18 -04:00
MarcoFalke
0000f55293
ci: Run fuzz target even if input folder is empty 2023-06-20 18:19:01 +02:00
glozow
f80db62b2d
Merge bitcoin/bitcoin#27622: Fee estimation: avoid serving stale fee estimate
d2b39e09bc test: ensure old fee_estimate.dat not read on restart and flushed (ismaelsadeeq)
cf219f29f3 tx fees, policy: read stale fee estimates with a regtest-only option (ismaelsadeeq)
3eb241a141 tx fees, policy: do not read estimates of old fee_estimates.dat (ismaelsadeeq)
5b886f2b43 tx fees, policy: periodically flush fee estimates to fee_estimates.dat (ismaelsadeeq)

Pull request description:

  Fixes #27555

  The issue arises when an old `fee_estimates.dat` file is sometimes read during initialization.
  Or after an unclean shutdown, the latest fee estimates are not flushed to `fee_estimates.dat`.
  If the fee estimates in the old file are old, they can cause transactions to become stuck in the mempool.
  This  PR ensures that nodes do not use stale estimates from the old file during initialization. If  `fee_estimates.dat`
  has not been updated for 60 hours or more, it is considered stale and will not be read during initialization. To avoid
  having old estimates, the `fee_estimates.dat` file will be flushed periodically every hour. As mentioned #27555

  > "The immediate improvement would be to store fee estimates to disk once an hour or so to reduce the chance of having an old file. From there, this case could probably be detected, and refuse to serve estimates until we sync."

  In addition, I will follow-up PR to persist the `mempoolminfee` across restarts.

ACKs for top commit:
  willcl-ark:
    ACK d2b39e09bc
  instagibbs:
    reACK d2b39e09bc
  glozow:
    ACK d2b39e09bc. One nit if you follow up.

Tree-SHA512: 4f6e0c296995d0eea5cf80c6aefdd79b7295a6a0ba446f2166f32afc105fe4f831cfda1ad3abd13c5c752b4fbea982cf4b97eaeda2af1fd7184670d41edcfeec
2023-06-20 16:48:29 +01:00
Sebastian Falbesoner
1a572ce7d6 test: refactor: introduce generate_keypair helper with WIF support
In functional tests it is a quite common scenario to generate fresh
elliptic curve keypairs, which is currently a bit cumbersome as it
involves multiple steps, e.g.:

    privkey = ECKey()
    privkey.generate()
    privkey_wif = bytes_to_wif(privkey.get_bytes())
    pubkey = privkey.get_pubkey().get_bytes()

Simplify this by providing a new `generate_keypair` helper function that
returns the private key either as `ECKey` object or as WIF-string
(depending on the boolean `wif` parameter) and the public key as
byte-string; these formats are what we mostly need (currently we don't
use `ECPubKey` objects from generated keypairs anywhere).

With this, most of the affected code blocks following the pattern above
can be replaced by one-liners, e.g.:

    privkey, pubkey = generate_keypair(wif=True)

Note that after this commit, the only direct uses of `ECKey` remain in
situations where we want to set the private key explicitly, e.g. in
MiniWallet (test/functional/test_framework/wallet.py) or the test for
the signet miner script (test/functional/tool_signet_miner.py).
2023-06-19 17:38:14 +02:00
virtu
61f4b9b7ad Manage exceptions in bcc callback functions
Exceptions are not propagated in ctype callback functions used by bcc.
This means an AssertionError exception raised by check_equal() to signal
a failed assertion is not getting caught and properly logged. Instead,
the error is logged to stdout and execution of the handler stops.

The current workaround to check whether all check_equal() assertions in
a callback succeeded is to increment a success counter after the
assertions (which only gets incremented if none exception is raised and
stops execution). Then, outside the callback, the success counter can be
used to check whether a callback executed successfully.

One issue with the described workaround is that when an exception
occurs, there is no way of telling which of the check_equal() statements
caused the exception; moreover, there is no way of inspecting how the
pieces of data that got compared in check_equal() differed (often
a crucial clue when debugging what went wrong).

Two fixes to this problem come to mind. The first involves having the
callback function make event data accessible outside the callback and
inspecting the event using check_equal() outside the callback. This
solution still requires a counter in the callback to tell whether
a callback was actually executed or if instead the call to
perf_buffer_poll() timed out.

The second fix entails wrapping all relevant check_equal() statements
inside callback functions into try-catch blocks and manually logging
AssertionErrors. While not as elegant in terms of design, this approach
can be more pragmatic for more complex tests (e.g., ones involving
multiple events, events of different types, or the order of events).

The solution proposed here is to select the most pragmatic fix on
a case-by-case basis: Tests in interface_usdt_net.py,
interface_usdt_mempool.py and interface_usdt_validation.py have been
refactored to use the first approach, while the second approach was
chosen for interface_usdt_utxocache.py (partly to provide a reference
for the second approach, but mainly because the utxocache tests are the
most intricate tests, and refactoring them to use the first approach
would negatively impact their readability). Lastly,
interface_usdt_coinselection.py was kept unchanged because it does not
use check_equal() statements inside callback functions.
2023-06-19 14:38:32 +02:00
fanquake
7f0b79ea13
Merge bitcoin/bitcoin#27906: doc: test: update TestShell instructions
14405e8d4d doc: test: update TestShell instructions (ismaelsadeeq)

Pull request description:

  Fixes  #27904

  From  #27904 and IRC.
  Update [Testshell instructions ](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test-shell.md#2-importing-testshell-from-the-bitcoin-core-repository)

  E.g `TestShell.setup()` throws
  ```
  AttributeError: type object 'TestShell' has no attribute 'setup'
  ```
  Parentheses are missing, it should be `TestShell().setup()`

ACKs for top commit:
  Sjors:
    utACK 14405e8d4d
  brunoerg:
    crACK 14405e8d4d
  hernanmarino:
    utACK 14405e8d4d

Tree-SHA512: ffe5fa1103a3b00ef0ee99879adae967b0da07cb8f8451c4c261b0a70b3b666af7aeaacd6f46f85a84ee5e9c7c7ed49700209b5b1f124d7a76efc420ad5c9cd9
2023-06-18 12:48:46 +02:00
ismaelsadeeq
14405e8d4d doc: test: update TestShell instructions
add missing parentheses `TestShell.method` should be `TestShell().method`.
2023-06-16 22:55:36 +01:00
Andrew Chow
f0758d8a66
Merge bitcoin/bitcoin#27757: rpc: remove deprecated "warning" field from {create,load,restore,unload}wallet
5524fa00fa doc: add release note about removal of `deprecatedrpc=walletwarningfield` flag (Sebastian Falbesoner)
5c77db7354 Restorewallet/createwallet help documentation fixups/improvements (Jon Atack)
a00ae31fcc rpc: remove deprecated "warning" field from {create,load,restore,unload}wallet (Sebastian Falbesoner)

Pull request description:

  The "warning" string field for wallet creating/loading RPCs (`createwallet`, `loadwallet`, `unloadwallet` and `restorewallet`) has been deprecated with the configuration option `-deprecatedrpc=walletwarningfield` in PR #27279 (released in v25.0). For the next release v26.0, the field and the configuration option can be removed.

ACKs for top commit:
  achow101:
    ACK 5524fa00fa
  jonatack:
    ACK 5524fa00fa

Tree-SHA512: 8212f72067d08095304018b8a95d2ebef630004b65123483fbbfb078cc5709c2d825bbc35b16ea5f6b28ae7377347382d7e9afaf7bdbf0575d2c229d970784de
2023-06-16 15:11:44 -04:00
fanquake
32e2ffc393
Remove the syscall sandbox
After initially being merged in #20487, it's no-longer clear that an
internal syscall sandboxing mechanism is something that Bitcoin Core
should have/maintain, especially when compared to better
maintained/supported alterantives, i.e firejail.

Note that given where it's used, the sandbox also gets dragged into the
kernel.

There is some related discussion in #24771.

This should not require any sort of deprecation, as this was only ever
an opt-in, experimental feature.

Closes #24771.
2023-06-16 10:38:19 +01:00
Andrew Chow
b3db18a012
Merge bitcoin/bitcoin#27712: test: p2p: check misbehavior for non-continuous headers messages
a97c59f12d test: p2p: check misbehavior for non-continuous headers messages (Sebastian Falbesoner)

Pull request description:

  This PR adds missing test coverage for a peer sending a `headers` message where the headers don't connect to each other, which should be treated as misbehaving (not disconnecting though, as the score increase is only 20). The relevant code path is `PeerManagerImpl::ProcessHeadersMessage` -> `PeerManagerImpl::CheckHeadersPoW` -> `PeerManagerImpl::CheckHeadersAreContinuous`:

  17acb2782a/src/net_processing.cpp (L2415-L2419)

  17acb2782a/src/net_processing.cpp (L2474-L2484)

ACKs for top commit:
  sr-gi:
    ACK a97c59f12d
  achow101:
    ACK a97c59f12d
  instagibbs:
    ACK a97c59f12d

Tree-SHA512: 3f8d6a2492e5c8b63c7b11be2e4ec455f83581b2c58f2d4e705baadfe8d7c6377296d6cd0eda679d291a13d8930b09443f8e3d183795df34b780c703d5d3aeb3
2023-06-15 15:11:32 -04:00
dimitaracev
6779e6ed7f test: clean up is node stopped 2023-06-15 14:14:22 +02:00
ismaelsadeeq
d2b39e09bc test: ensure old fee_estimate.dat not read on restart and flushed
This commit adds tests to ensure that old fee_estimates.dat files
are not read and that fee_estimates are periodically flushed to the
fee_estimates.dat file.

Additionaly it tests the -regtestonly option -acceptstalefeeestimates.
2023-06-14 22:40:20 +01:00
MarcoFalke
fa76f0d0ef
refactor: Make m_count_with_* in CTxMemPoolEntry int64_t, drop UBSAN supp
This is a refactor as long as no signed integer overflow appears. In
normal operation and absent bugs, signed integer overflow should never
happen in the touched code paths.

The main benefit of this refactor is to drop the file-wide ubsan
suppression unsigned-integer-overflow:txmempool.cpp.

For now, this only changes the internal private representation and the
publicly returned type remains uint64_t.
2023-06-14 23:15:20 +02:00
Hennadii Stepanov
28fff06afe
test: Make linter to look for BOOST_ASSERT macros
The `BOOST_ASSERT` macro requires to `#include boost/assert.hpp`.
2023-06-14 16:26:11 +01:00
Jon Atack
2547829272 test: -loglevel raises on invalid values 2023-06-14 08:27:46 -06:00
Jon Atack
b0c3995393 test: -debug and -debugexclude raise on invalid values 2023-06-14 08:27:46 -06:00
Ryan Ofsky
6663c802fe
Merge bitcoin/bitcoin#25634: wallet, tests: Expand and test when the blank wallet flag should be un/set
cdba23db35 wallet: Document blank flag use in descriptor wallets (Ryan Ofsky)
43310200dc wallet: Ensure that the blank wallet flag is unset after imports (Andrew Chow)
e9379f1ffa rpc, wallet: Include information about blank flag (Andrew Chow)

Pull request description:

  The `blank` wallet flag is used to indicate that the wallet intentionally does not have any keys, scripts, or descriptors, and it prevents the automatic generation of those things for such a wallet. Once the wallet contains any of those data, it is unnecessary, and possibly incorrect, to have `blank` set. This PR fixes a few places where this was not properly happening. It also adds a test for this unset behavior.

ACKs for top commit:
  S3RK:
    reACK cdba23db35
  ryanofsky:
    Code review ACK cdba23db35. Only change since last review is dropping the commit which makes createwallet RPC set BLANK flag automatically when DISABLE_PRIVATE_KEYS flag is set

Tree-SHA512: 85bc2a9754df0531575d5c8f4ad7e8f38dcd50083dc29b3283dacf56feae842e81f34654c5e1781f2dadb0560ff80e454bbc8ca3b2d1fab1b236499ae9abd7da
2023-06-14 09:30:39 -04:00
Andrew Chow
43310200dc wallet: Ensure that the blank wallet flag is unset after imports 2023-06-13 15:11:41 -04:00
Andrew Chow
58b36fc303
Merge bitcoin/bitcoin#23962: Use int32_t type for most transaction size/weight values
3ef756a5b5 Remove txmempool implicit-integer-sign-change sanitizer suppressions (Hennadii Stepanov)
d2f6d2a95a Use `int32_t` type for most transaction size/weight values (Hennadii Stepanov)

Pull request description:

  From bitcoin/bitcoin#23957 which has been incorporated into this PR:
  > A file-wide suppression is problematic because it will wave through future violations, potentially bugs.
  >
  > Fix that by using per-statement casts.
  >
  > This refactor doesn't change behavior because the now explicit casts were previously done implicitly.
  >
  > Similar to commit 8b5a4de904

ACKs for top commit:
  achow101:
    ACK 3ef756a5b5
  0xB10C:
    ACK 3ef756a5b5. I've focused my testing and code review on the tracepoint related changes. The docs, the test, and the mempool_monitor.py demo script are updated. I ran the `interface_usdt_mempool.py` test and the `mempool_monitor.py` script. The `mempool_monitor.py` output looks correct.
  Xekyo:
    codereview ACK 3ef756a5b5
  ryanofsky:
    Code review ACK 3ef756a5b5. Since last review, just rebased with more type changes in test and tracing code

Tree-SHA512: 397407f72165b6fb85ff1794eb1447836c4f903efed1a05d7a9704c88aa9b86f330063964370bbd59f6b5e322e04e7ea8e467805d58dce381e68f7596433330f
2023-06-13 10:37:25 -04:00
fanquake
8de9bb7a5a
Merge bitcoin/bitcoin#27864: test: fix intermittent failure in p2p_leak_tx.py
ee2417ed61 test: fix intermittent failure in p2p_leak_tx.py (Martin Zumsande)

Pull request description:

  Fixes #27860

  The problem was that the replacement tx `tx_b` would sometimes be sent out to the inbound peer after the `notfound`, so that threre  would be an unexpected `tx` message and the test fails.

  ```
   node0 2023-06-12T12:48:24.903204Z [msghand] [net.cpp:2856] [PushMessage] [net] sending notfound (73 bytes) peer=1
   node0 2023-06-12T12:48:24.903916Z [msghand] [net.cpp:2856] [PushMessage] [net] sending tx (133 bytes) peer=1
                                       File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_leak_tx.py", line 74, in test_notfound_on_replaced_tx
                                         assert "tx" not in inbound_peer.last_message

  ```

  Fix this by letting the peer wait for the initial broadcast of the replacement tx before continuing with the test.

ACKs for top commit:
  MarcoFalke:
    lgtm ACK ee2417ed61

Tree-SHA512: ecc8fb44cac6097a949e4ee622f6f654f49851d7966359532ab3af4c5ed9d587bf08110820b473a616cde3ae6fc8d0da9bb3cee39347655a8c433e819d4d1065
2023-06-13 09:43:53 +01:00
Andrew Chow
d80348ccb6
Merge bitcoin/bitcoin#27853: rest: bugfix, fix crash error when calling /deploymentinfo
7d452d826a test: add coverage for `/deploymentinfo` passing a blockhash (brunoerg)
ce887eaf49 rest: bugfix, fix crash error when calling `/deploymentinfo` (brunoerg)

Pull request description:

  Calling `/deploymentinfo` passing a valid blockhash makes bitcoind to crash. It happens because we're pushing a JSON value of type array when it expects type object. See:
  ```cpp
  jsonRequest.params = UniValue(UniValue::VARR);
  ```
  ```cpp
  jsonRequest.params.pushKV("blockhash", hash_str);
  ```

  This PR fixes it by changing `pushKV` to `push_back` and adds more test coverage.

ACKs for top commit:
  achow101:
    ACK 7d452d826a
  stickies-v:
    ACK 7d452d826a

Tree-SHA512: f01551e556aba2380c3eaed0bc59057304302c202d317d7c1eec5f7ef839851f672aed80819a8719cb1cbbad2aad735d6d44314ac7d6d98bff8217f5a16c312b
2023-06-12 18:34:42 -04:00
Andrew Chow
a1e653828b test: Add test for migrating default wallet and plain file wallet 2023-06-12 15:14:42 -04:00
Hennadii Stepanov
3ef756a5b5
Remove txmempool implicit-integer-sign-change sanitizer suppressions 2023-06-12 19:48:47 +01:00
Hennadii Stepanov
d2f6d2a95a
Use int32_t type for most transaction size/weight values
This change gets rid of a few casts and makes the following commit diff
smaller.
2023-06-12 19:47:19 +01:00
Martin Zumsande
ee2417ed61 test: fix intermittent failure in p2p_leak_tx.py 2023-06-12 14:46:15 -04:00