c5b404e8f1 Add functional tests for flexible whitebind/list (nicolas.dorier)
d541fa3918 Replace the use of fWhitelisted by permission checks (nicolas.dorier)
ecd5cf7ea4 Do not disconnect peer for asking mempool if it has NO_BAN permission (nicolas.dorier)
e5b26deaaa Make whitebind/whitelist permissions more flexible (nicolas.dorier)
Pull request description:
# Motivation
In 0.19, bloom filter will be disabled by default. I tried to make [a PR](https://github.com/bitcoin/bitcoin/pull/16176) to enable bloom filter for whitelisted peers regardless of `-peerbloomfilters`.
Bloom filter have non existent privacy and server can omit filter's matches. However, both problems are completely irrelevant when you connect to your own node. If you connect to your own node, bloom filters are the most bandwidth efficient way to synchronize your light client without the need of some middleware like Electrum.
It is also a superior alternative to BIP157 as it does not require to maintain an additional index and it would work well on pruned nodes.
When I attempted to allow bloom filters for whitelisted peer, my proposal has been NACKed in favor of [a more flexible approach](https://github.com/bitcoin/bitcoin/pull/16176#issuecomment-500762907) which should allow node operator to set fine grained permissions instead of a global `whitelisted` attribute.
Doing so will also make follow up idea very easy to implement in a backward compatible way.
# Implementation details
The PR propose a new format for `--white{list,bind}`. I added a way to specify permissions granted to inbound connection matching `white{list,bind}`.
The following permissions exists:
* ForceRelay
* Relay
* NoBan
* BloomFilter
* Mempool
Example:
* `-whitelist=bloomfilter@127.0.0.1/32`.
* `-whitebind=bloomfilter,relay,noban@127.0.0.1:10020`.
If no permissions are specified, `NoBan | Mempool` is assumed. (making this PR backward compatible)
When we receive an inbound connection, we calculate the effective permissions for this peer by fetching the permissions granted from `whitelist` and add to it the permissions granted from `whitebind`.
To keep backward compatibility, if no permissions are specified in `white{list,bind}` (e.g. `--whitelist=127.0.0.1`) then parameters `-whitelistforcerelay` and `-whiterelay` will add the permissions `ForceRelay` and `Relay` to the inbound node.
`-whitelistforcerelay` and `-whiterelay` are ignored if the permissions flags are explicitly set in `white{bind,list}`.
# Follow up idea
Based on this PR, other changes become quite easy to code in a trivially review-able, backward compatible way:
* Changing `connect` at rpc and config file level to understand the permissions flags.
* Changing the permissions of a peer at RPC level.
ACKs for top commit:
laanwj:
re-ACK c5b404e8f1
Tree-SHA512: adfefb373d09e68cae401247c8fc64034e305694cdef104bdcdacb9f1704277bd53b18f52a2427a5cffdbc77bda410d221aed252bc2ece698ffbb9cf1b830577
fa76285fdd test: Explain why -whitelist is used in feature_fee_estimation (MarcoFalke)
faff85a69a test: Format feature_fee_estimation with pep8 (MarcoFalke)
Pull request description:
ACKs for top commit:
practicalswift:
ACK fa76285fdd -- diff looks correct
Sjors:
ACK fa76285, every bit of clarification helps. It's clear that without `-whitelist` the test becomes extremely slow (it does pass).
Tree-SHA512: 13ec7e4cd0409e7bb76cbcd344e31c0f612c8ce4a1f1ec6ceaedf345f634bc09786ed38d38920c3469b2862c856ee3e5e42534ef90f531bd8dc83c3db3c06417
fa8489a155 test: Add test for BIP30 duplicate tx (MarcoFalke)
77770d95e2 test: Properly serialize BIP34 coinbase height (MarcoFalke)
Pull request description:
This adds a test for BIP30 to check that duplicate txs can exist in the blockchain given the first one was completely spent when the second one is added. (Requested by ajtowns in https://github.com/bitcoin/bitcoin/pull/16333#issuecomment-508604071)
We can not add a test that a later duplicate tx overwrites a previous one, because BIP30 is always enforced on regtest. If someone feels strongly about such a test, some Bitcoin Core code would have to be modified, which can be done in a follow up pull request.
Also, add a commit to fix the BIP34 test failures reported in https://github.com/bitcoin/bitcoin/pull/14633#issue-227712540
ACKs for top commit:
laanwj:
Code review ACK fa8489a155
Tree-SHA512: c707d0bdc93937263876b603425b53322a2a9f9ec3f50716ae2fa9de8ddc644beb22b26c1bfde7f4aab102633e096b354ef380db919176bd2cb44a2828f884aa
faf36838bd test: Avoid hardcoding the chain name in combine_logs (MarcoFalke)
fa8a1d7ba3 test: Adapt test framework for chains other than "regtest" (MarcoFalke)
68f546635d test: Fix “local variable 'e' is assigned to but never used” (Ben Woosley)
Pull request description:
This is required for various work in progress:
* testchains #8994
* signet #16411
* some of my locally written tests
While it will be unused in the master branch as of now, it will make all of those pull requests shorter. Thus review for non-regtest tests can focus on the actual changes and not some test framework changes.
ACKs for top commit:
jonatack:
ACK faf36838bd, ran tests and reviewed the code.
Tree-SHA512: 35add66c12cab68f2fac8f7c7d47c604d3f24eae9336ff78f83e2c92b3dc08a25e7f4217199bac5393dd3fb72f945bba9c001d6fbb8efd298c88858075fcb3d6
fab3658356 [qa] Test that getdata requests work as expected (Suhas Daftuar)
fa883ab35a net: Use mockable time for tx download (MarcoFalke)
Pull request description:
Two commits:
* First commit changes to mockable time for tx download (refactoring, should only have an effect on regtest)
* Second commit adds a test that uses mocktime to test tx download
ACKs for top commit:
laanwj:
code review ACK 16197/commits/fab365835639a3da03f8ad9a58a0db6c6c4c2314
jamesob:
ACK fab3658356
Tree-SHA512: 3a64a3e283ec4bab1f6e506404b11f0a564a5b61d2a7508ae738a61f035e57220484c66e0ae47d847fe9f7e3ff5cc834909d7b34a9bbcea6abe01f8742806908
e6f649cb2c test: Make tests arg type specific (Hennadii Stepanov)
b70cc5d733 Revamp option negating policy (Hennadii Stepanov)
db08edb303 Replace IsArgKnown() with FlagsOfKnownArg() (Hennadii Stepanov)
dde80c272a Use ArgsManager::NETWORK_ONLY flag (Hennadii Stepanov)
9a12733508 Remove unused m_debug_only member from Arg struct (Hennadii Stepanov)
fb4b9f9e3b scripted-diff: Use ArgsManager::DEBUG_ONLY flag (Hennadii Stepanov)
1b4b9422ca scripted-diff: Use Flags enum in AddArg() (Hennadii Stepanov)
265c1b58d8 Add Flags enum to ArgsManager (Hennadii Stepanov)
e0d187dfeb Refactor InterpretNegatedOption() function (Hennadii Stepanov)
e0e18a1017 refactoring: Check IsArgKnown() early (Hennadii Stepanov)
Pull request description:
This PR adds the `Flags` enum to the `ArgsManager` class. Also the `m_flags` member is added to the `Arg` struct. Flags denote an allowed type of an arg value and special hints.
This PR is only a refactoring and does not change behavior.
ACKs for top commit:
jamesob:
ACK e6f649cb2c
MarcoFalke:
ACK e6f649cb2c thanks for adding types to the command line options
Tree-SHA512: b867f8a9cbce2d2473c293d534af662d8cd5be15060ff0682e97af678974bdaac35e8bc6328ccba32f105034bcd38f169b92a6fb67798667891ce14d5d2a2dea
d6b3640ac7 [test] walletcreatefundedpsbt: check RBF is disabled when -walletrbf=0 (Sjors Provoost)
9ed062b568 [doc] rpc: remove "fallback to" from RBF default help (Sjors Provoost)
4fcb698bc2 [rpc] walletcreatefundedpsbt: use wallet default RBF (Sjors Provoost)
Pull request description:
The `walletcreatefundedpsbt` RPC call currently ignores `-walletrbf` and defaults to not use RBF. This PR fixes that.
This PR also replaces UniValue in `ConstructTransaction` with a `bool` in preparation of moving this helper method out of the RPC codebase entirely. This may be a bit overkill, but does slightly simplify it.
Fixes#15878
ACKs for top commit:
achow101:
Code Review ACK d6b3640ac7
l2a5b1:
re-ACK d6b3640
MarcoFalke:
ACK d6b3640ac7
Tree-SHA512: 55b9bccd1ef36b54f6b34793017dc0721103099ad3761b3b04862291ee13d6915915d4dbb1a8567924fa56e5e95dfe10eec070e06701610e70c87f8ea92b2a00
fac2e6a604 test: Fail early on disconnect in mininode.wait_for_* (MarcoFalke)
Pull request description:
The node might crash or disconnect when our mininode waits for data. Due to the crash, the data is guaranteed to never arrive and we can fail early with an assert
ACKs for top commit:
laanwj:
ACK fac2e6a604
Tree-SHA512: 32ca844eb66bd70ea49103d51c76b953242b1886e0834d96fca8840fc984ff40346d0a799adf8f76b03514a783cb9cec69d45e00bdd328c5192c31b5d8d17af2
c5d3787367 Allow createwallet to take empty passwords to make unencrypted wallets (Andrew Chow)
Pull request description:
Allow createwallet to take the empty string as a password and interpret that as leaving the wallet unencrypted. Also warn when that happens.
This fixes a bug where it was not possible to use the `avoid_reuse` option for new unencrypted wallets without using named arguments.Thus this allows more `createwallet` options to be added that can be set on unencrypted wallets when using positional arguments.
ACKs for top commit:
jnewbery:
code review ACK c5d3787367
meshcollider:
re-utACK c5d3787367
ryanofsky:
utACK c5d3787367. Changes since last review are rebasing, concatenating warning strings to avoid discarding warnings, adding release notes, and choosing an unambiguous wallet name for the test.
Tree-SHA512: 146737a728dd614ba94d4b166b27e8c9e195badd1709ccab2315afe59176d9b493dfba9b61c3ed81090f059c7e464d709deb06d99451b9a3fff667f527d6f7c9
faf8318c55 test: Split fundrawtx test into subtests (MarcoFalke)
fa6fba3bc8 test: Make local symbols in run_test members (MarcoFalke)
Pull request description:
This prevents scope-leak of symbols that are supposed to be local to one test case.
Top commit has no ACKs.
Tree-SHA512: 9b2a4ca2cdd631ef915d2f7e6cd62375df9a0919448350aa6e5ae4aa8a8fe3ba53870f7a9a25a57736894b4e3a45e861018253ed2d57d9a64c2bb65fa270fad8
62d3f5057f qa: fix deprecated log.warn in feature_dbcrash test (Jon Atack)
Pull request description:
This clears up the following deprecation message when running test/functional/feature_dbcrash.py:
```
test/functional/feature_dbcrash.py:270:
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
self.log.warn("Node %d never crashed during utxo flush!", i)
```
Git grepping indicates that this was the last remaining use of `log.warn` in the functional tests.
ACKs for top commit:
fanquake:
ACK 62d3f5057f - checked that there were no more occurrences.
Tree-SHA512: 2fe87400f82488e44391f4897876003a98736013e819a7dbc3b3e87a5ffbfba8d5ccab81cf2b7577f40135c95e4db96e93bb8cb24de396efb4ad814fbda09559
This clears up the following deprecation message when running the test:
```
test/functional/feature_dbcrash.py:270: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
self.log.warn("Node %d never crashed during utxo flush!", i)
```
Git grepping indicates that this was the last remaining use of `log.warn` in the functional tests.
42a5e912ee [mempool] log correct messages when CPFP fails (John Newbery)
Pull request description:
Fixes a logging issue introduced in #15681
ACKs for top commit:
laanwj:
ACK 42a5e912ee (+utACK from bluematt that isn't registered because it has no commit id)
Tree-SHA512: ff5f423cc4d22838eea00c5b1d39ceda89cd61474c72f256a97c698eb0ec3f2156a97139f537669376132902c1e3943bf84c356a4b98a9a306b4ec57302c2761
bf3be5297a [qa] Ensure we don't generate a too-big block in p2sh sigops test (Suhas Daftuar)
Pull request description:
There's a bug in the loop that is calculating the block size in the p2sh sigops test -- we start with the size of the block when it has no transactions, and then increment by the size of each transaction we add, without regard to the changing size of the encoding for the number of transactions in the block.
This might be fine if the block construction were deterministic, but the first transaction in the block has an ECDSA signature which can be variable length, so we see intermittent failures of this test when the initial transaction has a 70-byte signature and the block ends up being one byte too big.
Fix this by double-checking the block size after construction.
ACKs for top commit:
jonasschnelli:
utACK bf3be5297a
jnewbery:
tested ACK bf3be5297a
Tree-SHA512: f86385b96f7a6feafa4183727f5f2c9aae8ad70060b574aad13b150f174a17ce9a0040bc51ae7a04bd08f2a5298b983a84b0aed5e86a8440189ebc63b99e64dc
2f7eb772f6 Add RPC bumpfee totalFee deprecation test (Jon Atack)
a92d9ce8cf deprecate totalFee argument in bumpfee RPC call (Gregory Sanders)
Pull request description:
totalFee argument is of questionable use, and should be removed in favor of feerate-based features.
I first moved IsDeprecatedRPCEnabled because `bitcoin-wallet` doesn't link `libbitcoin_server`.
ACKs for top commit:
ryanofsky:
utACK 2f7eb772f6. Only change since last review is leaving IsDeprecatedRPCEnabled in its happy home, and switching to rpcEnableDeprecated instead. (Thanks!)
jonatack:
ACK 2f7eb772f6. Built locally, manually tested rpc bumpfee, help output ([gist](https://gist.github.com/jonatack/863673eacc02f9da39ff6d6712f9d837)), all tests pass. Travis failures appears to be unrelated, the [bitcoin builds are green](https://bitcoinbuilds.org/index.php?build=121).
meshcollider:
Code Review ACK 2f7eb772f6
Tree-SHA512: c97465205ee59575df37894bcbb6c4ecf8858dd8fe9d89503f9342b226768c1dcb553153bc9eb3055f7bf5eb41573e48b8efa57e083cd255793cbe5280f0026a
a47df13471 [qa] Test disconnect block failure -> shutdown (Suhas Daftuar)
4433ed0f73 [validation] Crash if disconnecting a block fails (Suhas Daftuar)
Pull request description:
If we're unable to disconnect a block during normal operation, then that is a
failure of our local system (such as disk failure) or the chain that we are on
(eg CVE-2018-17144), but cannot be due to failure of the (more work) chain that
we're trying to validate.
We should abort rather than stay on a less work chain.
Fixes#14341.
ACKs for top commit:
practicalswift:
utACK a47df13471
TheBlueMatt:
utACK a47df13471. Didn't bother to review the test in detail, it looked fine. Debated whether invalidateblock should ever crash the node, but *not* crashing in the case of hitting a pruned block (which is the only change here) is clearly better, even if there are other cases I'd argue we should crash in.
ryanofsky:
utACK a47df13471. Only change since last review is new comment.
promag:
ACK a47df1347, it takes awhile to quit (RPC connection timeouts) but that's unrelated - hope to fix that soon.
fanquake:
ACK a47df13471
Tree-SHA512: 4dec8cef6e7dbbe513c138fc5821a7ceab855e603ece3c16185b51a3830ab7ebbc844a28827bf64e75326f45325991dcb672f13bd7baede53304f27289c4af8d
-BEGIN VERIFY SCRIPT-
sed -i 's/inline std::string _(const char\* psz)/inline bilingual_str _(const char\* psz)/' src/util/translation.h
sed -i 's/return G_TRANSLATION_FUN ? (G_TRANSLATION_FUN)(psz) : psz;/return bilingual_str{psz, G_TRANSLATION_FUN ? (G_TRANSLATION_FUN)(psz) : psz};/' src/util/translation.h
sed -i 's/\b_("\([^"]\|\\"\)*")/&.translated/g' $(git grep --files-with-matches '\b_("' src)
echo Hard cases - multiline strings.
sed -i 's/"Visit %s for further information about the software.")/&.translated/g' src/init.cpp
sed -i "s/\"Only rebuild the block database if you are sure that your computer's date and time are correct\")/&.translated/g" src/init.cpp
sed -i 's/" restore from a backup.")/&.translated/g' src/wallet/db.cpp
sed -i 's/" or address book entries might be missing or incorrect.")/&.translated/g' src/wallet/wallet.cpp
echo Special case.
sed -i 's/_(COPYRIGHT_HOLDERS)/&.translated/' src/util/system.cpp test/lint/lint-format-strings.py
-END VERIFY SCRIPT-
c3dfc91032 test: Skip flaky p2p_invalid_messages test on macOS (Fabian Jahr)
Pull request description:
This mitigates https://github.com/bitcoin/bitcoin/issues/15400
I had a look into the issue today and this seems to be the best we can do given that the root causes some unexpected custom error code from the macOS kernel that python/asyncio doesn't know how to handle properly yet.
Top commit has no ACKs.
Tree-SHA512: 20a0551d45c405b6eb0ae58190b701c7026c52eff6c434bc678f723a4dabf0074e5b52a8bb3d51ee7132dc29419d1e67a24696761c444c62cd4d429ec768e67d
fa6f402bde Call node->initError instead of InitError from GUI code (Russell Yanofsky)
fad2502240 init: Use InitError for all errors in bitcoind/qt (MarcoFalke)
Pull request description:
Using the same InitError for startup error in the daemon and the gui makes it possible to run the tests with the gui again:
```sh
BITCOIND=bitcoin-qt ./test/functional/test_runner.py feature_includeconf feature_config_args
ACKs for top commit:
hebasto:
ACK fa6f402bde
ryanofsky:
utACK fa6f402bde. Only changes since last review are removing more includes and adding Node::initError method to avoid accessing node `InitError` function and global variables from GUI code.
Tree-SHA512: bd19e08dcea4019dfe40356bc5c63cb583cefed54b6c9dcfb82f1b5b00308d8e2b363549afcaea5e93bf83864dbe0917400c3b70f43a8a5bdff45c9cd34cc294
50cede3f5a [mempool] Allow one extra single-ancestor transaction per package (Matt Corallo)
Pull request description:
This implements the proposed policy change from [1], which allows
certain classes of contract protocols involving revocation
punishments to use CPFP. Note that some such use-cases may still
want some form of one-deep package relay, though even this alone
may greatly simplify some lightning fee negotiation.
[1] https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-November/016518.html
ACKs for top commit:
ajtowns:
ACK 50cede3f5a -- looked over code again, compared with previous commit, compiles, etc.
sdaftuar:
ACK 50cede3f5a
ryanofsky:
utACK 50cede3f5a. Changes since last review: adding EXTRA_DESCENDANT_TX_SIZE_LIMIT constant, changing max ancestor size from 1,000,000 to nLimitAncestorSize constant (101,000), fixing test comment and getting rid of unused test node.
Tree-SHA512: b052c2a0f384855572b4579310131897b612201214b5abbb225167224e4f550049e300b471dbf320928652571e92ca2d650050b7cf39ac92b3bc1d2bcd386c1c
bead32e31e Add release notes for DEFAULT_BLOOM change (Matt Corallo)
f27309f55c Move DEFAULT_PEERBLOOMFILTERS from validation.h to net_processing.h (Matt Corallo)
5efcb77283 Disable bloom filtering by default. (Matt Corallo)
Pull request description:
BIP 37 bloom filters have been well-known to be a significant DoS
target for some time. However, in order to provide continuity for
SPV clients relying on it, the NODE_BLOOM service flag was added,
and left as a default, to ensure sufficient nodes exist with such a
flag.
NODE_BLOOM is, at this point, well-established and, as long as
there exist 0.18 nodes with default config (which I'd anticipate
will be true for many years), will be available from some peers. By
that time, the continued slowdown of BIP 37-based filtering will
likely have rendered it useless (though this is already largely the
case). Further, BIP 37 was deliberately never updated to support
witness-based filtering as newer wallets are expected to migrate to
some yet-to-be-network-exposed filters.
ACKs for top commit:
jnewbery:
ACK bead32e31e
kallewoof:
ACK bead32e31e
Tree-SHA512: ecd901898e8efe1a7c82b471af0acc2373c2282ac633eb58d9aae7c35deda1999d0f79fb0485e6cecbda7246aeda00206cd82c7fa36866e2ac64705ba93f9390
e142ee03e7 doc: describe how to pass wildcard names to test runner (Jon Atack)
6a7a70b8cf test: enable passing wildcards with path to test runner (Jon Atack)
Pull request description:
Currently, passing wildcard testname args to the test runner from outside the test/functional/ directory does not work, even though developers expect it to. See these recent IRC discussions for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323) and http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-11.html#l-134.
1. [BUGFIX] Enable passing wildcards with paths. Examples:
- `test/functional/test_runner.py test/functional/wallet*`
- `functional/test_runner.py functional/wallet*`
- `test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*`
- A current limitation this PR does not change: 9 test files with arguments in their filename are not picked up by wildcard search.
2. [Docs] Describe how to pass wildcard names (multiple and with paths) to the test runner in test/README.md.
ACKs for top commit:
jnewbery:
tested ACK e142ee03e7
jachiang:
Tested ACK e142ee03e7. Thanks a lot for this fix!
MarcoFalke:
ACK e142ee03e7, fine with me
Tree-SHA512: cb3d994880cdc9b8918546b573a25faa5b4c7339826ac7cfe20f076aac6e731a34271609c0cf5a7ee5e4a2d5ae205298319d24bf36ef5b5d569a1a0c57883e54
fa89badf88 test: Require standard txs in regtest (MarcoFalke)
fa9b419160 test: Add test that mainnet requires standard txs (MarcoFalke)
fa613ca0a8 chainparams: Remove unused fMineBlocksOnDemand (MarcoFalke)
Pull request description:
I don't see a reason why regtest should allow non-standard txs, as it makes testing mainnet behaviour such as #15846 unnecessarily hard and unintuitive.
Of course, testnet policy remains unchanged to allow propagation of non-standard txs.
ACKs for top commit:
ajtowns:
ACK fa89badf88
Tree-SHA512: c4c675affb054868850bd2683aa07f4c741a448cbacb2ea8334191e105f426b0790fe6a468be61e9c5880d24154f7bf1c7075051697172dce92180c1bc3a1c90
