Commit Graph

27919 Commits

Author SHA1 Message Date
Hennadii Stepanov
d8239362e2
build, doc: Drop libcap-dev from macOS cross-compiling dependencies 2021-03-03 22:38:45 +02:00
Wladimir J. van der Laan
47b99ab1a9
Merge #20406: util: Avoid invalid integer negation in FormatMoney and ValueFromAmount
1f05dbd06d util: Avoid invalid integer negation in ValueFromAmount: make ValueFromAmount(const CAmount& n) well-defined also when n is std::numeric_limits<CAmount>::min() (practicalswift)
7cc75c9ba3 util: Avoid invalid integer negation in FormatMoney: make FormatMoney(const CAmount& n) well-defined also when n is std::numeric_limits<CAmount>::min() (practicalswift)

Pull request description:

  Avoid invalid integer negation in `FormatMoney` and `ValueFromAmount`.

  Fixes #20402.

  Before this patch:

  ```
  $ CC=clang CXX=clang++ ./configure --with-sanitizers=undefined
  $ make -C src/ test/test_bitcoin
  $ src/test/test_bitcoin -t rpc_tests/rpc_format_monetary_values -t util_tests/util_FormatMoney
  core_write.cpp:21:29: runtime error: negation of -9223372036854775808 cannot be represented in type 'CAmount'
    (aka 'long'); cast to an unsigned type to negate this value to itself
  SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior core_write.cpp:21:29 in
  test/rpc_tests.cpp(186): error: in "rpc_tests/rpc_format_monetary_values":
    check ValueFromAmount(std::numeric_limits<CAmount>::min()).write() == "-92233720368.54775808" has failed
    [--92233720368.-54775808 != -92233720368.54775808]
  util/moneystr.cpp:16:34: runtime error: negation of -9223372036854775808 cannot be represented in type 'CAmount'
    (aka 'long'); cast to an unsigned type to negate this value to itself
  SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior util/moneystr.cpp:16:34 in
  test/util_tests.cpp(1188): error: in "util_tests/util_FormatMoney":
    check FormatMoney(std::numeric_limits<CAmount>::min()) == "-92233720368.54775808" has failed
    [--92233720368.-54775808 != -92233720368.54775808]
  ```

  After this patch:

  ```
  $ CC=clang CXX=clang++ ./configure --with-sanitizers=undefined
  $ make -C src/ test/test_bitcoin
  $ src/test/test_bitcoin -t rpc_tests/rpc_format_monetary_values -t util_tests/util_FormatMoney
  ```

ACKs for top commit:
  laanwj:
    re-ACK 1f05dbd06d

Tree-SHA512: 5aaeb8e2178f1597921f53c12bdfc2f3d5993d10c41658dcd25943e54e8cc2116a411bc71d928f890b33bc0b3761a8ee4449b0532bce41125b6c60692808c8c3
2021-03-03 19:04:36 +01:00
Wladimir J. van der Laan
cabe63759c
Merge #20877: netinfo: user help and argument parsing improvements
7d3343fb8e cli: update -netinfo help doc following the merge of 882ce251 (Jon Atack)
ef614bb408 cli: small -netinfo simplification and performance improvement (Jon Atack)
6b45ef3233 cli: improve -netinfo invalid argument error message (Jon Atack)
3732404afa cli: warn in help that -netinfo is not intended to be a stable API (Jon Atack)
7afdd72258 cli: enable -netinfo help to run without a remote server (Jon Atack)

Pull request description:

  A few updates, some per IRC discussion today at http://www.erisian.com.au/bitcoin-core-dev/log-2021-01-07.html#l-87 with respect to -netinfo:

  - enable `-netinfo help` to run without a remote server
  - warn in `-netinfo help` that -netinfo is not intended to be a stable API
  - improve the -netinfo invalid argument error message
  - make a performance improvement and simplification I noticed after the merge of #20764
  - update the -netinfo help doc following the merge of #21192
  -----

  How to test manually:  ๐Ÿ”ฌ ๐Ÿงช  ๐Ÿ“ˆ

  1. check out and build this branch locally; if you need help, don't hesitate to refer to https://jonatack.github.io/articles/how-to-review-pull-requests-in-bitcoin-core#pull-down-the-code-locally or https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests
  2. while it is compiling, look at the code changes
  3. stop signet (if it is running) with `./src/bitcoin-cli -signet stop`
  4. once the build is completed, run `./src/bitcoin-cli -signet -netinfo help`
  5. the help should be printed even though the signet server is not running
  6. near the top you should see the new warning, "This human-readable interface will change regularly and is not intended to be a stable API" as well as a bit more description about the integer argument values.
  7. start signet with `./src/bitcoind -signet`
  8. test the improved invalid argument error message if you run `./src/bitcoin-cli -signet -netinfo 256` or `./src/bitcoin-cli -signet -netinfo a` (valid values are from 0 to 255)
  9. leave review feedback or `ACK <commit hash>` -- done ๐Ÿป

ACKs for top commit:
  michaelfolkson:
    Re-ACK 7d3343fb8e
  pinheadmz:
    RE-ACK 7d3343fb8e

Tree-SHA512: 28c5e9f295ffccba5c2a70faac4987d45f35d4758cf8f10daa767e83212316c4cfc65930e4066f7ad627e9d15b92d43439d1ba9c2f755dfde61885c6a70aa155
2021-03-03 15:19:35 +01:00
MarcoFalke
ebd8d66454
Merge #19203: net: Add regression fuzz harness for CVE-2017-18350. Add FuzzedSocket.
366e3e1f89 fuzz: Add FUZZED_SOCKET_FAKE_LATENCY mode to FuzzedSock to allow for fuzzing timeout logic (practicalswift)
b22d4c1607 fuzz: Add fuzzing harness for Socks5(...) (practicalswift)

Pull request description:

  Add [regression fuzz harness](https://twitter.com/kayseesee/status/1205287895923212289) for CVE-2017-18350. This fuzzing harness would have found CVE-2017-18350 within a minute of fuzzing :)

  See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).

  Happy fuzzing :)

ACKs for top commit:
  vasild:
    ACK 366e3e1f89

Tree-SHA512: 5d8e1863b635efd10ccb11678b71472ba1523c3ef16affa7f9cd638635c1a9c307e28f432d5b87eb0c9cd1c3c1aeafbb24fa7ae86fe4e5090fda2e20d542b6ca
2021-03-03 14:41:05 +01:00
fanquake
97a35f3ae5
Merge #21320: build: fix libnatpmp macos cross compile
bd49ac4168 build: fix libnatpmp macos cross compile (fanquake)

Pull request description:

  Currently, our cross-compile of libnatpmp for macOS doesn't work at all.
  The wrong archiver is used, which produces an archive the linker doesn't like.
  This becomes clear when configuring:
  ```bash
  configure:25722: checking for initnatpmp in -lnatpmp
  configure:25747: env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/ubuntu/bitcoin/depends/x86_64-apple-darwin18/native/bin/clang++ --target=x86_64-apple-darwin18 <trim>  -Wl,-headerpad_max_install_names -Wl,-dead_strip -Wl,-dead_strip_dylibs conftest.cpp -lnatpmp   >&5
  ld: archive has no table of contents for architecture x86_64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  ```

  Fix this by using the right `ar` (we do the same for upnp).

  While we're at it, fix the build so that we are using our c/ppflags.
  In practice this basically means building with `-O2` rather than `-Os`.

  Note that this fixes an issue that is also fixed by #21209. However, given there are reservations about updating to use a newer libnatpmp source, we should just fix this for now.

ACKs for top commit:
  hebasto:
    ACK bd49ac4168, tested:

Tree-SHA512: 2efc2c788ef3ebebfbf564ef07b6cf63a72d8a0bccc22b0ba36537216aa575436b7e87088477e85f6d9191ad34f0b13f1c22cf88c90e1cb81641bfee5dc3058a
2021-03-03 19:58:25 +08:00
Wladimir J. van der Laan
b1a457ab1d
Merge #21324: doc: Update build instructions for Fedora
4899fa3abd doc: Update build instructions for Fedora (Hennadii Stepanov)

Pull request description:

  This PR updates build instructions for Fedora, as Fedora 33 has no `libdb4-devel` and `libdb4-cxx-devel` packages in its default repos.

ACKs for top commit:
  practicalswift:
    ACK 4899fa3abd

Tree-SHA512: 0c23e2ce0ea690ed5eeaa56514d4246e0057a77b7c71f28af4ee1e480521d465122f81cea37cc773ce2db4fc189d5ab3c8f8ffdd65f150cc006390aa1e2a4ac8
2021-03-03 12:49:59 +01:00
Wladimir J. van der Laan
dd8f474d52
Merge #19288: fuzz: Add fuzzing harness for TorController
10d4477dae tests: Add fuzzing harness for TorController (practicalswift)
64219c01dc torcontrol: Move TorControlReply, TorControlConnection and TorController to improve testability (practicalswift)

Pull request description:

  Add fuzzing harness for `TorController`.

  See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).

  Happy fuzzing :)

ACKs for top commit:
  laanwj:
    ACK 10d4477dae

Tree-SHA512: 2da4b1000afe0e65a234636b8fbf6a26fe9e257852bd837168ca73aa3575959e9aded19054620439e4ed0b2787c70cad4541a8c2d210f5238d7f5e9e0545b734
2021-03-03 11:50:00 +01:00
MarcoFalke
9212e67463
Merge #21346: doc: install qt5 when building on macOS
bec7f2caf7 doc: install qt5 when building on macOS (fanquake)

Pull request description:

  Brew has updated such that qt now refers to [Qt 6.0.1](https://github.com/Homebrew/homebrew-core/blob/master/Formula/qt.rb). If builders
  install this, configure will not pick up qt. For now, install
  [qt@5 (5.15.2)](https://github.com/Homebrew/homebrew-core/blob/master/Formula/qt@5.rb), until required build system and likely source changes
  are made.

ACKs for top commit:
  hebasto:
    ACK bec7f2caf7, tested on Tested on macOS Big Sur 11.2.2 (20D80).

Tree-SHA512: 86663cfbc68c8c6f5d608d60cd59b37d3faf1e7f33ae17ec2e1a7c076e835eb8200181a17575f121929ea6ecded74b1619096fe5a763106f56de0bdbea9ae4fa
2021-03-03 10:08:01 +01:00
fanquake
fca3e98f64
Merge #18298: build: Fix Qt processing of configure script for depends with DEBUG=1
76f52e3da3 build: Fix Qt processing of configure script for depends with DEBUG=1 (Hennadii Stepanov)

Pull request description:

  This PR:
  - makes the `configure` script correctly pickup Qt if depends is built with `DEBUG=1`:
      - for Windows -- fix #19266
      - for macOS -- fix #16391
  - is an alternative to #18117 (without downsides)

ACKs for top commit:
  fanquake:
    ACK 76f52e3da3. Tested native darwin, and darwin/win cross compile with `DEBUG=1`.

Tree-SHA512: 8fde99302b4b06faf109315bddba9e3063b156c50f8f9863c2bd51718538c719429a63fdced071730c18022f2e559d3b25c1dcec3efa81fe79f657253680956a
2021-03-03 15:48:11 +08:00
fanquake
bec7f2caf7
doc: install qt5 when building on macOS
Brew has updated such that qt now refers to Qt 6.0.1. If builders
install this, configure will not work pick up qt. For now, install
qt@5 (5.15.2), until required build system and likely source changes
are made.
2021-03-03 12:00:06 +08:00
fanquake
4f223e93e9
Merge #21339: build: Make AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER]) unconditional
a4128138b4 build: Make AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER]) unconditional (Hennadii Stepanov)
9fef209945 build, refactor: Fix indentation for if..then..fi (Hennadii Stepanov)

Pull request description:

  #16546 introduced a regression in the `configure`:

  ```
  $ ./autogen.sh
  $ ./configure --disable-wallet --without-utils --without-daemon --without-gui --disable-tests --disable-bench
  ...
  checking whether to build test_bitcoin... no
  checking whether to reduce exports... no
  checking that generated files are newer than configure... done
  configure: error: conditional "ENABLE_EXTERNAL_SIGNER" was never defined.
  Usually this means the macro was only invoked conditionally.
  ```

  This PR fixes this bug, and refactors indentation to make easier to spot similar bugs in the future.

ACKs for top commit:
  Sjors:
    utACK a4128138b4
  fanquake:
    ACK a4128138b4 - this fixes the bug described, and improves readability.

Tree-SHA512: 4469dcc006690f38f93c3cdf8d15b76f5fc8ea76e87a1b5db5ee891dc9851f6ec539f2a6fd02a361aa76baa4f4b2b9fe8289137f5d9734ee5984f265cb131ef5
2021-03-03 10:54:38 +08:00
fanquake
3a8fc51a56
Merge #21333: build: set Unicode true for NSIS installer
9086e0dd3c build: set Unicode true for NSIS installer (fanquake)

Pull request description:

  Now that we are using Focal for gitian builds, and have [NSIS 3.0+ available](https://packages.ubuntu.com/focal/nsis) (also in Guix), we can create installers that [support unicode](https://nsis.sourceforge.io/Docs/Chapter4.html#aunicodetarget).

  Unicode is only becoming the NSIS default [beginning with the 3.07 release](https://nsis.sourceforge.io/Docs/AppendixF.html#v3.07-cl), so we need to set this attribute to get support.

  Should close: #13817

  Gitian builds:
  ```bash
  b8553615b6b4be5e4459e03796e700b30b5d198a7f184f27be6983ff901b5592  bitcoin-9086e0dd3c92-win-unsigned.tar.gz
  a6b024a5a68e0196e8e118168c918285e820f2d0ffe9c38db680580459da8bf3  bitcoin-9086e0dd3c92-win64-debug.zip
  ff4003d4f61127c707e44b5235eaf924b30351f20cde27e775131982a1b4cf92  bitcoin-9086e0dd3c92-win64-setup-unsigned.exe
  1876bee55fa9ea99b91203975c13d0ad8a046b4b58068bde41c977fd1d12de13  bitcoin-9086e0dd3c92-win64.zip
  000f2778f8f166a89b4ab35f155156c1c34800be6e47d29b5308043c50128392  src/bitcoin-9086e0dd3c92.tar.gz
  d650a9b8f2dd1df777bf42439dfcbcf6bc358e30ec148b9992a18b39f76b1ecf  bitcoin-core-win-22-res.yml
  ```

ACKs for top commit:
  laanwj:
    ACK 9086e0dd3c
  hebasto:
    ACK 9086e0dd3c, tested on Windows 10 Pro (20H2, build 19042.804):

Tree-SHA512: cc7b7ca05877571d0a29a7d36e40279f54d886d8ab27facfa722c2ee95a1fc06c2bad8ef1eb1980d283ae981659d737021a46c8f4618e24d510b5ab384990e09
2021-03-03 09:57:52 +08:00
fanquake
0459e7abba
Merge #21336: Make .gitignore ignore src/test/fuzz/fuzz.exe
35da623574 Make .gitignore ignore src/test/fuzz/fuzz.exe (Hennadii Stepanov)

Pull request description:

ACKs for top commit:
  MarcoFalke:
    cr ACK 35da623574
  practicalswift:
    cr ACK 35da623574

Tree-SHA512: 7e88f4f3ad409b38269fcc7b81c075567aa8a5c2365ff8d3e5b148309e6110722eb84d5732a19f90b81e778bf134d93a886287da70dda708c15477f124e87b03
2021-03-03 08:07:54 +08:00
fanquake
0b294c0ba8
Merge #21342: doc: Remove outdated comment
f1f63ac3f8 doc: Remove outdated comment (Hennadii Stepanov)

Pull request description:

  The removed commit has been wrong [since v0.20.0](https://github.com/bitcoin/bitcoin/pull/18331).

ACKs for top commit:
  fanquake:
    ACK - f1f63ac3f8

Tree-SHA512: ef6191fef389fa0ee5e6cf224e4990a1804aeefd1c3e9d9a4870cf46e1833fbb8701c379b6ce4e13caa02ae2f4f86778fa2c1e994c89392c08fcf01701482d7a
2021-03-03 07:20:41 +08:00
practicalswift
366e3e1f89 fuzz: Add FUZZED_SOCKET_FAKE_LATENCY mode to FuzzedSock to allow for fuzzing timeout logic 2021-03-02 21:44:51 +00:00
practicalswift
b22d4c1607 fuzz: Add fuzzing harness for Socks5(...) 2021-03-02 21:43:42 +00:00
Hennadii Stepanov
f1f63ac3f8
doc: Remove outdated comment
The removed commit is wrong since v0.21.0.
2021-03-02 22:14:18 +02:00
Hennadii Stepanov
a4128138b4
build: Make AM_CONDITIONAL([ENABLE_EXTERNAL_SIGNER]) unconditional
This change fixes an error when the value of the "use_boost" variable is
equal to "no".
2021-03-02 19:07:31 +02:00
Hennadii Stepanov
9fef209945
build, refactor: Fix indentation for if..then..fi
This makes easier to spot conditional macros.
2021-03-02 18:55:07 +02:00
practicalswift
1f05dbd06d util: Avoid invalid integer negation in ValueFromAmount: make ValueFromAmount(const CAmount& n) well-defined also when n is std::numeric_limits<CAmount>::min() 2021-03-02 16:05:28 +00:00
practicalswift
7cc75c9ba3 util: Avoid invalid integer negation in FormatMoney: make FormatMoney(const CAmount& n) well-defined also when n is std::numeric_limits<CAmount>::min() 2021-03-02 16:05:28 +00:00
Hennadii Stepanov
35da623574
Make .gitignore ignore src/test/fuzz/fuzz.exe 2021-03-02 16:16:24 +02:00
practicalswift
10d4477dae tests: Add fuzzing harness for TorController 2021-03-02 12:21:32 +00:00
practicalswift
64219c01dc torcontrol: Move TorControlReply, TorControlConnection and TorController to improve testability 2021-03-02 12:21:32 +00:00
Wladimir J. van der Laan
b9f41df1ea
Merge #20685: Add I2P support using I2P SAM
a701fcf01f net: Do not skip the I2P network from GetNetworkNames() (Vasil Dimov)
0181e24439 net: recognize I2P from ParseNetwork() so that -onlynet=i2p works (Vasil Dimov)
b905363fa8 net: accept incoming I2P connections from CConnman (Vasil Dimov)
0635233a1e net: make outgoing I2P connections from CConnman (Vasil Dimov)
9559bd1404 net: add I2P to the reachability map (Vasil Dimov)
76c35c60f3 init: introduce I2P connectivity options (Vasil Dimov)
c22daa2ecf net: implement the necessary parts of the I2P SAM protocol (Vasil Dimov)
5bac7e45e1 net: extend Sock with a method to check whether connected (Vasil Dimov)
42c779f503 net: extend Sock with methods for robust send & read until terminator (Vasil Dimov)
ea1845315a net: extend Sock::Wait() to report a timeout (Vasil Dimov)
78fdfbea66 net: dedup MSG_NOSIGNAL and MSG_DONTWAIT definitions (Vasil Dimov)
34bcfab562 net: move the constant maxWait out of InterruptibleRecv() (Vasil Dimov)
cff65c4a27 net: extend CNetAddr::SetSpecial() to support I2P (Vasil Dimov)
f6c267db3b net: avoid unnecessary GetBindAddress() call (Vasil Dimov)
7c224fdac4 net: isolate the protocol-agnostic part of CConnman::AcceptConnection() (Vasil Dimov)
1f75a653dd net: get the bind address earlier in CConnman::AcceptConnection() (Vasil Dimov)
25605895af net: check for invalid socket earlier in CConnman::AcceptConnection() (Vasil Dimov)
545bc5f81d util: fix WriteBinaryFile() claiming success even if error occurred (Vasil Dimov)
8b6e4b3b23 util: fix ReadBinaryFile() returning partial contents (Vasil Dimov)
4cba2fdafa util: extract {Read,Write}BinaryFile() to its own files (Vasil Dimov)

Pull request description:

  Add I2P support by using the [I2P SAM](https://geti2p.net/en/docs/api/samv3) protocol. Unlike Tor, for incoming connections we get the I2P address of the peer (and they also receive ours when we are the connection initiator).

  Two new options are added:

  ```
    -i2psam=<ip:port>
         I2P SAM proxy to reach I2P peers and accept I2P connections (default:
         none)

    -i2pacceptincoming
         If set and -i2psam is also set then incoming I2P connections are
         accepted via the SAM proxy. If this is not set but -i2psam is set
         then only outgoing connections will be made to the I2P network.
         Ignored if -i2psam is not set. Notice that listening for incoming
         I2P connections is done through the SAM proxy, not by binding to
         a local address and port (default: true)
  ```

  # Overview of the changes

  ## Make `ReadBinary()` and `WriteBinary()` reusable

  We would need to dump the I2P private key to a file and read it back later. Move those two functions out of `torcontrol.cpp`.

  ```
  util: extract {Read,Write}BinaryFile() to its own files
  util: fix ReadBinaryFile() returning partial contents
  util: fix WriteBinaryFile() claiming success even if error occurred
  ```

  ## Split `CConnman::AcceptConnection()`

  Most of `CConnman::AcceptConnection()` is agnostic of how the socket was accepted. The other part of it deals with the details of the `accept(2)` system call. Split those so that the protocol-agnostic part can be reused if we accept a socket by other means.

  ```
  net: check for invalid socket earlier in CConnman::AcceptConnection()
  net: get the bind address earlier in CConnman::AcceptConnection()
  net: isolate the protocol-agnostic part of CConnman::AcceptConnection()
  net: avoid unnecessary GetBindAddress() call
  ```

  ## Implement the I2P [SAM](https://geti2p.net/en/docs/api/samv3) protocol (not all of it)

  Just the parts that would enable us to make outgoing and accept incoming I2P connections.

  ```
  net: extend CNetAddr::SetSpecial() to support I2P
  net: move the constant maxWait out of InterruptibleRecv()
  net: dedup MSG_NOSIGNAL and MSG_DONTWAIT definitions
  net: extend Sock::Wait() to report a timeout
  net: extend Sock with methods for robust send & read until terminator
  net: extend Sock with a method to check whether connected
  net: implement the necessary parts of the I2P SAM protocol
  ```

  ## Use I2P SAM to connect to and accept connections from I2P peers

  Profit from all of the preceding commits.

  ```
  init: introduce I2P connectivity options
  net: add I2P to the reachability map
  net: make outgoing I2P connections from CConnman
  net: accept incoming I2P connections from CConnman
  net: recognize I2P from ParseNetwork() so that -onlynet=i2p works
  net: Do not skip the I2P network from GetNetworkNames()
  ```

ACKs for top commit:
  laanwj:
    re-ACK a701fcf01f
  jonatack:
    re-ACK a701fcf01f reviewed diff per `git range-diff ad89812 2a7bb34 a701fcf`, debug built and launched bitcoind with i2pd v2.35 running a dual I2P+Torv3 service with the I2P config settings listed below (did not test `onlynet=i2p`); operation appears nominal (same as it has been these past weeks), and tested the bitcoind help outputs grepping for `-i i2p` and the rpc getpeerinfo and getnetworkinfo helps

Tree-SHA512: de42090c9c0bf23b43b5839f5b4fc4b3a2657bde1e45c796b5f3c7bf83cb8ec6ca4278f8a89e45108ece92f9b573cafea3b42a06bc09076b40a196c909b6610e
2021-03-02 11:50:13 +01:00
MarcoFalke
cfce346508
Merge #21310: zmq test: fix sync-up by matching notification to generated block
8a8c6383f6 zmq test: fix sync-up by matching notification to generated block (Sebastian Falbesoner)

Pull request description:

  This is a follow-up PR for #21008, fixes #21216.

  In the course of investigating the problem with jnewbery (analyzing the Cirrus log https://cirrus-ci.com/task/4660108304056320), it turned out that the "sync up" procedure of repeatedly generating a block and waiting for a notification with timeout is too brittle in its current form, as the following scenario could happen:

  - generate block A
  - receive notification, timeout happens => repeat procedure
  - generate block B
  - node publishes block A notification
  - receive notification, we receive the one caused by block A (!!!) => sync-up procedure is completed
  - node publishes block B notification
  - the actual test starts
  - on the first notification reception, the one caused by block B is received, rather than the one actually caused by test code => assertion failure

  This change in the PR ensures that after each test block generation, we wait for the notification that is actually caused by that block and ignore others from possibly earlier blocks. The matching is kind of ugly, it assumes that one out of four components in the block is contained in the notification: the block hash, the tx id, the raw block data or the raw transaction data. (Unfortunately we have to support all publisher topics.)

  I'm aware that this is quite a lot of code now only for establishing a robust test setup. OTOH I wouldn't know of a better method right now, suggestions are very welcome.

  Note for potential reviewers: for both reproducing the issue on master branch and verifying on PR branch, one can simply generate two blocks in the sync-up procedure rather than one.

ACKs for top commit:
  MarcoFalke:
    Concept ACK 8a8c6383f6

Tree-SHA512: a2eb78ba06dfd0fda7b1c111b6bbfb5dab4ab08500cc19c7ea02c3239495d5c74cc7d45250a8b3ecc78ca42d97ee6719bf73db8a137839e5e09a3cfcf08ed29e
2021-03-02 11:30:17 +01:00
fanquake
9086e0dd3c
build: set Unicode true for NSIS installer
Now that we are using Focal for gitian builds, and have NSIS 3.0+
available (also in Guix), we can create installers that support unicode.

Unicode is only becoming the default beginning with the 3.07 release,
https://nsis.sourceforge.io/Docs/AppendixF.html#v3.07-cl,
so we need to set this attribute to get support.

Should close: #13817

Co-authored-by: Chun Kuan Lee <ken2812221@gmail.com>
2021-03-02 16:36:21 +08:00
Hennadii Stepanov
4899fa3abd
doc: Update build instructions for Fedora
Fedora 33 requires adding the Cheese third-party repo to install
libdb4-devel and libdb4-cxx-devel packages.
2021-03-02 10:09:19 +02:00
fanquake
72e6979b31
Merge #21325: lint: Fix spelling errors in comments
fbbb2d4fc1 lint: Fix spelling errors in comments (fyquah)

Pull request description:

  Found some spelling errors while running spelling linter  https://github.com/bitcoin/bitcoin/pull/21245

  This PR fixes them.

ACKs for top commit:
  fanquake:
    ACK fbbb2d4fc1 - I thought we just fixed all of these.

Tree-SHA512: 95525040001f94e899b778c616cb66ebafb679dff88835b66fccf6349d8eb942d6b7374c536a44e393f13156bce9a32ed57e6a82bb02074d2b3cddb2696addb2
2021-03-02 09:43:52 +08:00
fanquake
a28c053c88
Merge #21298: guix: Bump time-machine, glibc, and linux-headers
c33b199456 guix: Bump glibc and linux-headers (Carl Dong)
65363a1bd8 guix: Rebase on 95aca2991b (1.2.0-12.dffc918) (Carl Dong)

Pull request description:

  On bumping the time-machine:

  ```
  A few changes which are useful for us:

  1. 'gnu: cross-gcc-arguments: Enable 128 bit long double for POWER9.' is
     now merged into master.
  2. gnutls is bumped to 3.6.15 and the temporal test failure in
     status-request-revoked is fixed. Note that this does not fix the case
     where one has installed Guix v1.2.0 and is running a substitute-less
     bootstrap build, since the `guix time-machine` command itself has a
     dependency on gnutls v3.6.12 (the one with the broken test) and will
     thus try to build it before attempting to jump forwards in time. This
     does however, mean that those who build a version of Guix that also
     contains this fix will not go backwards in time to build the broken
     gnutls v3.6.12.
  ```

  On bumping the rest:

  ```
  Bump glibc and linux-headers to match those of our Gitian counterparts.

  We also require a glibc >= 2.28 for the test-symbol-check scripts to
  work properly.

  The default BASE-GCC-FOR-LIBC also has to be bumped since glibc 2.31
  requires a gcc >= 6.2
  ```

  This is a prerequisite for #20980

ACKs for top commit:
  fanquake:
    ACK c33b199456 - I think going ahead with this now and to sycn back up to gitian is fine. It will also unblock #20980. Potential code signing related issues can be sorted out in #21239 and later PRs.

Tree-SHA512: 31f022aadb93ba44813b0da005b1f2e5d67d76e8cdcdb53368924d1ea6cb076a21218c26831a6b0dcdcfe33507f54934330489ba557371d740f5587b7d727b95
2021-03-02 09:37:27 +08:00
fanquake
d1abe54fce
Merge #21323: guix, doc: Update default HOSTS value
a0a7a4337d guix, doc: Update default HOSTS value (Hennadii Stepanov)

Pull request description:

  This is a #21089 follow up.

ACKs for top commit:
  fanquake:
    ACK a0a7a4337d

Tree-SHA512: c1813cc2b9212a79fd34d4e25cd0816b58264e1890daf777cd59411bd20fcc9affe312871d06fab1308b8f55c1a78ac1101e631882c18360a4709ecef4529f05
2021-03-02 09:03:06 +08:00
Wladimir J. van der Laan
05e821ee19
Merge #21170: bench: Add benchmark to write JSON into a string
e3e0a2432c Add benchmark to write JSON into a string (Martin Ankerl)

Pull request description:

  The benchmark `BlockToJsonVerbose` only tests generating (and destroying)
  the JSON data structure, but serializing into a string is also a
  performance critical aspect of the RPC calls.

  Extracts test setup into a `struct TestBlockAndIndex`, and uses it in
  both `BlockToJsonVerbose` and `BlockToJsonVerboseWrite`.

  Also, use `ankerl::nanobench::doNotOptimizeAway` to make sure the compiler
  can't optimize the result of the calls away.

  Here are benchmark results on my Intel i7-8700:

  |               ns/op |                op/s |    err% |          ins/op |          cyc/op |    IPC |         bra/op |   miss% |     total | benchmark
  |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:----------
  |       71,807,017.00 |               13.93 |    0.4% |  555,782,961.00 |  220,788,645.00 |  2.517 | 102,279,341.00 |    0.4% |      0.80 | `BlockToJsonVerbose`
  |       27,916,835.00 |               35.82 |    0.1% |  235,084,034.00 |   89,033,525.00 |  2.640 |  42,911,139.00 |    0.3% |      0.32 | `BlockToJsonVerboseWrite`

ACKs for top commit:
  laanwj:
    Code review ACK e3e0a2432c

Tree-SHA512: bc4d6d1588d47d4bd7af8e7908e44b8561bc391a2d73eccd7c0aa37fc40e8a9ce1fa1f3c29b416eef24a73c6bce3036839c0bbfe1b8dbd6d1bba3718b7ca5383
2021-03-01 19:12:09 +01:00
Vasil Dimov
a701fcf01f
net: Do not skip the I2P network from GetNetworkNames()
So that help texts include "i2p" in:
* `./bitcoind -help` (in `-onlynet` description)
* `getpeerinfo` RPC
* `getnetworkinfo` RPC

Co-authored-by: Jon Atack <jon@atack.com>
2021-03-01 18:19:47 +01:00
Vasil Dimov
0181e24439
net: recognize I2P from ParseNetwork() so that -onlynet=i2p works 2021-03-01 18:19:47 +01:00
Vasil Dimov
b905363fa8
net: accept incoming I2P connections from CConnman 2021-03-01 18:19:47 +01:00
Vasil Dimov
0635233a1e
net: make outgoing I2P connections from CConnman 2021-03-01 18:19:46 +01:00
Vasil Dimov
9559bd1404
net: add I2P to the reachability map
Update `CNetAddr::GetReachabilityFrom()` to recognize the I2P network so
that we would prefer to advertise our I2P address to I2P peers.
2021-03-01 18:19:46 +01:00
Vasil Dimov
76c35c60f3
init: introduce I2P connectivity options
Introduce two new options to reach the I2P network:

* `-i2psam=<ip:port>` point to the I2P SAM proxy. If this is set then
  the I2P network is considered reachable and we can make outgoing
  connections to I2P peers via that proxy. We listen for and accept
  incoming connections from I2P peers if the below is set in addition to
  `-i2psam=<ip:port>`

* `-i2pacceptincoming` if this is set together with `-i2psam=<ip:port>`
  then we accept incoming I2P connections via the I2P SAM proxy.
2021-03-01 18:19:46 +01:00
Vasil Dimov
c22daa2ecf
net: implement the necessary parts of the I2P SAM protocol
Implement the following commands from the I2P SAM protocol:

* HELLO: needed for all of the remaining ones
* DEST GENERATE: to generate our private key and destination
* NAMING LOOKUP: to convert .i2p addresses to destinations
* SESSION CREATE: needed for STREAM CONNECT and STREAM ACCEPT
* STREAM CONNECT: to make outgoing connections
* STREAM ACCEPT: to accept incoming connections
2021-03-01 18:19:37 +01:00
Vasil Dimov
5bac7e45e1
net: extend Sock with a method to check whether connected
This will be convenient in the I2P SAM implementation.
2021-03-01 17:36:17 +01:00
Vasil Dimov
42c779f503
net: extend Sock with methods for robust send & read until terminator
Introduce two high level, convenience methods in the `Sock` class:

* `SendComplete()`: keep trying to send the specified data until either
  successfully sent all of it, timeout or interrupted.

* `RecvUntilTerminator()`: read until a terminator is encountered (never
  after it), timeout or interrupted.

These will be convenient in the I2P SAM implementation.

`SendComplete()` can also be used in the SOCKS5 implementation instead
of calling `send()` directly.
2021-03-01 17:36:16 +01:00
fyquah
fbbb2d4fc1 lint: Fix spelling errors in comments 2021-03-01 15:24:28 +00:00
Vasil Dimov
ea1845315a
net: extend Sock::Wait() to report a timeout
Previously `Sock::Wait()` would not have signaled to the caller whether
a timeout or one of the requested events occurred since that was not
needed by any of the callers.

Such functionality will be needed in the I2P implementation, thus extend
the `Sock::Wait()` method.
2021-03-01 13:22:18 +01:00
Vasil Dimov
78fdfbea66
net: dedup MSG_NOSIGNAL and MSG_DONTWAIT definitions
Deduplicate `MSG_NOSIGNAL` and `MSG_DONTWAIT` definitions from `net.cpp`
and `netbase.cpp` to `compat.h` where they can also be reused by other
code.
2021-03-01 13:22:17 +01:00
Vasil Dimov
34bcfab562
net: move the constant maxWait out of InterruptibleRecv()
Move `maxWait` out of `InterruptibleRecv()` and rename it to
`MAX_WAIT_FOR_IO` so that it can be reused by other code.
2021-03-01 13:22:17 +01:00
Vasil Dimov
cff65c4a27
net: extend CNetAddr::SetSpecial() to support I2P
Recognize also I2P addresses in the form `base32hashofpublickey.b32.i2p`
from `CNetAddr::SetSpecial()`.

This makes `Lookup()` support them, which in turn makes it possible to
manually connect to an I2P node by using
`-proxy=i2p_socks5_proxy:port -addnode=i2p_address.b32.i2p:port`

Co-authored-by: Lucas Ontivero <lucasontivero@gmail.com>
2021-03-01 13:22:11 +01:00
Vasil Dimov
f6c267db3b
net: avoid unnecessary GetBindAddress() call
Our local (bind) address is already saved in `CNode::addrBind` and there
is no need to re-retrieve it again with `GetBindAddress()`.

Also, for I2P connections `CNode::addrBind` would contain our I2P
address, but `GetBindAddress()` would return something like
`127.0.0.1:RANDOM_PORT`.
2021-03-01 12:57:01 +01:00
Vasil Dimov
7c224fdac4
net: isolate the protocol-agnostic part of CConnman::AcceptConnection()
Isolate the second half of `CConnman::AcceptConnection()` into a new
separate method, which could be reused if we accept incoming connections
by other means than `accept()` (first half of
`CConnman::AcceptConnection()`).
2021-03-01 12:57:01 +01:00
Vasil Dimov
1f75a653dd
net: get the bind address earlier in CConnman::AcceptConnection()
Call `GetBindAddress()` earlier in `CConnman::AcceptConnection()`. That
is specific to the TCP protocol and makes the code below it reusable for
other protocols, if the caller provides `addr_bind`, retrieved by other
means.
2021-03-01 12:57:01 +01:00
Vasil Dimov
25605895af
net: check for invalid socket earlier in CConnman::AcceptConnection()
This check is related to an `accept()` failure. So do the check earlier,
closer to the `accept()` call.

This will allow to isolate the `accept()`-specific code at the beginning
of `CConnman::AcceptConnection()` and reuse the code that follows it.
2021-03-01 12:57:00 +01:00