8b8d8eeae9 Remove travis_wait from lint script (Graham Krizek)
Pull request description:
Using the `travis_wait` command in conjunction with `set -o errexit` causes problems. The `travis_wait` command will correctly log the command's output if successful, but if the command fails the process exits before the `travis_wait` command can dump the logs. This will hide important debugging information like error messages and stack traces. We ran into this in #15196 and it was very hard to debug because output was being suppressed.
`travis_wait` was being used because the `contrib/verify-commits/verify-commits.py` script can sometimes run for a long time without producing any output. If a script runs for 10 minutes without logging anything, the CI run times out. The `travis_wait` command will extend this timeout by logging a message for you, while sending stderr and stdout to a file.
This PR removes the `travis_wait` command from our CI system and adds additional logging to the `verify-commits.py` script so it doesn't make Travis timeout.
ACKs for commit 8b8d8e:
MarcoFalke:
utACK 8b8d8eeae9
Tree-SHA512: 175a8dd3f4d4e03ab272ddba94fa8bb06875c9027c3f3f81577feda4bc8918b5f0e003a19027f04f8cf2d0b56c68633716a6ab23f95b910121a8d1132428767d
b8705a0915 Set 'distro' explicitly (Hennadii Stepanov)
Pull request description:
The [gitian-builder](https://github.com/devrandom/gitian-builder) implicitly uses `ubuntu` as a default distro.
[bin/gbuild#L237](81edd2fc8e/bin/gbuild (L237)):
```ruby
distro = build_desc["distro"] || "ubuntu"
```
This PR sets a gitian building distro explicitly in description files.
Tree-SHA512: d2a692047f3466a5c637433610854d1100fe68a78fb03b4a81c70911fa14b0228d9cf25bcf115930aab9cc8c5063dacaf79bcd151f1c3f6fee6419389aefcb8b
ff7f31e07d [doc] productivity: more advanced git range-diff (Sjors Provoost)
3a21905a4e [doc] devtools: mention clang-format dependency (Sjors Provoost)
bf12093191 [doc] productivity: fix broken link (Sjors Provoost)
Pull request description:
Fixes a broken link to `devtools/README.md`, points out the `clang-format` dependency and adds a `git range-diff` incantation that works even with rebases and squashes.
Tree-SHA512: 36e46282f1e28d1bf3f48ada995fbac548f61b7747091eb032b60919cf76c7bdad0fa8aecb0c47adbdaa9ef986d3ec7752b0bb94c63191401856e2ddeec48f3e
32da92bdf6 gitian: Improve error handling (Wladimir J. van der Laan)
Pull request description:
Improve error handling in gitian builds:
- Set fail-on-error and pipefail flag, this causes a command to fail when either of the pipe stages fails, not only when the last of the stages fails, so this improves error detection.
- Also use `xargs` instead of `find -exec`, because `find` will not propagate errors in the executed command, but `xargs` will.
This will avoid some issues like #15541 where non-determinism is silently introduced due to errors caused by environment conditions (such as lack of disk space in that case).
Tree-SHA512: d5d3f22ce2d04a75e5c25e935744327c3adc704c2d303133f2918113573a564dff3d3243d5569a2b93ee7eb0e97f8e1b1ba81767e966af9015ea711a14091035
fa2797808e test: Remove python3.4 workaround in feature_dbcrash (MarcoFalke)
dddd1d05d3 .python-version: Specify full version 3.5.6 (MarcoFalke)
faa7cdf764 scripted-diff: Update copyright in ./test (MarcoFalke)
fa0e65b772 scripted-diff: test: Remove brackets after assert (MarcoFalke)
fab5a1e0f4 build: Require python 3.5 (MarcoFalke)
fa6bf21f5e scripted-diff: test: Use py3.5 bytes::hex() method (MarcoFalke)
Pull request description:
Python 3.4 is EOL after March 2019, so switch to 3.5. See https://devguide.python.org/#status-of-python-branches
This pull does the following in a bunch of commits:
* scripted diff to use the `bytes::hex()` method in place of previous wrappers (`b2x`, `bytes_to_hex_str`, `hexlify`, ...)
* Update the build system (gitian and travis) to remove python2.7 and replace it with python3.5
* Another scripted-diff to remove brackets after `assert`. This is unrelated to the python3.5 switch, but a stylistic commit, so probably not worth to split up. The motivation behind it is to avoid asserting on data structures (such as tuples of length one), which never fails:
```py
>>> assert(False,) # with brackets
>>> assert False, # without brackets
SyntaxError: invalid syntax
>>> assert False # proper assertion
AssertionError
```
* And then a final scripted diff to update the copyright headers in the `test` subfolder, since I touched most of the files anyway and it wouldn't make sense to split this commit out into a separate pull.
For reference (contributed by luke-jr):
Ubuntu LTS (bionic): 3.6.5
Debian stable (stretch): 3.5.3
RHEL 8 (expected before v0.19): 3.6.x
Gentoo stable: 3.6.5
Arch: 3.7.1
Tree-SHA512: 643c28cd2d5b9543ce4bf8ad2a8b282bc79b37dc5b25c9c8358e6ce201e2a67a546463e5f3430b16652eb2489d7c3ed4b0772cd2e2bf790fe68a5e3cc8a25029
f7efd87c8f Change brace formatting (Russell Yanofsky)
a1df1b48a8 Remove use of IsInitialBlockDownload in wallet code (Russell Yanofsky)
1106a6fde4 Remove use of uiInterface.LoadWallet in wallet code (Russell Yanofsky)
318f41fb2c circular-dependencies: Avoid treating some .h/.cpp files as a unit (Russell Yanofsky)
d02b34c8a8 Remove use of AcceptToMemoryPool in wallet code (Russell Yanofsky)
e2c8ba9f6e Remove uses of InitMessage/Warning/Error in wallet code (Russell Yanofsky)
c5e59a96a8 Remove uses of GetAdjustedTime in wallet code (Russell Yanofsky)
6d6bcc77c0 Remove use of g_connman / PushInventory in wallet code (Russell Yanofsky)
00dfb2a440 Remove uses of g_connman in wallet code (Russell Yanofsky)
cc3836e8f9 Remove uses of fPruneMode in wallet code (Russell Yanofsky)
cc02c796d3 Remove uses of fee globals in wallet code (Russell Yanofsky)
1fb0a4a04e Remove use of CalculateMemPoolAncestors in wallet code (Russell Yanofsky)
cd32160af0 Remove use of GetTransactionAncestry in wallet code (Russell Yanofsky)
291276f7f4 Remove use of GetCountWithDescendants in wallet code (Russell Yanofsky)
bdc6628683 Remove use of IsRBFOptIn in wallet code (Russell Yanofsky)
80f52a2267 Remove uses of CheckFinalTx in wallet code (Russell Yanofsky)
Pull request description:
This change removes wallet calls to node functions that access global chain and mempool state.
This is the next step in the larger #10973 refactoring change, which removes all other accesses to node global variables from wallet code. Doing this is useful to provide a better defined interface between the wallet and node, and necessary to allow wallet and node code to run in separate processes in #10102.
Tree-SHA512: 40dbaf1f59fb22b32e70b054b30ba5638d638aa3240fa30e0f721d53c721cd6138a7ab4d423a24d7d2fda0b956e68d44c733abc2c9259c3d6c9fd6d4be89aa23
43206239a8 tests: Add script checking for deterministic line coverage (practicalswift)
Pull request description:
Add script checking for deterministic line coverage in unit tests.
Context: #14343 ("coverage reports non-deterministic")
When the coverage is deterministic this script can be invoked from Travis to guard against regressions, but left inactive for now.
Output in case of determinism:
```
$ contrib/test_deterministic_coverage.sh 2
[2019-01-30 20:08:46] Measuring coverage, run #1 of 2
[2019-01-30 20:10:45] Measuring coverage, run #2 of 2
Coverage test passed: Deterministic coverage across 2 runs.
```
Output in case of non-determinism:
```
$ contrib/test_deterministic_coverage.sh 2
[2019-01-30 20:08:46] Measuring coverage, run #1 of 2
[2019-01-30 20:10:45] Measuring coverage, run #2 of 2
The line coverage is non-deterministic between runs.
The test suite must be deterministic in the sense that the set of lines executed at least
once must be identical between runs. This is a neccessary condition for meaningful coverage
measuring.
--- gcovr.run-1.txt 2019-01-30 23:14:07.419418694 +0100
+++ gcovr.run-2.txt 2019-01-30 23:15:57.998811282 +0100
@@ -471,7 +471,7 @@
test/crypto_tests.cpp 270 270 100%
test/cuckoocache_tests.cpp 142 142 100%
test/dbwrapper_tests.cpp 148 148 100%
-test/denialofservice_tests.cpp 225 225 100%
+test/denialofservice_tests.cpp 225 224 99% 363
test/descriptor_tests.cpp 116 116 100%
test/fs_tests.cpp 24 3 12% 14,16-17,19-20,23,25-26,29,31-32,35-36,39,41-42,45-46,49,51-52
test/getarg_tests.cpp 111 111 100%
@@ -585,5 +585,5 @@
zmq/zmqpublishnotifier.h 5 0 0% 12,31,37,43,49
zmq/zmqrpc.cpp 21 0 0% 16,18,20,22,33-35,38-45,49,52,56,60,62-63
------------------------------------------------------------------------------
-TOTAL 61561 27606 44%
+TOTAL 61561 27605 44%
------------------------------------------------------------------------------
```
In this case line 363 of `test/denialofservice_tests.cpp` was executed only in the second run. Non-determinism detected!
Tree-SHA512: 03f45590e70a87146f89aa7838beeff0925d7fd303697ff03e0e69f8a5861694be5f0dd10cb0020e3e3d40c9cf662f71dfcd838f6affb31bd5212314e0a4e3a9
This avoids a bogus circular dependency error in the next commit:
interfaces/chain -> interfaces/wallet -> wallet/wallet -> interfaces/chain
Which is incorrect, because interfaces/chain.cpp depends only on the
interfaces/wallet.h file, not the interfaces/wallet.cpp file, and it is
wrong to treat these as a unit. Inside the interfaces directory, .h files
contain abstract class definitions and .cpp files contain implementations of
those classes, so you don't need to link against .cpp files if you're only
using the abstract class definition in the .h file.
An alternative fix might be to rename all the cpp files in the interfaces
directory like: chain.cpp->chain_impl.cpp, node.cpp->node_impl.cpp. But just
getting the linter to treat these files as independent dependencies seemed
like it would allow keeping code organization straightforward and avoiding
the need to rename things.
a786c3b306 Slight tweak to the verify-commits script directions (Douglas Roark)
Pull request description:
Clarify that GnuPG may be used on both Linux and macOS to obtain the keys required to verify the commits.
Tree-SHA512: cec556370f03e00bbd6f585d26b360ca236cf55cb5c0996f6d950d8a98f77c92cc02f1719c8f9b9dc9eac6900eb341a13b50a012752832f39095b7e84046f2cd
851380ce17 remove deprecated mentions of signrawtransaction from fundraw help (Gregory Sanders)
Pull request description:
RPC call has been removed as of 0.17.99.
Tree-SHA512: a6a12a0e4572acd9b532c1719be85ed6f29d1c1a28f9ce691398528b8dde4fb4a3222b8f68632fcb1a8eddfe2d31e96d5efd5bc51c041af8e7cb99b61ca3a167
bad1716c6d init: Modify docs and add release note for 12255 (Carl Dong)
b0c7b54d0c init: Use systemd automatic directory creation (Carl Dong)
Pull request description:
- `-datadir` option specified.
- Ask systemd to create and set the right mode for PID directory, configuration directory, and data directory.
- Tell systemd our group so it will set the right owner for aforementioned directories.
More information: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
Tree-SHA512: a6fad1efa2be433c1fdd863df3ff232736ed709a9e281f51a003b40987d8c213dc64a52bc13a19c85bf85680e78f0be112ecaf32ac274b1ff93bac84a1208845
This script compared paths relative to the report directory to test for exclusion,
meaning the directory exclusions did not work properly, as they were relative to
the project root.
Fix this by creating absolute paths through the combination of:
'git ls-files --full-name' and 'git rev-parse --show-toplevel'
fdf82ba18 Update all subprocess.check_output functions in CI scripts to be Python 3.4 compatible (Graham Krizek)
Pull request description:
CI is failing the `lint` stage on every Cron run (regular PR/Push runs still pass). The failure was introduced in 74ce326 and has been broken since. The Python version running in CI was downgraded to 3.4 from 3.6. There were a couple files that were using the `encoding` argument in the `subprocess.check_output` function. This was introduced in Python 3.6 and therefore broke the scripts that were using it. The `universal_newlines` argument was used as well, but in order to use it we must be able to set encoding because of issues on some BSD systems.
To get CI to pass, I removed all `universal_newline` and `encoding` args to the `check_ouput` function. Then I decoded all `check_output` return values. This should keep the same behavior but be Python 3.4 compatible.
Tree-SHA512: f5e5885e98cf4777be9cc254446a873eedb03bdccbd8e06772a964db95e9fcf46736aa9cdcab1d8f123ea9f4947ed6020679898d8b2f47ffb1d94c21a4b08209
Removing the 'universal_newlines' and 'encoding' args from the subprocess.check_outputs fuction. 'universal_newlines' is supported in 3.4, but 'encoding' is not. Without specifying 'encoding' it will make a guess at encoding, which can break things on BSD systems. We must handle encoding/decoding ourselves until we can use Python 3.6
f1bd219a5b contrib: Allow use of github API authentication in github-merge (Wladimir J. van der Laan)
a4c5bbfcd3 contrib: Add support for http[s] URLs in github-merge (Wladimir J. van der Laan)
059a3cffdf contrib: Detailed reporting for http errors in github-merge (Wladimir J. van der Laan)
Pull request description:
Three commits I had locally for `github-merge.py`:
- *Detailed reporting for http errors in github-merge*: Print detailed error, this makes it easier to diagnose github API issues.
- *Add support for http[s] URLs in github-merge*: Sometimes it can be useful to use github-merge with read-only access (say, for reviewing and testing from untrusted VMs).
- *Allow use of github API authentication in github-merge*: The API request limit for unauthenticated requests is quite low. I started running into rate limiting errors. The limit for authenticated requests is much higher. This patch adds an optional configuration setting `user.ghtoken` that, when set, is used to authenticate requests to the API.
Tree-SHA512: ca8ae1874a787263e49d915d7cf31c0c0f50aba229c9440265bf1fda69f7e00641d1492512b93d76c17ff1766859283d640d37770acb120898736ad97efbd5c2
The API request limit for unauthenticated requests is quite low.
I started running into rate limiting errors. The limit
for authenticated requests is much higher.
This patch adds an optional configuration setting `user.ghtoken`
that, when set, is used to authenticate requests to the API.
This adds a checksum in the gitian build script to make sure that
ossl tool and theuni's patch matches what is expected. Also changes
the url to use https and adds the same instructions to the release docs.
Tell systemd to create, set, and ensure the right mode for the PID,
configuration, and data directories.
Only the exec bit is set for groups for the aforementioned directories.
This is the least privilege perm that allows for the
reading/writing/execing of files under the directory _if_ the files
themselves give permission to its group to do so (e.g. when -sysperms is
specified). Note that this does not allow for the listing of files under
the directory.
522b80b33f add `--retry 5` to curl opts in install_db4.sh (qubenix)
Pull request description:
I ran into some network issue that caused the clang patch to not download and the script exited. A retry would have solved it. The fallback choice, `wget`, has a default 20 retries.
I chose 5 retries because `curl` backs of after each try, starting at one second and doubling each time. 5 retries means that worst case scenario would be a total of 31 seconds waiting between attempts. IMO that should be enough tries if internet is working, but not too much if internet is not working.
Tree-SHA512: ff48e1f8a9d39b9c6fabc616223974f46fadc5a1b30793cdaaf5ea4ce79f7c76efd259bfa9e81a5a7a7cff60b7133d89448faa3f86cc2f9b0c5ac9adf8dd2a9b
