935acdcc79 refactor: modernize the implementation of uint256.* (pasta)
Pull request description:
- Constructors of uint256 to utilize Span instead of requiring a std::vector
- converts m_data into a std::array
- Prefers using `WIDTH` instead of `sizeof(m_data)`
- make all the things constexpr
- replace C style functions with c++ equivalents
- memset -> std::fill
This may also be replaced by std::memset, but I think that std::fill is more idiomatic of modern c++ and readable.
- memcpy -> std::copy
Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
This could also likely be replaced by std::memcpy, but as said above, I believe the using std::copy is the more c++ way to do anything and is almost guaranteed to compile to the same asm
- memcmp -> std::memcmp
ACKs for top commit:
achow101:
ACK 935acdcc79
hebasto:
Approach ACK 935acdcc79.
aureleoules:
reACK 935acdcc79
john-moffett:
ACK 935acdcc79
stickies-v:
Approach ACK 935acdcc7
Tree-SHA512: 4f1ba54ff2198eea0e505d41e73d552c84c60f6878d5c85a94a8ab57f39afc94ef8d79258e7afd01fa84ec2a99f4404bb877eecd671f65e1ee9273f3129fc650
fa6986a66b ci: Print iwyu patch in git diff format (MarcoFalke)
Pull request description:
Seems more dev friendly to also have a patch to copy-paste
ACKs for top commit:
hebasto:
ACK fa6986a66b, tested on Ubuntu 22.04 locally.
fanquake:
ACK fa6986a66b - did not test but example CI output looks ok.
stickies-v:
utACK fa6986a66b
Tree-SHA512: 7cfd8584bf12e03c28af23f4712c6bcafd648d87ddb92788b9cd35455b2db49f4bd4aef8ad4711f75c7f11ad2bb2492c2eb6044007086c20e36016575c060603
fa486de212 ci: Cache package manager install step (MarcoFalke)
Pull request description:
Use the local podman or docker image cache to skip the slow `apt` step
ACKs for top commit:
jamesob:
ACK fa486de212 ([`jamesob/ackr/26976.1.MarcoFalke.ci_cache_package_manager`](https://github.com/jamesob/bitcoin/tree/ackr/26976.1.MarcoFalke.ci_cache_package_manager))
Tree-SHA512: 3495346c6c862b63296d2691cc492bf52a0a99ee7fae798887c792609904546013eba788045cd508a5f669f2c52e3479c122c18a5275c87af38237a1b5c9da17
Don't enable `-Werror` (in the CI) for compilers at least older than
our current release compiler (GCC 10). It provides little-to-no value,
other than turning compiler bugs & false positives into build failures,
and we aren't going to mutate perfectly fine/working code, for the sake
of avoid a warning that shouldn't even exist.
I also do not see the point of playing whack-a-mole and turning off various
warnings/trying to further work around the broken compiler, just to
acheive warningless builds for the sake of warningless builds.
One anecdote from "How SQLite Is Tested":
> Static analysis has found a few bugs in SQLite, but those are the
> exceptions. More bugs have been introduced into SQLite while trying
> to get it to compile without warnings than have been found by static
> analysis.
https://www.sqlite.org/testing.html.
6d58117a31 build: Build minisketch test in `make check`, not in `make` (Hennadii Stepanov)
Pull request description:
On master (d1e42659bb):
```
$ ./autogen.sh && ./configure --without-gui --disable-wallet && make clean
$ make 2>&1 | grep LD | grep -v .la
CXXLD bitcoind
CXXLD bitcoin-cli
CXXLD bitcoin-tx
CXXLD bitcoin-util
CXXLD test/test_bitcoin
CXXLD bench/bench_bitcoin
CXXLD minisketch/test
CXXLD test/fuzz/fuzz
CXXLD univalue/test/object
CXXLD univalue/test/unitester
$ make check 2>&1 | grep LD
CCLD exhaustive_tests
CCLD tests
```
With this PR:
```
$ ./autogen.sh && ./configure --without-gui --disable-wallet && make clean
$ make 2>&1 | grep LD | grep -v .la
CXXLD bitcoind
CXXLD bitcoin-cli
CXXLD bitcoin-tx
CXXLD bitcoin-util
CXXLD test/test_bitcoin
CXXLD bench/bench_bitcoin
CXXLD test/fuzz/fuzz
CXXLD univalue/test/object
CXXLD univalue/test/unitester
$ make check 2>&1 | grep LD
CXXLD minisketch/test
CCLD exhaustive_tests
CCLD tests
```
In fact, this PR restores behavior that was before bitcoin/bitcoin#22646, and that behavior looks more optimal.
As an outcome, the `contrib/guix/libexec/build.sh` does not spend resources to build binaries which are not a part of the release package.
ACKs for top commit:
TheCharlatan:
ACK 6d58117a31
Tree-SHA512: 4957c8f88a01aca005813bf4c1c26f433756bf68ea0c958481c638ead229fa8e23ecae3a8ac31ea555876ba6f2cc10ecd91caf2e2f664de5cb529ec05fb38fa7
faba08b5b4 refactor: Remove stray cs_main redundant declaration (MarcoFalke)
fa02591edf doc: Export threadsafety.h from sync.h (MarcoFalke)
Pull request description:
Looks like this was forgotten when introducing kernel/cs_main ?
Also, there is a commit to export threadsafety.h from sync.h.
ACKs for top commit:
hebasto:
ACK faba08b5b4
Tree-SHA512: 0aa58e7693b6fcd504f9da7339f8baa463a6407f67b27f68002db705f4642321ac3765f16c3d906c925ee24085591b79160a62fa5f4aaf6f2e5dcc788411800d
a3a2bd9e8a ci: Drop no longer needed package-specific flags (Hennadii Stepanov)
071eef1e97 build: Propagate user-defined flags to host packages (Hennadii Stepanov)
Pull request description:
On master (4f8b1f8759) `{CPP,C,CXX,LD}FLAGS` that are specified in the command line are not propagated to packages:
```
$ make --no-print-directory -C depends print-libevent_cxxflags CXXFLAGS=-some-fancy-flag
libevent_cxxflags=-pipe -O2
```
This PR:
- propagates `{CPP,C,CXX,LD}FLAGS` to host packages:
```
$ make --no-print-directory -C depends print-libevent_cxxflags CXXFLAGS=-some-fancy-flag
libevent_cxxflags= -some-fancy-flag
```
- does not propagate `{CPP,C,CXX,LD}FLAGS` to native packages:
```
$ make --no-print-directory -C depends print-native_b2_cxxflags CXXFLAGS=-some-fancy-flag
native_b2_cxxflags=
```
- actually addresses the https://github.com/bitcoin/bitcoin/pull/23551#issuecomment-973896518
ACKs for top commit:
TheCharlatan:
Code review ACK a3a2bd9e8a
Tree-SHA512: 243d6b1b0e9c5de46debc36de62a77b6b4d6f638940fd530040c219956ec624e321b0c25290fed164e3a8c88befa7b97b20f765d7b9a428c269b3720f21da099
b68e5a7fef lint: specify the right commit range when running locally (James O'Beirne)
dff7ed5732 test: add an easy way to run linters locally (James O'Beirne)
Pull request description:
Adds a Dockerfile configuration ~~(originally written mostly by fanquake)~~ that allows straightforward running of linters with compatible versions locally. This removes a ton of annoyance when trying to appease CI, because many of the linter versions are quite old and difficult to maintain locally.
I realize that people may not be thrilled to add more ancillary tooling to the repo, but I think this makes a lot of sense given the linter versions listed in this container configuration are dictated by this repo (within the CI configuration), so having these things live in two separate places is a recipe for version mismatches.
Eventually we can likely just use this container on CI directly to avoid any chance of inconsistencies between local dev experience and CI.
ACKs for top commit:
aureleoules:
ACK b68e5a7fef
stickies-v:
ACK b68e5a7fe
john-moffett:
ACK b68e5a7fef
Tree-SHA512: 7ef7a5dae023d81fdb6296d5d92dfa074ee321c7993e607c9f014d0f21c91558611aa00fc3ce1edc7b5e68371aea0d27fa1931291a79bb867a6c783bb536775f
Adds a Dockerfile configuration
that allows straightforward running of linters with compatible versions
locally. This removes a ton of annoyance when trying to appease CI,
because many of the linter versions are quite old and difficult to
maintain locally.
I realize that people may not be thrilled to more ancillary tooling to
the repo, but I think this makes a lot of sense given the linter
versions listed in this container configuration are dictated by this
repo (within the CI configuration), so having these things live in
two separate places is a recipe for version mismatches.
Eventually we can likely just use this container on CI directly to avoid
any chance of inconsistencies between local dev experience and CI.
fa87d71872 ci: Add missing lint dependency (MarcoFalke)
Pull request description:
Also, document each dependency.
Adding `gpg` avoids errors when running a release or dev branch in the CI:
01ec5308bf/ci/lint/06_script.sh (L30-L42)
```
bash: line 1: gpg: command not found
```
https://cirrus-ci.com/task/4582854860996608?logs=lint#L185
ACKs for top commit:
hebasto:
ACK fa87d71872
Tree-SHA512: 869a3d2feab764b2c8d47d481359680a1d2c54a33b13ca26c5f8ce56cf2f368d4c74637dcbc53fdbf323f10940965c1c0e592e2fb4ce725d5cd467e77e62b6e5
123043e99c ci: Bump lint task image to Ubuntu Jammy (Hennadii Stepanov)
9b86114058 ci: Use pyenv's `python-build` to install Python in lint task (Hennadii Stepanov)
Pull request description:
This PR:
- is an alternative to bitcoin/bitcoin#26581 and bitcoin/bitcoin#26637
- closesbitcoin/bitcoin#26548
Key advantages of this PR over others:
- it uses pyenv's `python-build` [standalone](https://github.com/pyenv/pyenv/tree/master/plugins/python-build#using-python-build-standalone)
- requires no additional computational resources
Note for testing. The lint task must success regardless of whether the `python_cache` is populated or invalidated.
ACKs for top commit:
MarcoFalke:
ACK 123043e99c
fanquake:
ACK 123043e99c
Tree-SHA512: ba0fcdd4f2939a59692b173dcd1f5704444cfcfbb8111538c6f8160056d0536bba250e4f9b0f8c66f8b454e52034bf36ffe6afae76cdc0f7cc5b58b576d790ba
DOCKER in names is confusingly used as synonym for "image", "container",
and "ci". Fix the confusion by picking the term that fits the context.
-BEGIN VERIFY SCRIPT-
ren() { sed -i "s:$1:$2:g" $( git grep -l "$1" ) ; }
ren DOCKER_PACKAGES CI_BASE_PACKAGES
# This better reflects that they are the common base for all CI
# containers.
ren DOCKER_ID CI_CONTAINER_ID
# This is according to the documentation of "--detach , -d: Run
# container in background and print container ID".
ren DOCKER_NAME_TAG CI_IMAGE_NAME_TAG
# This avoids confusing with CONTAINER_NAME and clarifies that it is an
# image.
ren DOCKER_ADMIN CI_CONTAINER_CAP
# This clarifies that it is a capability added to the container.
ren DOCKER_CI_CMD_PREFIX CI_EXEC_CMD_PREFIX
# This brings it in line with the CI_EXEC naming.
-END VERIFY SCRIPT-
fa5cbf2290 ci: Properly set COMMIT_RANGE in lint task (MarcoFalke)
Pull request description:
Currently the variable holds (apart from the commits in the pull request) all commits to master since the pull was opened.
This is problematic, because already merged commits are linted in unrelated pulls, leading to:
* Wasted resources. For example, currently the lint task may take 9 minutes, when it should take 1. See https://cirrus-ci.com/task/6032782770569216?logs=lint#L1449
* False failures. For example, when a "wrong" commit is in master it can lead to some pulls failing unrelatedly, and others not.
Now that the CI has the `/merge` commit (since commit fad7281d78), `COMMIT_RANGE` can simply be set to `HEAD~..HEAD` to only hold the changes in the pull.
ACKs for top commit:
fanquake:
ACK fa5cbf2290
Tree-SHA512: e85fca4ca9d2615ddd2544403485e06885769a3f70bca297e23eefda2a1d28f47c5271f6adfa6ce0e5e972335c78098b76e0db4b109f59d0986bf508cef7528f
fabb6af850 ci: Remove duplicate CC and CXX from tsan task (MarcoFalke)
fa5d9a0e24 Revert "ci: Use clang-15 in tsan task" (MarcoFalke)
faa835e7e5 Revert "test: Drop no longer needed `race:epoll_ctl` TSan suppression" (MarcoFalke)
Pull request description:
Looks like there are still bugs in clang-15, so we need to roll back all the way to the previously used version (clang-13).
ACKs for top commit:
hebasto:
ACK fabb6af850, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: d62203049847ab9095ee3fc89e18bdd721d1d9d5a7ef7a9f524c80e6be58d1d9f6aa2f14533df1ea77eb59597fba6fa9b987b17eb03b2c3f7cb577ab59cd59c0
- Constructors of uint256 to utilize Span instead of requiring a std::vector
- converts m_data into a std::array
- Prefers using `WIDTH` instead of `sizeof(m_data)`
- make all the things constexpr
- replace C style functions with c++ equivalents
- memset -> std::fill
- memcpy -> std::copy
Note: In practice, implementations of std::copy avoid multiple assignments and use bulk copy functions such as std::memmove if the value type is TriviallyCopyable and the iterator types satisfy LegacyContiguousIterator. (https://en.cppreference.com/w/cpp/algorithm/copy)
- memcmp -> std::memcmp
2811f40f30 ci: only run USDT interface tests on CirrusCI (0xb10c)
Pull request description:
As mentioned in #26571, the task running the USDT interface tests fail when run in docker. cc7335edc8 in #25528 added that the tests are run in a **VM** in Cirrus CI. Running them locally in docker containers might not work:
- We use [bcc] as tracing toolkit which requires the kernel headers to compile the BPF bytecode. As docker containers use the hosts kernel and don't run their own, there is a potential for mismatches between kernel headers available in the container and the host kernel. This results in a failure loading the BPF byte code.
- Privilges are required to load the BPF byte code into the kernel. Normally, the docker containers aren't run with these.
- We currently use an untrusted third-party PPA to install the bpfcc-tools package on Ubuntu 22.04. Using this on a local dev system could be a security risk.
To not hinder the ASan + LSan + UBSan part of the CI task, the USDT tests are disabled on non-CirrusCI runs.
[bcc]: https://github.com/iovisor/bcc
Top commit has no ACKs.
Tree-SHA512: 7c159b59630b36d5ed927b501fa0ad6f54ff3d98d0902f975cc8122b4147a7f828175807d40a470a85f0f3b6eeb79307d3465a287dbd2f3d75b803769ad0b6e7
As mentioned in #26571, the task running the USDT interface tests
fail when run in docker. cc7335edc8
in #25528 added that the tests are run in a **VM** in Cirrus CI.
Running them locally in docker containers might not work:
- We use [bcc] as tracing toolkit which requires the kernel headers
to compile the BPF bytecode. As docker containers use the hosts
kernel and don't run their own, there is a potential for mismatches
between kernel headers available in the container and the host
kernel. This results in a failure loading the BPF byte code.
- Privilges are required to load the BPF byte code into the kernel.
Normally, the docker containers aren't run with these.
- We currently use an untrusted third-party PPA to install the
bpfcc-tools package on Ubuntu 22.04. Using this on a local dev
system could be a security risk.
To not hinder the ASan + LSan + UBSan part of the CI task, the USDT
tests are disabled on non-CirrusCI runs.
[bcc]: https://github.com/iovisor/bcc
fad1c55301 lint: Skip COMMIT_RANGE if no CIRRUS_PR (MarcoFalke)
Pull request description:
It doesn't make sense to run this for non-PRs, because:
* There are known whitespace "violations" in previous commits, so the lint may fail
* Once the changes are merged, it is too late to fix them up (force pushes are illegal)
* It isn't possible to determine which commits to run on if there is no reference branch (target branch of the pull request)
Moreover, the test fails on non-master:
* https://github.com/bitcoin/bitcoin/runs/8664441400
Fix all issues by skipping it.
ACKs for top commit:
hebasto:
ACK fad1c55301, also tested in my personal Cirrus account.
Tree-SHA512: be15f00e2b2a9069583833545883e0e5968a33d2455dad59e6fb47c1102b4dd16ef932e9ba945e29e9d941e6c17bd531a02c66b0491097801be6bda476875537
54dd8f51ce ci: use ci_exec_root for clang install (josibake)
Pull request description:
fixes a bug introduced in #25900 ; see https://github.com/bitcoin/bitcoin/pull/25900#issuecomment-1327311069
the general idea of #25900 was to use a non-root user as much as possible to avoid modifying the user's local filesystem. however, it appears the root user is needed to correctly install clang.
ACKs for top commit:
hebasto:
ACK 54dd8f51ce, tested on Ubuntu 22.04.
Tree-SHA512: beb01d4b6127fbba3c8d18e85cf7ec7d1b2ec93ea05c475ab51bcaa04ef1b0591d886f1a7e0732c5ae86806013f022c0b44027380d2b0cfb1bfdc843e40f99b4
849f20a6d3 ci: create and use non-root user for docker image (josibake)
Pull request description:
Previously, everything in the ci docker image ran as the root user. This would lead to certain directories (`ci/scratch`, `depends`) being owned by `root` after running the ci locally which would lead to annoying behavior such as subsequent guix builds failing due to `depends/` being owned by root.
This PR adds a non-root user in the container and chowns the mounted working directory. All the `docker exec` commands now run as the non-root user, except for the few that still need to run as root (mainly, installing packages).
To test this I checked out a fresh copy of the repo, applied my changes, ran the CI, and verified all the local file permissions were unchanged after the CI was finished running.
ACKs for top commit:
hebasto:
ACK 849f20a6d3, tested on Ubuntu 22.04 by running commands as follows:
Tree-SHA512: 734dca0f36157fce5fab243b4ff657fc17ba980e8e4e4644305f41002ff21bd5cef02c306ea1e0b5c841d4c07c095e8e4be16722e6a38c890717c60a3f5ec62a
b89530483d util: move threadinterrupt into util (fanquake)
Pull request description:
Alongside thread and threadnames. It's part of libbitcoin_util.
ACKs for top commit:
ryanofsky:
Code review ACK b89530483d. No changes since last review other than rebase
theuni:
ACK b89530483d.
Tree-SHA512: 0421f4d1881ec295272446804b27d16bf63e6b62b272f8bb52bfecde9ae6605e8109ed16294690d3e3ce4b15cc5e7c4046f99442df73adb10bdf069d3fb165aa
Running all commands as the root user in the docker image
will change local file permissions in the ci and depends directory.
Add a non-root user to the container and use this user whenever
possible when running docker exec commands.
