228d6a2969 build: Fix regression in "ARMv8 CRC32 intrinsics" test (Hennadii Stepanov)
Pull request description:
In the master branch, the `aarch64` binaries lack support for CRC32 intrinsics.
The `vmull_p64` is a part of the Crypto extensions from the ACLE. They are optional extensions, so they get enabled with a `+crypto` for architecture flags.
The regression was introduced in https://github.com/bitcoin/bitcoin/pull/26183 (v25.0).
The `./configure` script log excerpts:
- the master branch @ d752349029:
```
checking whether C++ compiler accepts -march=armv8-a+crc... yes
checking whether C++ compiler accepts -march=armv8-a+crypto... yes
checking for ARMv8 CRC32 intrinsics... no
checking for ARMv8 SHA-NI intrinsics... yes
```
- this PR:
```
checking whether C++ compiler accepts -march=armv8-a+crc+crypto... yes
checking whether C++ compiler accepts -march=armv8-a+crypto... yes
checking for ARMv8 CRC32 intrinsics... yes
checking for ARMv8 SHA-NI intrinsics... yes
```
Guix build:
```
x86_64
2afd81f540c6d3b36ff305e88bafe935e4272cd3efef3130aa69d49a0522541b guix-build-228d6a2969e4/output/aarch64-linux-gnu/SHA256SUMS.part
6c704d6d30d495adb3fb86befdb500eb389a02c1167163f14ab5c3c3e630e6b3 guix-build-228d6a2969e4/output/aarch64-linux-gnu/bitcoin-228d6a2969e4-aarch64-linux-gnu-debug.tar.gz
e4419963c9c0d99adc4e38538900b648f2c14f793b60c8ee2e6f5acc9d3fadd3 guix-build-228d6a2969e4/output/aarch64-linux-gnu/bitcoin-228d6a2969e4-aarch64-linux-gnu.tar.gz
7d11052b6bd28cdf26d5f2a4987f02d32c93a061907bcd048fb6d161a0466ca9 guix-build-228d6a2969e4/output/dist-archive/bitcoin-228d6a2969e4.tar.gz
```
ACKs for top commit:
TheCharlatan:
ACK 228d6a2969
Tree-SHA512: 4c27ca8acb953bf56e972d907a282ee19e3f30f7a4bf8a9822395fe0e28977cd6233e8b65b4a25cc1d3d5ff6a796d7af07653e18531c44ee3efaff1563d96d32
f95af98128 guix: default ssp for Windows GCC (fanquake)
95d55b96c2 guix: remove ssp workaround from Windows GCC (fanquake)
8f43302a0a build: remove explicit libssp linking from Windows build (fanquake)
Pull request description:
I was expecting this to fail to compile somewhere, maybe in the CI, but that doesn't seem to be the case?
Seems workable given the SSP related changes in the newer mingw-w64 headers (which are in Guix):
> Implement some of the stack protector functions/variables so -lssp is now optional when _FORTIFY_SOURCE or -fstack-protector-strong is used.
However I think this would still be broken in some older environments, so we might have to wait for a compiler bump, or similar. The optional -lssp also seems to work when using older headers, which doesn't make sense.
Would fix#28104.
ACKs for top commit:
hebasto:
ACK f95af98128, I've verified binaries from `bitcoin-f95af98128f1-win64.zip` on Windows 11 Pro 23H2.
TheCharlatan:
ACK f95af98128
Tree-SHA512: 71169ec513cfe692dfa7741d2bf37b45da05627c0af1cbd50cf8c3c04cc21c4bf88f3284532bddc1e3e648391ec78dbaca5170987a13c21ac204a7bcaf27f349
fa01f884d3 ci: Add missing COPY for ./test/lint/test_runner (MarcoFalke)
faff3e3b46 lint: Report all lint errors instead of early exit (MarcoFalke)
Pull request description:
`all-lint.py` currently collects all failures. However, the `06_script.sh` does not, since July this year (https://github.com/bitcoin/bitcoin/pull/28103#discussion_r1268115806).
Fix this by printing all failures before exiting.
Can be tested by modifying (for example) two subtrees in the same commit and then running the linters.
ACKs for top commit:
kevkevinpal:
ACK [fa01f88](fa01f884d3)
TheCharlatan:
lgtm ACK fa01f884d3
Tree-SHA512: c0f3110f2907d87e29c755e3b77a67dfae1f8a25833fe6ef8f2f2c58cfecf1aa46f1a20881576b62252b04930140a9e416c78b4edba0780d3c4fa7aaebabba81
21bfee0720 depends: bump libmultiprocess to fix capnproto deprecation warnings (Ryan Ofsky)
Pull request description:
This incorporates PR chaincodelabs/libmultiprocess#88 and reverts the NO_WERROR CI workaround added in #28735
Upstream diff: 61d5a0e661...414542f81e
---
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/issues/28722).
ACKs for top commit:
maflcko:
lgtm ACK 21bfee0720
hebasto:
ACK 21bfee0720, I have reviewed the code and it looks OK. I've also skimmed through the related changes in the https://github.com/chaincodelabs/libmultiprocess repository.
Tree-SHA512: b5addb0deed694eeec62a0ae08b4715a811110201f39f3e6cadee8fc4e6231b0e66c844a98512072a1445bac122ab561dc1711e27fb4d7ac5c08ac46780a4acf
5e7cc4144b test: add unit test for CConnman::AddedNodesContain() (Jon Atack)
cc62716920 p2p: do not make automatic outbound connections to addnode peers (Jon Atack)
Pull request description:
to allocate our limited outbound slots correctly, and to ensure addnode
connections benefit from their intended protections.
Our addnode logic usually connects the addnode peers before the automatic
outbound logic does, but not always, as a connection race can occur. If an
addnode peer disconnects us and if it was the only one from its network, there
can be a race between reconnecting to it with the addnode thread, and it being
picked as automatic network-specific outbound peer. Or our internet connection
or router or the addnode peer could be temporarily offline, and then return
online during the automatic outbound thread. Or we could add a new manual peer
using the addnode RPC at that time.
The race can be more apparent when our node doesn't know many peers, or with
networks like cjdns that currently have few bitcoin peers.
When an addnode peer is connected as an automatic outbound peer and is the only
connection we have to a network, it can be protected by our new outbound
eviction logic and persist in the "wrong role".
Finally, there does not seem to be a reason to make block-relay or short-lived
feeler connections to addnode peers, as the addnode logic will ensure we connect
to them if they are up, within the addnode connection limit.
Fix these issues by checking if the address is an addnode peer in our automatic
outbound connection logic.
ACKs for top commit:
mzumsande:
Tested ACK 5e7cc4144b
brunoerg:
utACK 5e7cc4144b
vasild:
ACK 5e7cc4144b
guggero:
utACK 5e7cc4144b
Tree-SHA512: 2438c3ec92e98aebca2a0da960534e4655a9c6e1192a24a085fc01326d95cdb1b67d8c44e4ee706bc1d8af8564126d446a21b5579dcbec61bdea5fce2f0115ee
007d6f0e85 test: fix `AddNode` unit test failure on OpenBSD (Sebastian Falbesoner)
Pull request description:
On OpenBSD 7.4, the following check of the unit test `test_addnode_getaddednodeinfo_and_connection_detection` currently fails:
```
BOOST_CHECK(!connman->AddNode({/*m_added_node=*/"127.1", /*m_use_v2transport=*/true}));
```
The reason for that is that this OS seemingly doesn't support the IPv4 shorthand notation with omitted zero-bytes:
```
$ ping 127.1
ping: no address associated with name
```
As a simple fix, this PR skips the check for this with a pre-processor #if. On NetBSD and FreeBSD, `127.1` is resolved correctly to localhost and hence the test passes (thanks to vasild for verifying on the latter!).
ACKs for top commit:
vasild:
ACK 007d6f0e85
Tree-SHA512: 8ab8393c490e1ecc140e8ff74f6fa4d26d0dd77e6a77a241cd198314b8c5afee7422f95351ca05f4c1742433dab77016a8ccb8d28062f8edd4b703a918a2bbda
d5b4c0b69e pool: change memusage_test to use int64_t, add allocation check (Martin Leitner-Ankerl)
ce881bf9fc pool: make sure PoolAllocator uses the correct alignment (Martin Leitner-Ankerl)
Pull request description:
The class `CTxOut` has a member `CAmount` which is an int64_t, and on ARM 32bit int64_t are 8 byte aligned, which is larger than the pointer alignment of 4 bytes.
So for `CCoinsMap` to be able to use the pool, we need to use the alignment of the member instead of just `alignof(void*)`.
This fixes#28906 (first noted in https://github.com/bitcoin/bitcoin/issues/28718#issuecomment-1807197107) and #28440.
ACKs for top commit:
pinheadmz:
ACK d5b4c0b69e
hebasto:
re-ACK d5b4c0b69e, the only change since my recent [review](https://github.com/bitcoin/bitcoin/pull/28913#pullrequestreview-1739334189) is an updated test.
theStack:
Tested ACK d5b4c0b69e
Tree-SHA512: 4446793fad6d56f0fe22e09ac9ade051e86de11ac039cd61c0f6b7f79874242878a6a46a2c76ac3b8f1d53464872620d39139f54b1471daccad26d6bb1ae8ca1
710da28c72 ci: Switch from `apt` to `apt-get` (Hennadii Stepanov)
a6cc059ea5 ci: Update apt cache (Hennadii Stepanov)
Pull request description:
This PR aims to fix the recent errors in the "test each commit" CI job.
ACKs for top commit:
maflcko:
lgtm ACK 710da28c72
ismaelsadeeq:
utACK 710da28c72
Tree-SHA512: b42340aea00e80f791000e19791629f27df2da98adefb839cb4389d81b5eee094089ea5092a2d7b56b3990683a72e4d2fa986fc86c823c7245649af37873b790
If alignment of the PoolAllocator would be insufficient, then the test would fail. This also catches the issue with ARM 32bit,
where int64_t is aligned to 8 bytes but void* is aligned to 4 bytes. The test adds a check to ensure the pool has allocated
a minimum number of chunks
The `vmull_p64` is a part of the Crypto extensions from the ACLE. They
are optional extensions, so they get enabled with a `+crypto` for
architecture flags.
This changes the PoolAllocator to default the alignment to the given type. This makes the code simpler, and most importantly
fixes a bug on ARM 32bit that caused OOM: The class CTxOut has a member CAmount which is an int64_t and on ARM 32bit int64_t
are 8 byte aligned which is larger than the pointer alignment of 4 bytes. So for CCoinsMap to be able to use the pool, we
need to use the alignment of the member instead of just alignof(void*).
fa552e8a4e doc: Simplify guix install doc, after 1.4 release (MarcoFalke)
Pull request description:
Now that 1.4 is out (for a while), remove the recommendation to build a random commit.
ACKs for top commit:
fanquake:
ACK fa552e8a4e
hebasto:
ACK fa552e8a4e.
Tree-SHA512: f5642df201ff0e2af8a7ae9660a66920ddbb5f522b3e921f6f4aa7c411ced23afa91bdfe43b943ac012228eebbaad3396df505d00aa8f721a4358f03fda9d8e3
a478c817b2 test: replace `Callable`/`Iterable` with their `collections.abc` alternative (PEP 585) (stickies-v)
4b9afb18e6 scripted-diff: use PEP 585 built-in collection types for verify-binary script (Sebastian Falbesoner)
d516cf83ed test: use built-in collection types for type hints (Python 3.9 / PEP 585) (Sebastian Falbesoner)
Pull request description:
With Python 3.9 / [PEP 585](https://peps.python.org/pep-0585/), [type hinting has become a little less awkward](https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections), as for collection types one doesn't need to import the corresponding capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can use the built-in types directly (see https://peps.python.org/pep-0585/#implementation for the full list).
This PR applies the replacement for all Python scripts (i.e. in the contrib and test folders) for the basic types, i.e.:
- typing.Dict -> dict
- typing.List -> list
- typing.Set -> set
- typing.Tuple -> tuple
For an additional check, I ran mypy 1.6.1 on both master and the PR branch via
```
$ mypy --ignore-missing-imports --explicit-package-bases $(git ls-files "*.py")
```
and verified that the output is identical -- (from the 22 identified problems, most look like false-positives, it's probably worth it to go deeper here and address them in a follow-up though).
ACKs for top commit:
stickies-v:
ACK a478c817b2
fanquake:
ACK a478c817b2
Tree-SHA512: 6948c905f6abd644d84f09fcb3661d7edb2742e8f2b28560008697d251d77a61a1146ab4b070e65b0d27acede7a5256703da7bf6eb1c7c3a897755478c76c6e8
83986f464c Include version.h in fewer places (Anthony Towns)
c7b61fd61b Convert some CDataStream to DataStream (Anthony Towns)
1410d300df serialize: Drop useless version param from GetSerializeSize() (Anthony Towns)
bf574a7501 serialize: drop GetSerializeSizeMany (Anthony Towns)
efa9eb6d7c serialize: Drop nVersion from [C]SizeComputer (Anthony Towns)
Pull request description:
Drops the version field from `GetSerializeSize()`, simplifying the code in various places. Also drop `GetSerializeSizeMany()` (as just removing the version parameter could result in silent bugs) and remove unnecessary instances of `#include <version.h>`.
ACKs for top commit:
maflcko:
ACK 83986f464c📒
theuni:
ACK 83986f464c.
Tree-SHA512: 36617b6dfbb1b4b0afbf673e905525fc6d623d3f568d3f86e3b9d4f69820db97d099e83a88007bfff881f731ddca6755ebf1549e8d8a7762437dfadbf434c62e
faa25718b3 fuzz: AutoFile with XOR (MarcoFalke)
fab5cb9066 fuzz: Reduce LIMITED_WHILE limit for file fuzzing (MarcoFalke)
fa5388fad3 fuzz: Remove FuzzedAutoFileProvider (MarcoFalke)
Pull request description:
This should help to get fuzz coverage for https://maflcko.github.io/b-c-cov/fuzz.coverage/src/streams.cpp.gcov.html
Also, remove unused code and fix a timeout bug.
ACKs for top commit:
dergoegge:
ACK faa25718b3
Tree-SHA512: 56f1e6fd5cb2b66ffd9a7d9c09c9b8e396be3e7485feb03b35b6bd3c48e624fdaed50b472e4ffec21f09efb5e949d7ee32a13851849c9140b6b4cf25917dd7ac
to allocate our limited outbound slots correctly, and to ensure addnode
connections benefit from their intended protections.
Our addnode logic usually connects the addnode peers before the automatic
outbound logic does, but not always, as a connection race can occur. If an
addnode peer disconnects us and if it was the only one from its network, there
can be a race between reconnecting to it with the addnode thread, and it being
picked as automatic network-specific outbound peer. Or our internet connection
or router, or the addnode peer, could be temporarily offline, and then return
online during the automatic outbound thread. Or we could add a new manual peer
using the addnode RPC at that time.
The race can be more apparent when our node doesn't know many peers, or with
networks like cjdns that currently have few bitcoin peers.
When an addnode peer is connected as an automatic outbound peer and is the only
connection we have to a network, it can be protected by our new outbound
eviction logic and persist in the "wrong role".
Examples on mainnet using logging added in the same pull request:
2023-08-12T14:51:05.681743Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic network-specific outbound-full-relay connection
to i2p peer selected for manual (addnode) connection: [geh...odq.b32.i2p]:0
2023-08-13T03:59:28.050853Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic block-relay-only connection to onion peer
selected for manual (addnode) connection: kpg...aid.onion:8333
2023-08-13T16:21:26.979052Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic network-specific outbound-full-relay connection
to cjdns peer selected for manual (addnode) connection: [fcc...8ce]:8333
2023-08-14T20:43:53.401271Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic network-specific outbound-full-relay connection
to cjdns peer selected for manual (addnode) connection: [fc7...59e]:8333
2023-08-15T00:10:01.894147Z [opencon] [net.cpp:1949] [ThreadOpenConnections]
[net:debug] Not making automatic feeler connection to i2p peer selected for
manual (addnode) connection: geh...odq.b32.i2p:8333
Finally, there does not seem to be a reason to make block-relay or short-lived
feeler connections to addnode peers, as the addnode logic will ensure we connect
to them if they are up, within the addnode connection limit.
Fix these issues by checking if the address is an addnode peer in our automatic
outbound connection logic.
43de4d3630 doc: fix typos (Sjors Provoost)
Pull request description:
This PR fixes typos found by lint-spelling.py using codespell 2.2.6.
Our CI linter job uses codespell 2.2.5 and found fewer typos that I did locally. In any case it's happy now.
ACKs for top commit:
pablomartin4btc:
re ACK 43de4d3630
Tree-SHA512: c032fe86cb49c924a468385653b31f309a9db68c478d70335bba3e65a1ff3826abe80284fe00a090ab5a509e1edbf17e476f6922fb15d055e50f1103dad2ccb0
88e09ac2a1 tests: Fix LCOV_OPTS to be in the correct position (Andrew Chow)
Pull request description:
`lcov`'s `-a` option takes an argument. With `LCOV_OPTS` immediately after `-a`, the first additional argument becomes the argument to `-a` which is incorrect.
Also add `LCOV_OPTS` to more `lcov` calls.
ACKs for top commit:
fanquake:
ACK 88e09ac2a1
Tree-SHA512: 1ed657c96395bfe882041ded883cb5fa4d04d6ede91f66c319b5bbdd1f88468f8abb2a741dd7898904a78ed7e6c844316f7958ce9e4ccf2dbe666ebec308b7fb
3b19100303 depends: remove PYTHONPATH from config.site (fanquake)
Pull request description:
We no-longer need this, as we no-longer build python packages.
ACKs for top commit:
hebasto:
ACK 3b19100303, this PR effectively reverts no longer needed de619a37fd.
Tree-SHA512: 775354773f83fc98922f1d4ee84d8f1e866fb6fb2a59a3eaf06a7a5f0d846f7dc1b84862c58195dfb91ddfb02b2dc86bee78b51459f91c65a5b1464df9f3c53c
6a917918b7 fuzz: allow fake and duplicate inputs in tx_package_eval target (Greg Sanders)
a0626ccdad fuzz: allow reaching MempoolAcceptResult::ResultType::DIFFERENT_WITNESS in tx_package_eval target (Greg Sanders)
Pull request description:
Exercises `DIFFERENT_WITNESS` by using "blank" WSH() and allowing witness to determine wtxid, and attempts to make invalid/duplicate inputs.
ACKs for top commit:
dergoegge:
Coverage looks good to me ACK 6a917918b7
Tree-SHA512: db894f5f5b81c6b454874baf11f296462832285f41ccb09f23c0db92b9abc98f8ecacd72fc8f60dc92cb7947f543a2e55bed2fd210b0e8ca7c7d5389d90b14af
821a8a1125 doc: remove x86_64 build assumption from depends doc (fanquake)
Pull request description:
This dates from the introduction of depends, and has not been the case for some time now.
ACKs for top commit:
maflcko:
lgtm ACK 821a8a1125
hebasto:
ACK 821a8a1125.
theuni:
ACK 821a8a1125
Tree-SHA512: 640967a3e6dfab495fd733d3379aa916ac7f67e89a92ef6a94c3bea0494dc7921a9d7485e1b90a1beab00548b575cdab8fb08eb9267dcc5e890cc796ae1b6875
30bd4b1e4a doc: remove mention of missing bdb being a configure error (fanquake)
Pull request description:
This is no-longer the case, unless you're passing additional flags, which is not the case in this example.
ACKs for top commit:
maflcko:
lgtm ACK 30bd4b1e4a
TheCharlatan:
ACK 30bd4b1e4a
hebasto:
ACK 30bd4b1e4a.
Tree-SHA512: b3730546d7ff1f49854b88e710c72c4f6e4b6d238147599d4c4e4adeeb256424c2096635f6c51dcfe2e5a9c1155c1c9915fe03a09c5c38605bee2722756c8f6e
defdf67765 contrib: use a raw string for a regular expression literal that contains backslashes in signet/miner (muxator)
Pull request description:
Running `contrib/signet/miner` under python >= 3.12 causes a `SyntaxWarning`. The problem was already present in previous versions, but it only triggered a `DeprecationWarning`, which was not shown by default.
The change is useful for future-proofing the code base, since future python versions will start to exit with a runtime exception (see the reference given later).
Command to see the warning at runtime under python3.11 (`DeprecationWarning`, needs "-Walways"):
```
$ python3.11 -Walways ./contrib/signet/miner
<BASE>/contrib/signet/miner:33: DeprecationWarning: invalid escape sequence '\d'
RE_MULTIMINER = re.compile("^(\d+)(-(\d+))?/(\d+)$")
2023-11-15 16:02:49 ERROR Must specify command
```
Command to see the warning at runtime under python3.12 (`SyntaxWarning`, no modifiers needed):
```
$ python3.12 ./contrib/signet/miner
<BASE>/contrib/signet/miner:33: SyntaxWarning: invalid escape sequence '\d'
RE_MULTIMINER = re.compile("^(\d+)(-(\d+))?/(\d+)$")
2023-11-15 16:03:00 ERROR Must specify command
```
Reference (https://docs.python.org/3.8/library/re.html):
> Regular expressions use the backslash character ('\') [...]. This collides with Python’s usage of the same character for the same purpose in string literals; [...]
>
> Also, please note that any invalid escape sequences in Python’s usage of the backslash in string literals now generate a DeprecationWarning and in the future this will become a SyntaxError.
>
> The solution is to use Python’s raw string notation for regular expression patterns;
ACKs for top commit:
maflcko:
lgtm ACK defdf67765
ajtowns:
utACK defdf67765
Tree-SHA512: 81bd4892938e7d40a226ca20b5b61ff2470ad763743528da290271faefc535167b56f44665e2d03ed2607c4f7bc8a3200e7931f98fe28dbaf0d2a842c96549f5
Protocol version is no longer needed to work out the serialized size
of objects so drop that information from CSizeComputer and rename the
class to SizeComputer.
