b536813cef build: add -fstack-clash-protection to hardening flags (fanquake)
076183b36b build: add -fcf-protection=full to hardening options (fanquake)
Pull request description:
Beginning with Ubuntu `19.10`, it's packaged GCC now has some additional hardening options enabled by default (in addition to existing defaults like `-fstack-protector-strong` and reducing the minimum ssp buffer size). The new additions are`-fcf-protection=full` and `-fstack-clash-protection`.
> -fcf-protection=[full|branch|return|none]
> Enable code instrumentation of control-flow transfers to increase program security by checking that target addresses of control-flow transfer instructions (such as indirect function call, function return, indirect jump) are valid. This prevents diverting the flow of control to an unexpected target. This is intended to protect against such threats as Return-oriented Programming (ROP), and similarly call/jmp-oriented programming (COP/JOP).
> -fstack-clash-protection
> Generate code to prevent stack clash style attacks. When this option is enabled, the compiler will only allocate one page of stack space at a time and each page is accessed immediately after allocation. Thus, it prevents allocations from jumping over any stack guard page provided by the operating system.
If your interested you can grab `gcc-9_9.3.0-10ubuntu2.debian.tar.xz` from https://packages.ubuntu.com/focal/g++-9. The relevant changes are part of the `gcc-distro-specs` patches, along with the relevant additions to the gcc manages:
> NOTE: In Ubuntu 19.10 and later versions, -fcf-protection is enabled by default for C, C++, ObjC, ObjC++, if none of -fno-cf-protection nor -fcf-protection=* are found.
> NOTE: In Ubuntu 19.10 and later versions, -fstack-clash-protection is enabled by default for C, C++, ObjC, ObjC++, unless -fno-stack-clash-protection is found.
So, if you're C++ using GCC on Ubuntu 19.10 or later, these options will be active unless you explicitly opt out. This can be observed with a small test:
```c++
int main() { return 0; }
```
```bash
g++ --version
g++ (Ubuntu 9.3.0-10ubuntu2) 9.3.0
g++ test.cpp
objdump -dC a.out
..
0000000000001129 <main>:
1129: f3 0f 1e fa endbr64
112d: 55 push %rbp
112e: 48 89 e5 mov %rsp,%rbp
1131: b8 00 00 00 00 mov $0x0,%eax
1136: 5d pop %rbp
1137: c3 retq
1138: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
113f: 00
# recompile opting out of control flow protection
g++ test.cpp -fcf-protection=none
objdump -dC a.out
...
0000000000001129 <main>:
1129: 55 push %rbp
112a: 48 89 e5 mov %rsp,%rbp
112d: b8 00 00 00 00 mov $0x0,%eax
1132: 5d pop %rbp
1133: c3 retq
1134: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
113b: 00 00 00
113e: 66 90 xchg %ax,%ax
```
Note the insertion of an `endbr64` instruction when compiling and _not_ opting out. This instruction is part of the Intel Control-flow Enforcement Technology [spec](https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf), which the GCC control flow implementation is based on.
If we're still doing gitian builds for the `0.21.0` and `0.22.0` releases, we'd likely update the gitian image to Ubuntu Focal, which would mean that the GCC used for gitian builds would also be using these options by default. So we should decide whether we want to explicitly turn these options on as part of our hardening options (although not just for this reason), or, we should be opting-out.
GCC has supported both options since 8.0.0. Clang has supported `-fcf-protection` from 7.0.0 and will support `-fstack-clash-protection` in it's upcoming [11.0.0 release](https://clang.llvm.org/docs/ReleaseNotes.html#id6).
ACKs for top commit:
jamesob:
ACK b536813cef ([`jamesob/ackr/18921.1.fanquake.build_add_stack_clash_an`](https://github.com/jamesob/bitcoin/tree/ackr/18921.1.fanquake.build_add_stack_clash_an))
laanwj:
Code review ACK b536813cef
Tree-SHA512: abc9adf23cdf1be384f5fb9aa5bfffdda86b9ecd671064298d4cda0440828b509f070f9b19c88c7ce50ead9ff32afff9f14c5e78d75f01241568fbfa077be0b7
ca185cf5a1 doc: Document differences in bitcoind and bitcoin-qt locale handling (practicalswift)
Pull request description:
Document differences in `bitcoind` and `bitcoin-qt` locale handling.
Since this seems to be the root cause to the locale dependency issues we've seen over the years I thought it was worth documenting :)
Note that 1.) `QLocale` (used by Qt), 2.) C locale (used by locale-sensitive C standard library functions/POSIX functions and some parts of the C++ standard library such as `std::to_string`) and 3.) C++ locale (used by the C++ input/output library) are three separate things. This comment is about the perhaps surprising interference with the C locale (2) that takes place as part of the Qt initialization.
ACKs for top commit:
hebasto:
re-ACK ca185cf5a1
Tree-SHA512: e51c32f3072c506b0029a001d8b108125e1acb4f2b6a48a6be721ddadda9da0ae77a9b39ff33f9d9eebabe2244c1db09e8502e3e7012d7a5d40d98e96da0dc44
8e35bf5906 scripted-diff: rename misbehavior members (John Newbery)
1f96d2e673 [net processing] Move misbehavior tracking state to Peer (John Newbery)
7cd4159ac8 [net processing] Add Peer (John Newbery)
aba03359a6 [net processing] Remove CNodeState.name (John Newbery)
Pull request description:
We currently have two structures for per-peer data:
- `CNode` in net, which should just contain connection layer data (eg socket, send/recv buffers, etc), but currently also contains some application layer data (eg tx/block inventory).
- `CNodeState` in net processing, which contains p2p application layer data, but requires cs_main to be locked for access.
This PR adds a third struct `Peer`, which is for p2p application layer data, and doesn't require cs_main. Eventually all application layer data from `CNode` should be moved to `Peer`, and any data that doesn't strictly require cs_main should be moved from `CNodeState` to `Peer` (probably all of `CNodeState` eventually).
`Peer` objects are stored as shared pointers in a net processing global map `g_peer_map`, which is protected by `g_peer_mutex`. To use a `Peer` object, `g_peer_mutex` is locked, a copy of the shared pointer is taken, and the lock is released. Individual members of `Peer` are protected by different mutexes that guard related data. The lifetime of the `Peer` object is managed by the shared_ptr refcount.
This PR adds the `Peer` object and moves the misbehaving data from `CNodeState` to `Peer`. This allows us to immediately remove 15 `LOCK(cs_main)` instances.
For more motivation see #19398
ACKs for top commit:
laanwj:
Code review ACK 8e35bf5906
troygiorshev:
reACK 8e35bf5906 via `git range-diff master 9510938 8e35bf5`
theuni:
ACK 8e35bf5906.
jonatack:
ACK 8e35bf5906 keeping in mind Cory's comment (https://github.com/bitcoin/bitcoin/pull/19607#discussion_r470173964) for the follow-up
Tree-SHA512: ad84a92b78fb34c9f43813ca3dfbc7282c887d55300ea2ce0994d134da3e0c7dbc44d54380e00b13bb75a57c28857ac3236bea9135467075d78026767a19e4b1
4ec49f8d1e qt: Leverage the default "Create new receiving address" button (Hennadii Stepanov)
4227a8e1f3 qt: Make "Create new receiving address" default unconditionally (Hennadii Stepanov)
Pull request description:
Fix#24
The first commit:
- visual improvement with no behavior change
The second commit:
- removes a bunch of LOCs
- slightly change behavior and makes it standard
With this PR:
ACKs for top commit:
Saibato:
Concept tACK 4227a8e1f34ec49f8d1e
promag:
Tested ACK 4ec49f8d1e on macos.
Tree-SHA512: 3403d5ee96ec139491c7e23b24a24d9239fe55c58d99cbd4cd13bc877f76f992ed011c09e2af35b2a63be1a2371b95f6ac719325396dcc8333cf3eb7fa2e3d2c
7b6d0f10a7 Remove old check for 3-byte shifted IP addresses from pre-0.2.9 node messages (Raúl Martínez (RME))
Pull request description:
The change removes an old check for IPv6 addresses in range ::ff:ff00:0:0:0/72 that were created due to a bug in size field of addr messages for 0.2.8 nodes and before.
This check is no longer needed as they are no more pre 0.2.9 nodes on the network (as per bitnodes network snapshot).
Credits for discovering this go to sipa in https://github.com/bitcoin/bitcoin/pull/19628#discussion_r475907453
Thanks for the attention!
ACKs for top commit:
sipa:
utACK 7b6d0f10a7
vasild:
ACK 7b6d0f1
Tree-SHA512: c5fab59dda2acafe143f607a4c5b636a54ac76fba651cad1ad1b09c94e88ab39503a31c2244c8f2664da68456c2a870c601d8894139c55cde9ece8161913ed2e
d3e8adfada util: remove c-string interfaces for DecodeBase58{Check} (Sebastian Falbesoner)
Pull request description:
This micro-PR gets rid of base58 function interfaces that are redundant in terms of c-string / std::string variants; the c-string interface for `DecodeBase58Check` is completely unused outside the base58 module, while the c-string interface for `DecodeBase58` is only used in unit tests, where an implicit conversion to std::string is not problematic.
ACKs for top commit:
practicalswift:
ACK d3e8adfada -- patch looks correct
laanwj:
Code review ACK d3e8adfada
Tree-SHA512: 006a4a1e23b11385f60820c188b8e6b1634a182ca36e29a6580f72150214c65a3fdb273ec439165f26ba88a42d2bf5bab1cf3666a9eaee222fb4e1c00aeba433
c91b241b48 Updated outdated help command for getblocktemplate (fixes#19625) (Jake Leventhal)
Pull request description:
**Summary of Changes**
* Removed coinbasetxn from the help outputs
* Added the missing name for transactions in the help outputs
* Added help outputs for longpollid and default_witness_commitment
* Added more clarity to capabilities, rules, and coinbaseaux
**Rationale**
The outputs from the help command for `getblocktemplate` are outdated and don't reflect the actual results from `getblocktemplate` (see #19625 for more details)
Fixes#19625.
ACKs for top commit:
laanwj:
ACK c91b241b48
fjahr:
utACK c91b241b48
Tree-SHA512: ee443af4bc3b2838dfd92e2705f344256ee785ae720e505fffea9b0ec5b75930e3b1374bae59b36d5da57c85c9aefe4d62504b028b893d6f2914dccf1e34c658
The change removes an old check for IPv6 addresses in range ::ff:ff00:0:0:0/72 that were created due to a bug in size field of addr messages for 0.2.8 nodes and before.
This check is no longer needed as they are no more pre 0.2.9 nodes on the network (as per bitnodes network snapshot).
Credits for discovering this go to sipa.
df536883d2 chain: Remove UB CChain comparison (Carl Dong)
Pull request description:
Comparing two empty `CChain`s is currently undefined behaviour, and resulted in false assertion failures when comparing identical empty `CChain`s in local testing.
Let's just remove this comparison operator since it doesn't seem to be used anywhere.
ACKs for top commit:
practicalswift:
ACK df536883d2 -- patch is guaranteed to be correct :)
MarcoFalke:
cr ACK df536883d2
Tree-SHA512: db10bac364fc965b56abf7a5bac48018786b14806ffe107e3e8eb24d5004a29331f3387dfe3409a3452a6750d3329e3f354265d787ebb3abfccabe77b28a54d5
cc26fab48d tests: Add fuzzing harness for CNode (practicalswift)
Pull request description:
Add fuzzing harness for `CNode`.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
Top commit has no ACKs.
Tree-SHA512: e6330e5de5b2eb44d3bd91a885e69ebb625bfd1cb2499338aeb3997ff0268848434e651126fe68a8cadd7235c391e61a40d6408ee26e457faf73572e0c375f6b
b6dcc6d741 gui: Clarify block height label (Hennadii Stepanov)
Pull request description:
Prefer "block height" instead of "number of blocks".
This was done while testing https://github.com/bitcoin/bitcoin/pull/16981.
ACKs for top commit:
michaelfolkson:
ACK b6dcc6d741. I don't think there are any other obvious examples in the GUI where "block height" should replace "number of blocks" except for translations.
MarcoFalke:
cr ACK b6dcc6d741
Tree-SHA512: ec3b48c1af5d613ed657ad51f2caddea774376736ecc02343d54518986e35ec37f1745b059814b5be92b5e5c2bb2970d17159b24c6e88b9316803d4de5327c31
Add WalletClient interface so node interface is cleaner and don't need
wallet-specific methods.
The new NodeContext::wallet_client pointer will also be needed to eliminate
global wallet variables like ::vpwallets, because createWallet(), loadWallet(),
getWallets(), etc methods called by the GUI need a way to get a reference to
the list of open wallets if it is no longer a global variable.
Also tweaks splash screen registration for load wallet events to be delayed
until after wallet client is created.
Add AppInitInterfaces function so wallet chain and chain client interfaces are
created earlier during initialization. This is needed in the next commit to
allow the gui splash screen to be able to register for wallet events through a
dedicated WalletClient interface instead managing wallets indirectly through
the Node interface. This only works if the wallet client interface is created
before the splash screen needs to use it.
a99a3c0bd6 rpc: Validate provided keys for query_options parameter in listunspent (pasta)
Pull request description:
At Dash, one of our developers was working with the `listunspent` RPC command, but instead of saying "minimumAmount" he said "minimmumAmount" as such the RPC wasn't working as expected.
In https://github.com/dashpay/dash/pull/3507 we implemented a check so that `listunspent` returns an error if an unrecognized option is given. I figured I might as well adapt the code and throw up a PR here.
Cheers!
ACKs for top commit:
adaminsky:
ACK `a99a3c0bd`
meshcollider:
Seems fine to me. utACK a99a3c0bd6
Tree-SHA512: 9fccf14979849879a51b352afa3e1932ce4a6cfc2ee97b8d405ec6e65673fe94e302795e3ec0b440e6d252f13acda620e1f6a0e86c3fa918883c3fb4600a372c
3de365e4f1 build: replace wingenminiupnpcstrings sed with a patch in miniupnpc package (fanquake)
bbc01a753d build: replace qtranslations lrelease sed with a patch in qt package (fanquake)
c723e4176e build: replace FreeType back-compat sed with a patch in qt package (fanquake)
3aaa39d436 build: replace pwd sed in qt package with a patch (fanquake)
9d440f4e11 build: remove no-longer needed qt workaround (fanquake)
bf85eace1a build: remove no-longer needed qt configure workaround (fanquake)
4af59a407a build: use patch rather than sed in zeromq package (fanquake)
cc107a3af1 build: use patch rather than sed in native_cctools package (fanquake)
865cb23a48 build: use patch rather than sed in fontconfig package (fanquake)
335bd7f8bc build: use patch rather than sed in Boost package (fanquake)
f36140d00c build: use patch rather than sed in bdb package (fanquake)
Pull request description:
While using `sed` can be handy to use for a quick-fix, these instances accumulate, and can become unmaintainable. Not only that, but using sed isn't necessarily robust and it can fail silently. Most of our usage is also missing any documentation explaining why something is being done, when it should be updated/removed etc.
Rather than relying on sed going forward, where possible, I've converted our sed usage into patches. These are easier to maintain, contain documentation, and should fail loudly when they don't apply.
The remaining sed usage, (1 in miniupnpc, the rest in qt), are non-trivial to remove, as they are using build-time variables, or some input from the environment.
This also steals 2 related commits out of #19716.
Related to #16838.
ACKs for top commit:
hebasto:
re-ACK 3de365e4f1, only `drop_lrelease_dependency.patch` updated. Travis makes ARM build without errors now.
theuni:
ACK 3de365e4f1.
Tree-SHA512: b39afcb237e4421f9caabbd665af93fd2e749a1cdd42b1d5ee2261059120005c0e82994d315e679c317d23794eab5c7727f51cae403c94a9c4e4fd7eee9e7ee6
d841301010 test: Add docstring to wait_until() in util.py to warn about its usage (Seleme Topuz)
1343c86c7c test: Update wait_until usage in tests not to use the one from utils (Seleme Topuz)
Pull request description:
Replace global (from [test_framework/util.py](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_framework/util.py#L228)) `wait_until()` usages with the ones provided by `BitcoinTestFramework` and `P2PInterface` classes.
The motivation behind this change is that the `util.wait_until()` expects a timeout, timeout_factor and lock and it is not aware of the context of the test framework. `BitcoinTestFramework` offers a `wait_until()` which has an understandable amount of default `timeout` and a shared `timeout_factor`. Moreover, on top of these, `mininode.wait_until()` also has a shared lock.
closes#19080
ACKs for top commit:
MarcoFalke:
ACK d841301010🦆
kallewoof:
utACK d841301010
Tree-SHA512: 81604f4cfa87fed98071a80e4afe940b3897fe65cf680a69619a93e97d45f25b313c12227de7040e19517fa9c003291b232f1b40b2567aba0148f22c23c47a88
fafc9d5af4 test: Fix intermittent issue in wallet_bumpfee (MarcoFalke)
fa347b2f25 test: Select at least the fee in wallet_bumpfee to avoid negative amounts (MarcoFalke)
Pull request description:
With a "dirty" mempool a transaction might fail to be accepted intermittently. For example,
* https://travis-ci.org/github/bitcoin-core/gui/jobs/719916499#L6773 Fails acceptance
* https://travis-ci.org/github/bitcoin-core/gui/jobs/719916499#L6954 Test fails
Fix the issue by clearing the mempool between subtests
ACKs for top commit:
promag:
Code review ACK fafc9d5af4.
Tree-SHA512: 23fb6decb6343d19eafddcbdb7da0551f6be11325d1c97c30e563944000aeb02bcc4b24904d204b132c093dc1acf28445fa1fd08bfe8d8b52ddd1de51c33eeb6
519cae8fd6 gui: Delay interfaces::Node initialization (Russell Yanofsky)
102abff9eb gui: Replace interface::Node references with pointers (Russell Yanofsky)
91aced7c7e gui: Remove unused interfaces::Node references (Russell Yanofsky)
e133631625 gui: Partially revert #10244 gArgs and Params changes (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This is a partial revert of https://github.com/bitcoin/bitcoin/pull/10244. It changes gui code to go back to using gArgs and Params() functions directly instead of using interfaces::Node to handle arguments.
These changes were originally pushed as part of https://github.com/bitcoin/bitcoin/pull/19461. Motivation is to support a new GUI process connecting to an already running node process. Details are explained in commit messages, but in addition to spawning a new bitcoin-node process, we want bitcoin-gui to connect to an existing bitcoin-node process. So for that reason it should be able to parse its own parameters, rather than rely on the node.
ACKs for top commit:
MarcoFalke:
re-ACK 519cae8fd6, only change is rebase and addressed nits of my previous review last week 🌄
Tree-SHA512: 9c339dd82ba78bcc7b887b84d872f35ccc7dfa3d271691e6eafe8a2048cbbe3bdde1e810ce33d0714d75d048c9de3470e9e9b6f8306a6047d1cb3548f6858dc8
and otherwise log that an unknown INV type was received.
In INV processing, when handling transaction type inv messages,
ToGenTxid() expects that we constructed the CInv ourselves or
that we verified that it is for a transaction type CInv.
Therefore, change this `else` branch into an `else if (inv.GenMsgTx())`
to make this safer and log any INVs that fall through.
The nFetchFlags code can be removed here because GetFetchFlags() can only add
the MSG_WITNESS_FLAG, which is added to the CInv::type field. That CInv is only
passed to AlreadyHave() or ToGenTxid(), and neither of those functions do
anything different depending on whether the CInv type is MSG_TX or
MSG_WITNESS_TX.
Co-authored by: John Newbery <john@johnnewbery.com>
e2aa1a585a util: make EncodeBase64 consume Spans (Sebastian Falbesoner)
2bc207190e util: make EncodeBase32 consume Spans (Sebastian Falbesoner)
Pull request description:
To simplify the interface of the Base32/Base64 encoding functions for raw data, this PR changes them from taking two arguments (pointer and length) to just one Span. Most calls to `EncodeBase64` pass data from `CDataStream` instances, which unfortunately internally work with `char*` pointers rather than `unsigned char*`, but thanks to the recently introduced `MakeUCharSpan` helper, converting them is quite easy.
ACKs for top commit:
MarcoFalke:
ACK e2aa1a585a🐮
vasild:
ACK e2aa1a585
Tree-SHA512: 43bd3bd2ee8e3be2474db0a81dae9d9e88fac2464b96d1b042147106ed7433799dcba3000c69990511ecfc697b0c7306ce85f2ecb2293e2e44fd356c9694b150
This is needed to allow bitcoin-gui to connect to existing node process with
-ipcconnect instead of spawning a new process. It's possible to spawn a new
bitcoin-node process without knowing the current data dir or network, but
connecting to an existing bitcoin-node requires knowing the datadir and network
first.
No change in behavior. Replacing references with pointers allows Node interface
creation to be delayed until later during gui startup next commit to support
implementing -ipcconnect option
