This change is part of the effort to add utxocache support to btcd.
The fresh flag indicates that the entry is fresh and that the parent
view (the database) hasn't yet seen the entry. This is very useful as
a performance optimization for the utxo cache as if a fresh entry is
spent, we can simply remove it from the cache and don't bother trying to
delete it from the database.
This change is part of the effort to add utxocache support to btcd.
mapslice allows the caller to allocate a fixed amount of memory for the
utxo cache maps without the mapslice going over that fixed amount of
memory. This is useful as we can have variable sizes (1GB, 1.1GB, 2.3GB,
etc) while guaranteeing a memory limit.
This change is part of the effort to add utxocache support to btcd.
sizehelper introduces code for 2 main things:
1: Calculating how many entries to allocate for a map given a size
in bytes.
2: Calculating how much a map takes up in memory given the entries
were allocated for the map.
These functionality are useful for allocating maps so that they'll be
allocating below a certain number of bytes. Since go maps will always
allocate in powers of B (where B is the bucket size for the given map),
it may allocate too much memory. For example, for a map that can store
8GB of entries, the map will grow to be 16GB once the map is full and
the caller puts an extra entry onto the map.
If we want to give a memory guarantee to the user, we can either:
1: Limit the cache size to fixed sizes (4GB, 8GB, ...).
2: Allocate a slice of maps.
The sizehelper code helps with (2).
To simplify building the release-grade (stripped and
reproducible) binaries from source, we add the install and
release-install make goals. Running either of the commands will create
binaries in the $GOPATH/bin directories.
The main difference between the two goals is that the release-install
will not contain any local paths and no debug information.
The use of the GO111MODULE environment variable doesn't have any effect
anymore and hasn't for a couple of versions. The default was set to "on"
a while back, so we can remove that variable everywhere.
On startup, Ancestor call was taking a lot of time when the node was
loading the blockindex onto memory. This change speeds up the Ancestor
function significantly and speeds up the node during startup.
On testnet3 at blockheight ~2,500,000, the startup was around 30seconds
on current main and was 5 seconds with this change. Below is a benchstat
result showing the significant speedup.
goos: darwin
goarch: arm64
pkg: github.com/utreexo/utreexod/blockchain
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Ancestor-8 120819.301µ ± 5% 7.013µ ± 19% -99.99% (p=0.000 n=10)
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
Ancestor-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
Ancestor-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
isSyncCandidate is now changed to return true even if the peer is a
pruned node if and only if our chaintip is within 288 blocks of the
peer.
Rationale:
Pruned nodes that signal NODE_NETWORK_LIMITED MUST serve 288 blocks from
their chaintip. If our chaintip is within that range, this peer can be
a sync candidate even if they aren't an archival node.
now testing that GetBestBlockHashAsync sends the getbestblockhash command via websocket connection and that the channel returned can be used to send the response when it is received
Recent commits 1d6e578 and 72ea23e introduced a change in the way Hashes
are serialized and deserialized. This change could cause errors in
downstream applications that persisted hashes serialized using the previous
methods.
This introduces support for legacy-serialized hashes unmarshaling and restores
the compatibility with previous versions.
This change is part of the effort to add pruning support to btcd.
cfIndex is a useful index even if the node has been pruned so it's
allowed to be enabled together with pruning. However, if the user had
disabled cfindex and enabled pruning, it's not possible to generate
them. In this case, we tell the user that it's impossible unless the
user deletes and start anew.
Additionally, if the user had enabled cfindex and also enabled pruning
from the start, don't let the user turn the cfindex off without dropping
it explicitly. This is to make sure that the user isn't left at an
inconsistent state where the cfindex isn't able to catch up to the tip
because the blocks have already been pruned.
This change is part of the effort to add pruning support to btcd.
It's not possible to generate the addr or tx indexes from scratch if the
block storage had been pruned previously as it's missing the block data.
When the user asks to create these indexes, tell them it's not possible
and the only way it's possible is if they delete and start anew.
This change is part of the effort to add pruning support to btcd.
The addr and tx indexes are not useful when the node is pruned as the
actual block data that the indexes point to are gone. If the user had
previously enabled them, then explicitly require an action from the user
to remove the indexes before letting the user enable pruning.
This change is part of the effort to add pruning support to btcd.
The added *Initialized() functions to each of the indexers allow for
callers to check if each of the indexes have been created. It's
useful for ux improvements where we force the user to manually drop
indexes that aren't compatible with pruning when the user enables
pruning.
This change is part of the effort to add pruning support to btcd.
Now that pruning is allowed in btcd, accurately report the prune status
back to the user.
This change is part of the effort to add pruning support to btcd.
Allowing the user to not pass in the --prune flag after pruning results
in inaccurate reporting of the prune status for getblockchaininfo and
for signaling NODE_NETWORK_LIMITED to peers. Anything that relies on
cfg.Prune to be accurate is at risk of being incorrect.
To solve the current problems and to prevent potential future problems,
just force the user to keep the prune flag on like bitcoind. In terms
of UX, there isn't that much of a loss since if the user wants to keep
more blocks than they previously did, they can just increase the size
passed to --prune.
This change is part of the effort to add pruning support to btcd.
BeenPruned will return true if the database has ever been pruned. This
allows for accurate prune status to be reported as well as ux
improvements by disallowing the user to accidently remove or enable
indexes.
You can have a addrindex but with the actual blocks gone, they won't be
much of a help. Consider allowing these option to be both on in the
future where the addrindex is only indexing the non-pruned blocks.
You can have a txindex but with the actual blocks gone, they won't be
much of a help. Consider allowing these option to be both on in the
future where the txindex is only indexing the non-pruned blocks.
This change is part of the effort to add pruning support to btcd.
Pruning is now available to the end user via --prune flag. There are
checks in place so that the user doesn't go below the minimum prune
target of 1536 MiB. The minimum is set so that we keep at least 288
blocks per the requirement for NODE_NETWORK_LIMITED nodes specified by
BIP0159. The default value of 0 will disable pruning.
This change is part of the effort to add pruning support to btcd.
Wire now supports the ability to signal NODE_NETWORK_LIMITED which
signals to peers that the node is able to serve the last 288 blocks.
Since archival nodes have all blocks, they can also signal for
NODE_NETWORK_LIMITED. SFNodeNetworkLimited flag is added to the default
services.
This change is part of the effort to add pruning support to btcd.
A field for pruning is added and the BlockChain struct is now able to be
configured with pruning. Prune is called on every block connect and
the prune target field is passed to PruneBlocks func. There's no check
to keep the latest 288 blocks to abide by the NODE_NETWORK_LIMITED rule.
That'll have to be enforced by the caller creating the BlockChain
struct.
This change is part of the effort to add pruning support to btcd.
PruneBlocks will prune the earliest block files until it reaches the
given target size. The returned hashes are the hashes of the blocks
that were pruned.
