On startup, Ancestor call was taking a lot of time when the node was
loading the blockindex onto memory. This change speeds up the Ancestor
function significantly and speeds up the node during startup.
On testnet3 at blockheight ~2,500,000, the startup was around 30seconds
on current main and was 5 seconds with this change. Below is a benchstat
result showing the significant speedup.
goos: darwin
goarch: arm64
pkg: github.com/utreexo/utreexod/blockchain
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Ancestor-8 120819.301µ ± 5% 7.013µ ± 19% -99.99% (p=0.000 n=10)
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
Ancestor-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
Ancestor-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
isSyncCandidate is now changed to return true even if the peer is a
pruned node if and only if our chaintip is within 288 blocks of the
peer.
Rationale:
Pruned nodes that signal NODE_NETWORK_LIMITED MUST serve 288 blocks from
their chaintip. If our chaintip is within that range, this peer can be
a sync candidate even if they aren't an archival node.
DoubleHashRaw provides a simple function for doing double hashes. Since
it uses the digest instead of making the caller allocate a byte slice, it
can be more memory efficient vs other double hash functions.
now testing that GetBestBlockHashAsync sends the getbestblockhash command via websocket connection and that the channel returned can be used to send the response when it is received
Recent commits 1d6e578 and 72ea23e introduced a change in the way Hashes
are serialized and deserialized. This change could cause errors in
downstream applications that persisted hashes serialized using the previous
methods.
This introduces support for legacy-serialized hashes unmarshaling and restores
the compatibility with previous versions.
This change is part of the effort to add pruning support to btcd.
cfIndex is a useful index even if the node has been pruned so it's
allowed to be enabled together with pruning. However, if the user had
disabled cfindex and enabled pruning, it's not possible to generate
them. In this case, we tell the user that it's impossible unless the
user deletes and start anew.
Additionally, if the user had enabled cfindex and also enabled pruning
from the start, don't let the user turn the cfindex off without dropping
it explicitly. This is to make sure that the user isn't left at an
inconsistent state where the cfindex isn't able to catch up to the tip
because the blocks have already been pruned.
This change is part of the effort to add pruning support to btcd.
It's not possible to generate the addr or tx indexes from scratch if the
block storage had been pruned previously as it's missing the block data.
When the user asks to create these indexes, tell them it's not possible
and the only way it's possible is if they delete and start anew.
This change is part of the effort to add pruning support to btcd.
The addr and tx indexes are not useful when the node is pruned as the
actual block data that the indexes point to are gone. If the user had
previously enabled them, then explicitly require an action from the user
to remove the indexes before letting the user enable pruning.
This change is part of the effort to add pruning support to btcd.
The added *Initialized() functions to each of the indexers allow for
callers to check if each of the indexes have been created. It's
useful for ux improvements where we force the user to manually drop
indexes that aren't compatible with pruning when the user enables
pruning.
This change is part of the effort to add pruning support to btcd.
Now that pruning is allowed in btcd, accurately report the prune status
back to the user.
This change is part of the effort to add pruning support to btcd.
Allowing the user to not pass in the --prune flag after pruning results
in inaccurate reporting of the prune status for getblockchaininfo and
for signaling NODE_NETWORK_LIMITED to peers. Anything that relies on
cfg.Prune to be accurate is at risk of being incorrect.
To solve the current problems and to prevent potential future problems,
just force the user to keep the prune flag on like bitcoind. In terms
of UX, there isn't that much of a loss since if the user wants to keep
more blocks than they previously did, they can just increase the size
passed to --prune.
This change is part of the effort to add pruning support to btcd.
BeenPruned will return true if the database has ever been pruned. This
allows for accurate prune status to be reported as well as ux
improvements by disallowing the user to accidently remove or enable
indexes.
You can have a addrindex but with the actual blocks gone, they won't be
much of a help. Consider allowing these option to be both on in the
future where the addrindex is only indexing the non-pruned blocks.
You can have a txindex but with the actual blocks gone, they won't be
much of a help. Consider allowing these option to be both on in the
future where the txindex is only indexing the non-pruned blocks.
This change is part of the effort to add pruning support to btcd.
Pruning is now available to the end user via --prune flag. There are
checks in place so that the user doesn't go below the minimum prune
target of 1536 MiB. The minimum is set so that we keep at least 288
blocks per the requirement for NODE_NETWORK_LIMITED nodes specified by
BIP0159. The default value of 0 will disable pruning.
This change is part of the effort to add pruning support to btcd.
Wire now supports the ability to signal NODE_NETWORK_LIMITED which
signals to peers that the node is able to serve the last 288 blocks.
Since archival nodes have all blocks, they can also signal for
NODE_NETWORK_LIMITED. SFNodeNetworkLimited flag is added to the default
services.
This change is part of the effort to add pruning support to btcd.
A field for pruning is added and the BlockChain struct is now able to be
configured with pruning. Prune is called on every block connect and
the prune target field is passed to PruneBlocks func. There's no check
to keep the latest 288 blocks to abide by the NODE_NETWORK_LIMITED rule.
That'll have to be enforced by the caller creating the BlockChain
struct.
This change is part of the effort to add pruning support to btcd.
PruneBlocks will prune the earliest block files until it reaches the
given target size. The returned hashes are the hashes of the blocks
that were pruned.
We add a new stepCallback as optional function closure on the Engine
that will be called every time a step has been performed during script
execution.
It is accessed by calling the NewDebugEngine constructor.
This is only meant to be used in debugging.
This change is part of the effort to add pruning support to btcd.
scanBlockFiles nows supports scanning files from an arbitrary block
number. When blocks are pruned, the file number may not start from 0.
To account for this, scanBlockFiles now scans and retreives the start
and the end block file numbers and scans those files.
