uniqueOpReturnScript is moved to testhelper and is exported so that the
code and be reused in package blockchain without introducing import
cycles. The test code for invalidateblock and reconsiderblock that are
gonna be added in later commits uses the functions.
The variables are moved to testhelper so that they can be reused in the
blockchain package without introducing an import cycle. The testing
code for invalidateblock and reconsiderblock that will be added in later
commits will be using these variables.
solveBlock is moved to testhelper and is exported. This is done so that
the code can be reused without introducing import cycles. The testing
code to be added in alter commits for invalidateblock and reconsider
block will use SolveBlock.
spendableOut and the functions related to it are is moved to package
testhelper and are exported. This is done to make the code reusable
without introducing an import cycle when the testing code for
invalidateblock and reconsiderblock are added in follow up commits.
Some of the functions in difficulty.go are not dependent on any external
functions and they are needed to introduce testing code for the
invalidateblock and reconsiderblock methods that are to be added on in
later commits. Having the workmath package let's us reuse the code and
avoid dependency cycles.
The existing functions that were exported already (HashToBig,
CompactToBig, BigToCompact, CalcWork) are still kept in difficulty.go
to avoid breaking external code that depends on those exported
functions.
Duplicate entries are currently possible in the following scenario:
1: Add entries to the mapslice.
2: 1st map is full. Move onto the 2nd map.
3: Delete any entry in the first map.
4: Attempt to add an entry in the 2nd map.
When attempting (4), the entry should just be overwritten but a
duplicate gets added.
Allowing the caller to fetch from either the database or the cache
resulted in inconsistencies if the cache were ever to be dirty.
Removing this option eliminates this problem.
The assumption in the previous code was incorrect in that we were
assuming that the chainLock is held throughout the entire chain reorg.
This is not the case since the chainLock is let go of during the
callback to the subscribers.
Because of this, we need to ensure that the utxo set is consistent on
each block disconnect. To achieve this, additional flushes are added
during block disconnects.
Also the utxocache is no longer avoided during block connects and when
we're checking for the validity of the block connects and disconnects as
we can just use the cache instead of trying to avoid it.
calculations
Since BlockHeightByHash only returns the heights for blocks that are in
the main chain, when a block that is stale gets pruned, this will cause
an error in the block height lookup and cause an error in block
processing.
Look up the node directly from the index and if the node isn't found,
just skip that node. For utxoCache.lastFlushHash, if that isn't found,
just force a flush.
For various b.sendNotifcation() callbacks, if a runtime panic happens,
we don't get any useful debugging information since the error that
happens first is the "unlock of unlocked mutex" error.
This is because we temporarily unlock the chainLock for callbacks and
then relock them. However, since the relocking code is executed after
the completion of the callback, if an error happens during that
callback, we never relock the chainLock.
Switching to an anonymous function and having the unlock code as a
defer will ensure that the lock always relocks.
existance/non-existance
New test instance BlockDisconnectExpectUTXO tests that a utxo
exists/doesn't exist after a specific block has been disconnected.
Pruning stale blocks will make the block validation fail for the block
that the prune was triggered on as BlockHeightByHash will not return the
height for blocks that are not in the main chain.
We add a test case to ensure that the test fails in the above case.
This commit creates a `RejectReasonMap` to map the errors returned from
`btcd` to bitcoind's `testmempoolaccept` so the `RejectReason` is
unified at the RPC level. To make sure the map keys are unique, the
error strings are modified in `btcd`.
On startup when the headers-first mode is off, when receiving the first
block, the periodic flush will trigger. The lastflushtime wasn't set
which resulted in the flush being triggered on the first block on
restart.
In this commit, we update the top-level btcd package to use the latest
version of btcutil and also the chainhash package. With this version
bump, we can now use the new optimized dsha256 routine where applicable.
With this commit, I've covered most of the areas we'll hash an entire
transaction/block/header, but we may want to optimize some other areas
further, in particular, the witness sighash calc.
If the prune will delete block past the last flush hash of the
utxocache, the cache will need to be flushed first to avoid a case
where the utxocache is irrecoverable. The newly added code adds this
flush logic to connectBlock.
flushNeededAfterPrune returns true if the utxocache needs to be flushed
after the pruning of the given slice of block hashes. For the utxo
cache to be recoverable while pruning is enabled, we need to make sure
that there exists blocks since the last utxo cache flush. If there are
blocks that are deleted after the last utxo cache flush, the utxo set is
irrecoverable. The added method provides a way to tell if a flush is
needed.
This change is part of the effort to add utxocache support to btcd.
utxo cache is now used by the BlockChain struct. By default it's used
and the minimum cache is set to 250MiB. The change made helps speed up
block/tx validation as the cache allows for much faster lookup of utxos.
The initial block download in particular is improved as the db i/o
bottleneck is remedied by the cache.
The implemented utxocache implements connectTransactions just like
utxoviewpoint and can be used as a drop in replacement for
connectTransactions.
One thing to note is that unlike the utxoViewpoint, the utxocache
immediately deletes the spent entry from the cache. This means that the
utxocache is unfit for functions like checkConnectBlock where you expect
the entry to still exist but be marked as spent.
disconnectTransactions is purposely not implemented as using the cache
during reorganizations may leave the utxo state inconsistent if there is
an unexpected shutdown. The utxoViewpoint will still have to be used
for reorganizations.
This change is part of the effort to add utxocache support to btcd.
fetchInputUtxos had mainly 2 functions:
1: Figure out which outpoints to fetch
2: Call fetchUtxosMain to fetch those outpoints
Functionality for (1) is refactored out to fetchInputsToFetch. This is
done to allow fetchInputUtxos to use the cache to fetch the outpoints
as well in a later commit.
This change is part of the effort to add utxocache support to btcd.
Require the caller to pass in the utxoBucket as the caller may be
fetching many utxos in one loop. Having the caller pass it in removes
the need for dbFetchUtxoEntry to grab the bucket on every single fetch.
This change is part of the effort to add utxocache support to btcd.
connectBlock may have an empty utxoviewpoint as the block verification
process may be using the utxo cache directly. In that case, a nil utxo
viewpoint will be passed in. Just return early on a nil utxoviewpoint.
This change is part of the effort to add utxocache support to btcd.
dbPutUtxoView handled putting and deleting new/spent utxos from the
database. These two functinalities are refactored to their own
functions: dbDeleteUtxoEntry and dbPutUtxoEntry.
Refactoring these out allows the cache to call these two functions
directly instead of having to create a view and saving that view to
disk.
This change is part of the effort to add utxocache support to btcd.
The utxoStateConsistency indicates what the last block that the utxo
cache got flush at. This is useful for recovery purposes as if the node
is unexpectdly shut down, we know which block to start rebuilding the
utxo state from.
This change is part of the effort to add utxocache support to btcd.
Getting the memory usage of an entry is very useful for the utxo cache
as we need to know how much memory all the cached entries are using to
guarantee a cache usage limit for the end user.
This change is part of the effort to add utxocache support to btcd.
The fresh flag indicates that the entry is fresh and that the parent
view (the database) hasn't yet seen the entry. This is very useful as
a performance optimization for the utxo cache as if a fresh entry is
spent, we can simply remove it from the cache and don't bother trying to
delete it from the database.
This change is part of the effort to add utxocache support to btcd.
mapslice allows the caller to allocate a fixed amount of memory for the
utxo cache maps without the mapslice going over that fixed amount of
memory. This is useful as we can have variable sizes (1GB, 1.1GB, 2.3GB,
etc) while guaranteeing a memory limit.
This change is part of the effort to add utxocache support to btcd.
sizehelper introduces code for 2 main things:
1: Calculating how many entries to allocate for a map given a size
in bytes.
2: Calculating how much a map takes up in memory given the entries
were allocated for the map.
These functionality are useful for allocating maps so that they'll be
allocating below a certain number of bytes. Since go maps will always
allocate in powers of B (where B is the bucket size for the given map),
it may allocate too much memory. For example, for a map that can store
8GB of entries, the map will grow to be 16GB once the map is full and
the caller puts an extra entry onto the map.
If we want to give a memory guarantee to the user, we can either:
1: Limit the cache size to fixed sizes (4GB, 8GB, ...).
2: Allocate a slice of maps.
The sizehelper code helps with (2).
On startup, Ancestor call was taking a lot of time when the node was
loading the blockindex onto memory. This change speeds up the Ancestor
function significantly and speeds up the node during startup.
On testnet3 at blockheight ~2,500,000, the startup was around 30seconds
on current main and was 5 seconds with this change. Below is a benchstat
result showing the significant speedup.
goos: darwin
goarch: arm64
pkg: github.com/utreexo/utreexod/blockchain
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Ancestor-8 120819.301µ ± 5% 7.013µ ± 19% -99.99% (p=0.000 n=10)
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
Ancestor-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
Ancestor-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
¹ all samples are equal
