Avoid repurposing the 'ProofOfWork.payload' field for Equihash puzzle
solutions, as that may be of later use in interactive PoW schemes such
as P2P network DoS protection (where the challenge may be a random nonce
instead of derived from the offer ID). Instead, make the payload the
UTF-8 bytes of the offer ID, just as with Hashcash.
Also, make the puzzle seed the SHA-256 hash of the payload concatenated
with the challenge, instead of just the 256-bit challenge on its own, so
that the PoW is tied to a particular payload and cannot be reused for
other payloads in the case of future randomly chosen challenges.
1. Reorder the PoW fields in the 'Filter' proto by field index, instead
of contextually.
2. Deduplicate expression for 'pow' & replace if-block with boolean op
to simplify 'FilterManager::isProofOfWorkValid'.
3. Avoid slightly confusing use of null char as a separator to prevent
hashing collisions in 'EquihashProofOfWorkService::getChallenge'. Use
comma separator and escape the 'itemId' & 'ownerId' arguments instead.
(based on PR #5858 review comments)
Fix a trivial bug in the iterator returned by 'IntListMultimap::get',
caused by mistaken use of the iterator index in place of the key when
doing lookups into the overspill map. This was causing puzzle solutions
to be invalid about 3% of the time, as well as substantially reducing
the average number of solutions found per nonce.
As the fix increases the mean solution count per nonce to the correct
value of 2.0 predicted by the paper (regardless of puzzle params k & n),
inline the affected constants to simplify 'Equihash::adjustDifficulty'.
Remove all the 'challengeValidation', 'difficultyValidation' and
'testDifficulty' BiPredicate method params from 'HashCashService' &
'ProofOfWorkService', to simplify the API. These were originally
included to aid testing, but turned out to be unnecessary.
Patches committed on behalf of @chimp1984.
Change the type of the 'difficulty' field in the Filter & ProofOfWork
proto objects from int32/bytes to double and make it use a linear scale,
in place of the original logarithmic scale which counts the (effective)
number of required zeros.
This allows fine-grained difficulty control for Equihash, though for
Hashcash it simply rounds up to the nearest power of 2 internally.
NOTE: This is a breaking change to PoW & filter serialisation (unlike
the earlier PR commits), as the proto field version nums aren't updated.
Add an abstract base class, 'ProofOfWorkService', for the existing PoW
implementation 'HashCashService' and a new 'EquihashProofOfWorkService'
PoW implementation based on Equihash-90-5 (which has 72 byte solutions &
5-10 MB peak memory usage). Since the current 'ProofOfWork' protobuf
object only provides a 64-bit counter field to hold the puzzle solution
(as that is all Hashcash requires), repurpose the 'payload' field to
hold the Equihash puzzle solution bytes, with the 'challenge' field
equal to the puzzle seed: the SHA256 hash of the offerId & makerAddress.
Use a difficulty scale factor of 3e-5 (derived from benchmarking) to try
to make the average Hashcash & Equihash puzzle solution times roughly
equal for any given log-difficulty/numLeadingZeros integer chosen in the
filter.
NOTE: An empty enabled-version-list in the filter defaults to Hashcash
(= version 0) only. The new Equihash-90-5 PoW scheme is version 1.
Provide a utility method, 'Equihash::adjustDifficulty', to linearise and
normalise the expected time taken to solve a puzzle, as a function of
the provided difficulty, by taking into account the fact that there
could be 0, 1, 2 or more puzzle solutions for any given nonce. (Wagner's
algorithm is supposed to give 2 solutions on average, but the observed
number is fewer, possibly due to duplicate removal.) For tractability,
assume that the solution count has a Poisson distribution, which seems
to have good agreement with the tests.
Also add some (disabled) benchmarks to EquihashTest. These reveal an
Equihash-90-5 solution time of ~146ms per puzzle per unit difficulty on
a Core i3 laptop, with a verification time of ~50 microseconds.
Add a numeric version field to the 'ProofOfWork' protobuf object, along
with a list of allowed version numbers, 'enabled_pow_versions', to the
filter. The versions are taken to be in order of preference from most to
least preferred when creating a PoW, with an empty list signifying use
of the default algorithm only (that is, version 0: Hashcash).
An explicit list is used instead of an upper & lower version bound, in
case a new PoW algorithm (or changed algorithm params) turns out to
provide worse resistance than an earlier version.
(The fields are unused for now, to be enabled in a later commit.)
Run the initial XorTable fillup in 'Equihash::computeAllHashes' in
parallel, using a parallel stream, to get an easy speed up. (The solver
spends about half its time computing BLAKE2b hashes before iteratively
building tables of partial collisions using 'Equihash::findCollisions'.)
As part of this, replace the use of 'java.nio.ByteBuffer' array wrapping
in 'Utilities::(bytesToIntsBE|intsToBytesBE)' with manual for-loops, as
profiling reveals an unexpected bottleneck in the former when used in a
multithreaded setting. (Lock contention somewhere in unsafe code?)
Manually iterate over colliding table rows using a while- loop and a
custom 'PrimitiveIterator.OfInt' implementation, instead of a foreach
lambda called on an IntStream, in 'Equihash::findCollisions'. Profiling
shows that this results in a slight speedup.
Provide a (vastly cut down) drop-in replacement for the Guava multimap
instance 'indexMultimap', of type 'ListMultimap<Integer, Integer>', used
to map table row indices to block values, to detect collisions at a
given block position (that is, in a given table column).
The replacement stores (multi-)mappings from ints to ints in a flat int-
array, only spilling over to a ListMultimap if there are more than 4
values added for a given key. This vastly reduces the amount of boxing
and memory usage when running 'Equihash::findCollisions' to build up the
next table as part of Wagner's algorithm.
Implement the Equihash (https://eprint.iacr.org/2015/946.pdf) algorithm
for solving/verifying memory-hard client-puzzles/proof-of-work problems
for ASIC-resistant DoS attack protection. The scheme is asymmetric, so
that even though solving a puzzle is slow and memory-intensive, needing
100's of kB to MB's of memory, the solution verification is instant.
Instead of a single 64-bit counter/nonce, as in the case of Hashcash,
Equihash solutions are larger objects ranging from 10's of bytes to a
few kB, depending on the puzzle parameters used. These need to be
stored in entirety, in the proof-of-work field of each offer payload.
Include logic for fine-grained difficulty control in Equihash with a
double-precision floating point number. This is based on lexicographic
comparison with a target hash, like in Bitcoin, instead of just
counting the number of leading zeros of a hash.
The code is unused at present. Also add some simple unit tests.
Replace 'BiFunction<T, U, Boolean>' with the primitive specialisation
'BiPredicate<T, U>' in HashCashService & FilterManager.
As part of this, replace similar predicate constructs found elsewhere.
NOTE: This touches the DAO packages (trivially @ VoteResultService).
Avoid repurposing the 'ProofOfWork.payload' field for Equihash puzzle
solutions, as that may be of later use in interactive PoW schemes such
as P2P network DoS protection (where the challenge may be a random nonce
instead of derived from the offer ID). Instead, make the payload the
UTF-8 bytes of the offer ID, just as with Hashcash.
Also, make the puzzle seed the SHA-256 hash of the payload concatenated
with the challenge, instead of just the 256-bit challenge on its own, so
that the PoW is tied to a particular payload and cannot be reused for
other payloads in the case of future randomly chosen challenges.
1. Reorder the PoW fields in the 'Filter' proto by field index, instead
of contextually.
2. Deduplicate expression for 'pow' & replace if-block with boolean op
to simplify 'FilterManager::isProofOfWorkValid'.
3. Avoid slightly confusing use of null char as a separator to prevent
hashing collisions in 'EquihashProofOfWorkService::getChallenge'. Use
comma separator and escape the 'itemId' & 'ownerId' arguments instead.
(based on PR #5858 review comments)
When doing a resync from genesis the number of blocks is limited to 6000
so that requires lots of requests and with that increases risk of broken
connections. Giving more tolerance for retries avoids that the user has
to restart the app.
When syncing from genesis the number of blocks are limited so we get the
`onParseBlockCompleteAfterBatchProcessing` called each time when the received
blocks are processed, and as we are not at wallet height we repeat requesting
blocks. But the new check for the BTC recipient triggers a resync from resource call.
We add now a check that we do this check only once the wallet is synced and our
block height from dao state matches wallet blockheight.
- At genesis we use the genesis height for request (not height+1)
- If wallet is not synced yet we do not call onParseBlockChainComplete (as it was before)
We added 1 as with the lite monitor mode we persist the most recent block,
thus we request with the start height for the next block.
But that cause a problem at a DAO full mode which has lite monitor mode set
as then the block parsing would not be triggered.
We refactor it so that we take the chainHeight from the dao state
directly and add 1 at the requests.
We add a check if we are at chain tip, and if so we skip requests
and call the onParseBlockChainComplete directly.
Add a check of 'scriptTypeId' field, against the output of the spending
tx, to the 'RawTransactionInput::validate' method. Also make the seller
as well as the buyer validate each raw BSQ/BTC input received from the
peer. This prevents either peer from claiming that any of their
non-segwit inputs are segwit in order to underpay the tx fee.
Prevent the seller from stealing the combined tx fee as change by lying
about the value of one or more of his BTC inputs, which are passed to
the buyer as raw inputs in the 'BsqSwapFinalizeTxRequest' message.
To this end, add a 'RawTransactionInput::validate' method to check the
'value' field against the output value of the respective spending tx and
run it on every seller input in 'ProcessBsqSwapFinalizeTxRequest', so
that the buyer is no longer just trusting those numbers.
Additionally, check that the spending txIds from the raw BTC inputs
supplied by the seller actually match those of his signed inputs in the
accompanying partially signed tx, thus tying the raw input values to the
seller's tx.
When doing a resync from genesis the number of blocks is limited to 6000
so that requires lots of requests and with that increases risk of broken
connections. Giving more tolerance for retries avoids that the user has
to restart the app.
