Remove all the 'challengeValidation', 'difficultyValidation' and
'testDifficulty' BiPredicate method params from 'HashCashService' &
'ProofOfWorkService', to simplify the API. These were originally
included to aid testing, but turned out to be unnecessary.
Patches committed on behalf of @chimp1984.
Change the type of the 'difficulty' field in the Filter & ProofOfWork
proto objects from int32/bytes to double and make it use a linear scale,
in place of the original logarithmic scale which counts the (effective)
number of required zeros.
This allows fine-grained difficulty control for Equihash, though for
Hashcash it simply rounds up to the nearest power of 2 internally.
NOTE: This is a breaking change to PoW & filter serialisation (unlike
the earlier PR commits), as the proto field version nums aren't updated.
Add an abstract base class, 'ProofOfWorkService', for the existing PoW
implementation 'HashCashService' and a new 'EquihashProofOfWorkService'
PoW implementation based on Equihash-90-5 (which has 72 byte solutions &
5-10 MB peak memory usage). Since the current 'ProofOfWork' protobuf
object only provides a 64-bit counter field to hold the puzzle solution
(as that is all Hashcash requires), repurpose the 'payload' field to
hold the Equihash puzzle solution bytes, with the 'challenge' field
equal to the puzzle seed: the SHA256 hash of the offerId & makerAddress.
Use a difficulty scale factor of 3e-5 (derived from benchmarking) to try
to make the average Hashcash & Equihash puzzle solution times roughly
equal for any given log-difficulty/numLeadingZeros integer chosen in the
filter.
NOTE: An empty enabled-version-list in the filter defaults to Hashcash
(= version 0) only. The new Equihash-90-5 PoW scheme is version 1.
Provide a utility method, 'Equihash::adjustDifficulty', to linearise and
normalise the expected time taken to solve a puzzle, as a function of
the provided difficulty, by taking into account the fact that there
could be 0, 1, 2 or more puzzle solutions for any given nonce. (Wagner's
algorithm is supposed to give 2 solutions on average, but the observed
number is fewer, possibly due to duplicate removal.) For tractability,
assume that the solution count has a Poisson distribution, which seems
to have good agreement with the tests.
Also add some (disabled) benchmarks to EquihashTest. These reveal an
Equihash-90-5 solution time of ~146ms per puzzle per unit difficulty on
a Core i3 laptop, with a verification time of ~50 microseconds.
Add a numeric version field to the 'ProofOfWork' protobuf object, along
with a list of allowed version numbers, 'enabled_pow_versions', to the
filter. The versions are taken to be in order of preference from most to
least preferred when creating a PoW, with an empty list signifying use
of the default algorithm only (that is, version 0: Hashcash).
An explicit list is used instead of an upper & lower version bound, in
case a new PoW algorithm (or changed algorithm params) turns out to
provide worse resistance than an earlier version.
(The fields are unused for now, to be enabled in a later commit.)
Run the initial XorTable fillup in 'Equihash::computeAllHashes' in
parallel, using a parallel stream, to get an easy speed up. (The solver
spends about half its time computing BLAKE2b hashes before iteratively
building tables of partial collisions using 'Equihash::findCollisions'.)
As part of this, replace the use of 'java.nio.ByteBuffer' array wrapping
in 'Utilities::(bytesToIntsBE|intsToBytesBE)' with manual for-loops, as
profiling reveals an unexpected bottleneck in the former when used in a
multithreaded setting. (Lock contention somewhere in unsafe code?)
Manually iterate over colliding table rows using a while- loop and a
custom 'PrimitiveIterator.OfInt' implementation, instead of a foreach
lambda called on an IntStream, in 'Equihash::findCollisions'. Profiling
shows that this results in a slight speedup.
Provide a (vastly cut down) drop-in replacement for the Guava multimap
instance 'indexMultimap', of type 'ListMultimap<Integer, Integer>', used
to map table row indices to block values, to detect collisions at a
given block position (that is, in a given table column).
The replacement stores (multi-)mappings from ints to ints in a flat int-
array, only spilling over to a ListMultimap if there are more than 4
values added for a given key. This vastly reduces the amount of boxing
and memory usage when running 'Equihash::findCollisions' to build up the
next table as part of Wagner's algorithm.
Implement the Equihash (https://eprint.iacr.org/2015/946.pdf) algorithm
for solving/verifying memory-hard client-puzzles/proof-of-work problems
for ASIC-resistant DoS attack protection. The scheme is asymmetric, so
that even though solving a puzzle is slow and memory-intensive, needing
100's of kB to MB's of memory, the solution verification is instant.
Instead of a single 64-bit counter/nonce, as in the case of Hashcash,
Equihash solutions are larger objects ranging from 10's of bytes to a
few kB, depending on the puzzle parameters used. These need to be
stored in entirety, in the proof-of-work field of each offer payload.
Include logic for fine-grained difficulty control in Equihash with a
double-precision floating point number. This is based on lexicographic
comparison with a target hash, like in Bitcoin, instead of just
counting the number of leading zeros of a hash.
The code is unused at present. Also add some simple unit tests.
Replace 'BiFunction<T, U, Boolean>' with the primitive specialisation
'BiPredicate<T, U>' in HashCashService & FilterManager.
As part of this, replace similar predicate constructs found elsewhere.
NOTE: This touches the DAO packages (trivially @ VoteResultService).
to startPeriodicTasks.
Call that from BisqExecutable.onApplicationLaunched
We have created a non-UI timer before for the
printSystemLoadPeriodically as that was called
before the configUserThread was called (where we
define the UI timer for desktop)
Improve logging
Add BsqBlockStore to protobuf
Remove DaoStateMonitoringService field
Do not persist the blocks in daoState anymore.
This improves persistence performance and reduces memory
requirements for snapshots.
Increase duration for autoReleaseMemory from 60 to 120 sec.
Improve logging.
Add print stack trace when in dev mode to show caller for debugging/tuning.
Remove inefficient GC calls (based on test runs when no reduction occurred at those calls).
Seed nodes have 4 GB allocated so the attempt to reduce memory
footprint is not needed for those cases. We use the fullDaoNode
prog arg as its expected that any full node will have allocated
sufficient memory and thus the intended reduction of memory by
calling the GC is not required and counter productive.
Some i18n property values can be used by the API if long strings are
wrapped before written as commments to json payment account forms, or
written to the CLI console.
This change anticipates the addition of the more complex Swift payment
method (PR 5672). PR 5672's i18n property value for key "payment.swift.info"
will be wrapped and appended to the comments of the Swift payment account's
json form.
