Remove all the 'challengeValidation', 'difficultyValidation' and
'testDifficulty' BiPredicate method params from 'HashCashService' &
'ProofOfWorkService', to simplify the API. These were originally
included to aid testing, but turned out to be unnecessary.
Patches committed on behalf of @chimp1984.
Change the type of the 'difficulty' field in the Filter & ProofOfWork
proto objects from int32/bytes to double and make it use a linear scale,
in place of the original logarithmic scale which counts the (effective)
number of required zeros.
This allows fine-grained difficulty control for Equihash, though for
Hashcash it simply rounds up to the nearest power of 2 internally.
NOTE: This is a breaking change to PoW & filter serialisation (unlike
the earlier PR commits), as the proto field version nums aren't updated.
Add an abstract base class, 'ProofOfWorkService', for the existing PoW
implementation 'HashCashService' and a new 'EquihashProofOfWorkService'
PoW implementation based on Equihash-90-5 (which has 72 byte solutions &
5-10 MB peak memory usage). Since the current 'ProofOfWork' protobuf
object only provides a 64-bit counter field to hold the puzzle solution
(as that is all Hashcash requires), repurpose the 'payload' field to
hold the Equihash puzzle solution bytes, with the 'challenge' field
equal to the puzzle seed: the SHA256 hash of the offerId & makerAddress.
Use a difficulty scale factor of 3e-5 (derived from benchmarking) to try
to make the average Hashcash & Equihash puzzle solution times roughly
equal for any given log-difficulty/numLeadingZeros integer chosen in the
filter.
NOTE: An empty enabled-version-list in the filter defaults to Hashcash
(= version 0) only. The new Equihash-90-5 PoW scheme is version 1.
Provide a utility method, 'Equihash::adjustDifficulty', to linearise and
normalise the expected time taken to solve a puzzle, as a function of
the provided difficulty, by taking into account the fact that there
could be 0, 1, 2 or more puzzle solutions for any given nonce. (Wagner's
algorithm is supposed to give 2 solutions on average, but the observed
number is fewer, possibly due to duplicate removal.) For tractability,
assume that the solution count has a Poisson distribution, which seems
to have good agreement with the tests.
Also add some (disabled) benchmarks to EquihashTest. These reveal an
Equihash-90-5 solution time of ~146ms per puzzle per unit difficulty on
a Core i3 laptop, with a verification time of ~50 microseconds.
Add a numeric version field to the 'ProofOfWork' protobuf object, along
with a list of allowed version numbers, 'enabled_pow_versions', to the
filter. The versions are taken to be in order of preference from most to
least preferred when creating a PoW, with an empty list signifying use
of the default algorithm only (that is, version 0: Hashcash).
An explicit list is used instead of an upper & lower version bound, in
case a new PoW algorithm (or changed algorithm params) turns out to
provide worse resistance than an earlier version.
(The fields are unused for now, to be enabled in a later commit.)
Run the initial XorTable fillup in 'Equihash::computeAllHashes' in
parallel, using a parallel stream, to get an easy speed up. (The solver
spends about half its time computing BLAKE2b hashes before iteratively
building tables of partial collisions using 'Equihash::findCollisions'.)
As part of this, replace the use of 'java.nio.ByteBuffer' array wrapping
in 'Utilities::(bytesToIntsBE|intsToBytesBE)' with manual for-loops, as
profiling reveals an unexpected bottleneck in the former when used in a
multithreaded setting. (Lock contention somewhere in unsafe code?)
Manually iterate over colliding table rows using a while- loop and a
custom 'PrimitiveIterator.OfInt' implementation, instead of a foreach
lambda called on an IntStream, in 'Equihash::findCollisions'. Profiling
shows that this results in a slight speedup.
Provide a (vastly cut down) drop-in replacement for the Guava multimap
instance 'indexMultimap', of type 'ListMultimap<Integer, Integer>', used
to map table row indices to block values, to detect collisions at a
given block position (that is, in a given table column).
The replacement stores (multi-)mappings from ints to ints in a flat int-
array, only spilling over to a ListMultimap if there are more than 4
values added for a given key. This vastly reduces the amount of boxing
and memory usage when running 'Equihash::findCollisions' to build up the
next table as part of Wagner's algorithm.
Implement the Equihash (https://eprint.iacr.org/2015/946.pdf) algorithm
for solving/verifying memory-hard client-puzzles/proof-of-work problems
for ASIC-resistant DoS attack protection. The scheme is asymmetric, so
that even though solving a puzzle is slow and memory-intensive, needing
100's of kB to MB's of memory, the solution verification is instant.
Instead of a single 64-bit counter/nonce, as in the case of Hashcash,
Equihash solutions are larger objects ranging from 10's of bytes to a
few kB, depending on the puzzle parameters used. These need to be
stored in entirety, in the proof-of-work field of each offer payload.
Include logic for fine-grained difficulty control in Equihash with a
double-precision floating point number. This is based on lexicographic
comparison with a target hash, like in Bitcoin, instead of just
counting the number of leading zeros of a hash.
The code is unused at present. Also add some simple unit tests.
Replace 'BiFunction<T, U, Boolean>' with the primitive specialisation
'BiPredicate<T, U>' in HashCashService & FilterManager.
As part of this, replace similar predicate constructs found elsewhere.
NOTE: This touches the DAO packages (trivially @ VoteResultService).
Problem: a
NoSuchMethodError: 'java.util.stream.Collector
com.google.common.collect.ImmutableMultiset.toImmutableMultiset()'
exception was being thrown when testing the previously-merged upgrade to
Gradle 7.3, as described at keybase://chat/bisq#testing/2466.
Solution: This problem is similar to the issue reported at
jeremylong/DependencyCheck#3221. The source of the problem was multiple
conflicting guava jars on the runtime classpath. This commit upgrades to
guava 30.1.1-jre which ensures a single jar on the classpath.
Problem: When merging #5824, the absence of this entry caused a build
failure at dependency verification time against JDK11 and JDK15 on
Ubuntu-latest [1]. It may also cause failures on other JDK/OS
combinations, but the GitHub workflow was aborted before those failures
couldhave occurred. In any case, this omission did not create build
failures on any of the local development machines that tested the
aforementioned PR. Reasons for this discrepancy are unknown.
Solution: manually fetch the pom from [2], run `sha256sum` on it locally
and commit the result to the verification metadata file.
[1]: https://github.com/bisq-network/bisq/runs/4249640611?check_suite_focus=true#step:6:33
[2]: https://repo1.maven.org/maven2/com/fasterxml/jackson/jackson-base/2.11.1/jackson-base-2.11.1.pom
Problem: after the upgrade from 6.6.1 to 7.3, the usual invocation of,
e.g. `./bisq-pricenode 2 2` started failing as reported by @emzy at [1].
bisq@ubuntu-4gb-fsn1-1:~$ /bisq/bisq/bisq-pricenode 2
Error: Could not find or load main class 2
Caused by: java.lang.ClassNotFoundException: 2
Solution: for unknown reasons, the bisq-pricenode script worked as
expected under 6.6.1, i.e. contained the fq main class name in its
scripted invocation of `java -jar ...`, but under 7.3, this main class
name was missing. Through trial and error, it turns out that setting
`mainClassName` explicitly in the :pricenode subproject configuration
solves this problem and makes the start script work as expected.
Presumably, this problem arose in conjunction with the major version
upgrade of the spring boot Gradle plugin that was necessary when
upgrading to Gradle 7.3, but this has not been verified.
[1]: https://github.com/bisq-network/bisq/pull/5824#issuecomment-968276686
Problem: ClassNotFoundError was getting thrown when attempting to run,
e.g. the `./bisq-desktop` or `./bisq-pricenode` start scripts.
Solution: Gradle's posix start scripts were changed significantly
between 6.6.1 and 7.3, including a change to the way the current
directory is determined. This change updates the way we customize start
script generation to allow running them from the root of the repo. This
change only affects the unix / posix variants. The Windows .bat files do
not need to be adapted similarly.
That check is done if:
- If we are after activation time
- If filter is available
- If value in filter is > 0
The check compares the paid fee and the fee value from filter
and requires that paid fee is > 70% of filter value.
See comments in code for more background.
These entries showed up as missing when @jmacxx ran this PR branch on
his local Linux machine under JDK 11. It is not clear why these
dependencies were required there and not elsewhere, e.g. under CI or on
my own Mac.
As mentioned in a prior commit, the upgrade to Gradle 7.x results in
many more dependency declarations in the file, many of which are
effectively duplicates. This change does not attempt to eliminate those
duplications in any clever way, but rather just tidies up and organizes
all dependency declarations by sorting them alphabetically.
