The large binary objects in p2p/src/main/resources/ are updated on every
Bisq release with the latest network data to avoid the need for new Bisq
clients to download all of this information from the network, which
would easily overload seed nodes and generally bog down the client.
This approach works well enough for its purposes, but comes with the
significant downside of storing all of this binary data in Git history
forever. The current version of these binary objects total about 65M,
and they grow with every release. In aggregate, this has caused the
total size of the repository to grow to 360M, making it cumbersome to
clone over a low-bandwith connection, and slowing down various local Git
operations.
To avoid further exacerbating this problem, this commit sets these files
up to be tracked via Git LFS. There's nothing we can do about the 360M
of files that already exist in history, but we can ensure it doesn't
grow in this unchecked way going forward. For an understanding of how
Git LFS works, see the reference material at [1], and see also the
sample project and README at [2].
The following command was used to track the files:
$ git lfs track "p2p/src/main/resources/*BTC_MAINNET"
Tracking "p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/BlindVoteStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/DaoStateStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/ProposalStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET"
We are using GitHub's built-in LFS service here, and it's important to
understand that there are storage and bandwidth limits there. We have
1G total storage and 1G per month of bandwidth on the free tier. We will
certainly exceed this, and so must purchase at least one "data pack"
from GitHub, possibly two. One gets us to 50G storage and bandwith.
In an attempt to avoid unnecessary LFS bandwidth usage, this commit also
updates the Travis CI build configuration to cache Git LFS files, such
that they are not re-downloaded on every CI build (see [3] and [4]
below). With that out of the way, the variable determining whether we
exceed the monthly limit is how many clones we have every month, and
there are many, though it's not clear how many are are Travis CI and how
many are users / developers.
Tracking these files via LFS means that developers will need to have Git
LFS installed in order to properly synchronize the files. If a developer
does not have LFS installed, cloning will complete successfully and the
build would complete successfully, but the app would fail when trying to
actually load the p2p data store files. For this reason, the build has
been updated to proactively check that the p2p data store files have
been properly synchronized via LFS, and if not, the build fails with a
helpful error message. The docs/build.md instructions have also been
updated accordingly.
It is important that we make this change now, not only to avoid growing
the repository in the way described above as we have been doing now for
many releases, but also because we are now considering adding yet more
binary objects to the repository, as proposed at
https://github.com/bisq-network/projects/issues/25.
[1]: https://git-lfs.github.com
[2]: https://github.com/cbeams/lfs-test
[3]: https://docs-staging.travis-ci.com/user/customizing-the-build/#git-lfs
[4]: https://github.com/travis-ci/travis-ci/issues/8787#issuecomment-394202791
Merging PR #4096, which moved protobuf defs out of core and common,
left :seednode without its required dependency on guava, causing
NoSuchMethodErrors.
This change forces :relay's grpc-auth version to match the
version of all other io.grpc-* dependencies, and gets rid of
the duplcate gson dependency v2.8.2
Replaced the Scanner input read loop with upgraded joptsimple
dependency. Cli now takes a single, non-option program argument, runs
it and exits. Also removed the "stop client" command because there is
no input loop, but shutdown() is called for orderly channel shudown
before the jvm terminates. Also changed cmd syntax from camel case
to lowercase, mimicking bitcoin-cli.
Configured logback to supress all debug & info level netty output, and
bypassed logback to print results to System.out.
This is done primarily for concision. This change also repackages
bisq.grpc => bisq.proto.grpc in anticipation of repackaging the
definitions in pb.proto from 'protobuf' to 'bisq.proto'. There should
not be any compatibility issues with doing this, but it's out of scope
here. When complete, the relationship between bisq.proto.grpc and
bisq.proto will be more intuitively clear, i.e. that bisq.proto.grpc has
certain dependencies on bisq.proto classes, but not the other way
around.
Protobuf definition files were moved from common and core to a new
protodefinition subproject.
The two main reasons for doing this are to speed up builds by not
having to regenerate common and core protobuf classes
every time a change is made in those subprojects, and to remove
the grpc cli's direct dependency on core, and the transitive dependency
on common.
In order to accomplish this, cli's BisqCliMain was stripped of
its dependencies on common and core. Cli can only get the version
and balance now.
gRPC stub boilerplate was moved from BisqCliMain to a CliCommand
class to avoid some of the bloat that is going to happen as the
read-response loop supports more rpc commands.
Some apache & logback dependency versions were bumped up, some
transitive dependencies declared as direct dependencies, and some
new exclusions were added to reduce the number of duplicated
dependencies in the build.
Also updated gradle-witness.gradle.
Here are some of the duplicated apache and logback dependencies
in the current build:
commons-codec-1.10.jar
commons-codec-1.9.jar
commons-io-2.4.jar
commons-io-2.6.jar
commons-logging-1.1.1.jar
commons-logging-1.2.jar
httpclient-4.0.1.jar
httpclient-4.5.3.jar
httpclient-4.5.5.jar
httpcore-4.0.1.jar
httpcore-4.4.6.jar
httpcore-4.4.9.jar
logback-classic-1.1.10.jar
logback-classic-1.1.11.jar
logback-core-1.1.10.jar
logback-core-1.1.11.jar
Upgraded:
codecVersion 1.9 -> 1.13
ioVersion 2.4 -> 2.6
langVersion 3.4 -> 3.8
httpclientVersion 4.5.3 -> 4.5.12
slf4jVersion 1.7.22 -> 1.7.25
New dependency declarations:
loggingVersion 1.2
httpcoreVersion 4.4.13
This commit reduces the number of build dependencies by 10.
Partial fix for #4086
In conjuction with the previous commits, this change removes entirely
Bisq's dependency on the Spring Framework, with the exception of the
pricenode module, which is not a Bisq node per se, but a standalone
Spring-based HTTP service.
Note that the removal of the Gradle `exclude` directive for
commons-logging is because Apache HttpClient still depends on it and
Spring had previously been providing it transitively. If this `exclude`
directive is not removed, NoClassDefFound errors get thrown at runtime
from HttpClient code unable to load the commons-logging Logger class.
Prior to this commit, BisqExecutable has been responsible for parsing
command line and config file options and BisqEnvironment has been
responsible for assigning default values to those options and providing
access to option values to callers throughout the codebase.
This approach has worked, but at considerable costs in complexity,
verbosity, and lack of any type-safety in option values. BisqEnvironment
is based on the Spring Framework's Environment abstraction, which
provides a great deal of flexibility in handling command line options,
environment variables, and more, but also operates on the assumption
that such inputs have String-based values.
After having this infrastructure in place for years now, it has become
evident that using Spring's Environment abstraction was both overkill
for what we needed and limited us from getting the kind of concision and
type saftey that we want. The Environment abstraction is by default
actually too flexible. For example, Bisq does not want or need to have
environment variables potentially overriding configuration file values,
as this increases our attack surface and makes our threat model more
complex. This is why we explicitly removed support for handling
environment variables quite some time ago.
The BisqEnvironment class has also organically evolved toward becoming a
kind of "God object", responsible for more than just option handling. It
is also, for example, responsible for tracking the status of the user's
local Bitcoin node, if any. It is also responsible for writing values to
the bisq.properties config file when certain ban filters arrive via the
p2p network. In the commits that follow, these unrelated functions will
be factored out appropriately in order to separate concerns.
As a solution to these problems, this commit begins the process of
eliminating BisqEnvironment in favor of a new, bespoke Config class
custom-tailored to Bisq's needs. Config removes the responsibility for
option parsing from BisqExecutable, and in the end provides "one-stop
shopping" for all option parsing and access needs.
The changes included in this commit represent a proof of concept for the
Config class, where handling of a number of options has been moved from
BisqEnvironment and BisqExecutable over to Config. Because the migration
is only partial, both Config and BisqEnvironment are injected
side-by-side into calling code that needs access to options. As the
migration is completed, BisqEnvironment will be removed entirely, and
only the Config object will remain.
An additional benefit of the elimination of BisqEnvironment is that it
will allow us to remove our dependency on the Spring Framework (with the
exception of the standalone pricenode application, which is Spring-based
by design).
Note that while this change and those that follow it are principally a
refactoring effort, certain functional changes have been introduced. For
example, Bisq now supports a `--configFile` argument at the command line
that functions very similarly to Bitcoin Core's `-conf` option.
- Rename package bisq.grpc => bisq.daemon.app
- Rename BisqGrpcApp => BisqDaemon
- Rename BisqGrpcServerMain => BisqDaemonMain
The script `bisq-grpc` has been renamed to `bisq-daemon` accordingly
(and will later be customized to `bisqd`). To see everything in action,
issue the following commands:
$ gradle build
$ ./bisq-daemon --appName=Bisq-throwaway --daoActivated=false
$ echo getVersion | ./bisq-cli # in a second terminal
1.2.3
The :grpc module will soon be renamed to :daemon. These two modules
represent two separate and equal modes of running bisq, either as a
desktop GUI or as a daemon. They are both applications, and one should
not depend on the other as it would be illogical and confusing to model
things that way. The reason for the current dependency from :desktop to
:grpc is because :grpc is the home of BisqGrpcServer. This change moves
this class up to :core, in a new bisq.core.grpc package, such that both
the :desktop and :daemon applications can pull it in cleanly.
The CoreApi 'facade' that BisqGrpcServer uses to abstract away bisq
internals has been moved from bisq.core to bisq.core.grpc as well and
for the same reasons detailed in 8b30c22d6.
This change also renames the Java package for generated grpc types from
bisq.grpc.protobuf to bisq.core.grpc (the same new package that
BisqGrpcServer and CoreApi now live in). Again, this is for reasons of
cohesion: BisqGrpcServer is the only user of these grpc-generated types,
and they should logically live in the same package (even if they
physically live in separate source dirs at the build level).
This change:
- Removes several superfluous dependencies not required for our
purposes with gRPC
- Cleans up the way Gradle source sets are managed for generated gRPC
sources and classes
- Makes use of Gradle's new `implementation`, `compileOnly` and
`runtimeOnly` dependency configurations where changes were otherwise
being made. See https://stackoverflow.com/a/47365147 for details.
Remaining uses of the now-deprecated `compile` and `runtime`
configurations should be eliminated in a refactoring separate and
apart from the present gRPC API work.
- Upgrades several existing dependencies to align with newer versions
of the same dependencies introduced transitively by grpc-* 1.25.0
libraries, including:
- protoc from 3.9.1 => 3.10.0
- gson from 2.7 => 2.8.5
Note that a number of the grpc-* libraries depend on Guava v28, and our
existing dependency on Guava v20 has *not* been upgraded to this newer
version because it is incompatible with the way we have used Guava's
Futures API. It appears that the grpc-* libraries function correctly
against this older version of Guava, and more investigation would be
required see whether upgrading our uses to the new Guava API is feasible
/ worth it. The way we are preventing this upgrade is with the use of
`exclude(module: "guava")` directives on grpc-* dependencies.
This change stubs out the `bisq-cli` utility with a placeholder main
method, such that the following now works:
$ gradle :cli:build
$ ./bisq-cli
Hello, World!
This reverts commit 26c053dae8 because
Kotlin compilation slows down the build, was applied too broadly to all
modules instead of just the one that needed it, and most importantly
because we never actually went ahead with converting anything of
importance to Kotlin. The commit being reverted was basically a demo,
converting a single test type to show what kind of difference it would
make.
This commit introduces a new `grpc` module including the following key
types:
- BisqGrpcServer: The API implementation itself, along with generated
gRPC Response/Reploy types defined in grpc/src/main/proto/grpc.proto.
- BisqGrpcServerMain: A 'headless' / daemon-like entry point for
running a Bisq node without the JavaFX desktop UI.
- BisqGrpcClient: A simple, repl-style client for the API that allows
the user to exercise the various endpoints as seen in the example
below.
In the `desktop` module, the BisqAppMain class has been modified to
start a BisqGrpcServer instance if the `--desktopWithGrpcApi` option has
been set to `true`.
In the `core` module, a new `CoreApi` class has been introduced
providing a kind of comprehensive facade for all Bisq functionality to
be exposed via the RPC API.
How to explore the proof of concept:
1. Run the main() method in BisqAppMain providing
`--desktopWithGrpcApi=true` as a program argument or alternatively, run
the main() method in BisqGrpcServerMain, where no special option is
required. In either case, you'll notice the following entry in the log
output:
INFO bisq.grpc.BisqGrpcServer: Server started, listening on 8888
2. Now run the main() method in BisqGrpcClient. Once it has started up
you are connected to the gRPC server started in step 1 above. To
exercise the API, type `getVersion` via stdin and hit return. You
should see the following response:
INFO bisq.grpc.BisqGrpcClient - 1.2.4
Likewise, you can type `getBalance` and you'll see the following
response:
INFO bisq.grpc.BisqGrpcClient - 0.00 BTC
and so forth for each of the implemented endpoints. For a list of
implemented endpoints, see BisqGrpcServer.start().
Note once again that the code here is merely a proof of concept and
should not be considered complete or production-ready in any way. In a
subsequent commit, the `--desktopWithGrpcApi` option will be disabled in
order to avoid any potential production use.
The content of this commit is the result of squashing a number of
commits originally authored by chimp1984 in the `chimp1984` fork's `grpc`
branch.
Co-authored-by: Chris Beams <chris@beams.io>
After the upgrade from Gradle 4.10.2 to 5.6.4 in commit 5fe71fa0a,
all of Bisq's shadowJar tasks started failing Gradle's incremental build
checks. This meant that repeated invocations of `gradle build` went from
a handful of seconds to more than a minute, because shadowJar tasks had
to be re-executed on every build.
For example, with --info enabled, one would see entries like this in the
build output:
> Task :seednode:shadowJar
Custom actions are attached to task ':seednode:shadowJar'.
Caching disabled for task ':seednode:shadowJar' because:
Caching has not been enabled for the task
Task ':seednode:shadowJar' is not up-to-date because:
Output property 'archiveFile' file [...]libs/seednode.jar has changed.
This problem was solved by in johnrengelman/shadow#524 and made
available in the project's recent 5.0.2 release. This patch simply bumps
the shadow plugin version to that value, and gets us back to snappy
incremental builds, e.g.:
$ gradle build
BUILD SUCCESSFUL in 7s
