The large binary objects in p2p/src/main/resources/ are updated on every
Bisq release with the latest network data to avoid the need for new Bisq
clients to download all of this information from the network, which
would easily overload seed nodes and generally bog down the client.
This approach works well enough for its purposes, but comes with the
significant downside of storing all of this binary data in Git history
forever. The current version of these binary objects total about 65M,
and they grow with every release. In aggregate, this has caused the
total size of the repository to grow to 360M, making it cumbersome to
clone over a low-bandwith connection, and slowing down various local Git
operations.
To avoid further exacerbating this problem, this commit sets these files
up to be tracked via Git LFS. There's nothing we can do about the 360M
of files that already exist in history, but we can ensure it doesn't
grow in this unchecked way going forward. For an understanding of how
Git LFS works, see the reference material at [1], and see also the
sample project and README at [2].
The following command was used to track the files:
$ git lfs track "p2p/src/main/resources/*BTC_MAINNET"
Tracking "p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/BlindVoteStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/DaoStateStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/ProposalStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET"
Tracking "p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET"
We are using GitHub's built-in LFS service here, and it's important to
understand that there are storage and bandwidth limits there. We have
1G total storage and 1G per month of bandwidth on the free tier. We will
certainly exceed this, and so must purchase at least one "data pack"
from GitHub, possibly two. One gets us to 50G storage and bandwith.
In an attempt to avoid unnecessary LFS bandwidth usage, this commit also
updates the Travis CI build configuration to cache Git LFS files, such
that they are not re-downloaded on every CI build (see [3] and [4]
below). With that out of the way, the variable determining whether we
exceed the monthly limit is how many clones we have every month, and
there are many, though it's not clear how many are are Travis CI and how
many are users / developers.
Tracking these files via LFS means that developers will need to have Git
LFS installed in order to properly synchronize the files. If a developer
does not have LFS installed, cloning will complete successfully and the
build would complete successfully, but the app would fail when trying to
actually load the p2p data store files. For this reason, the build has
been updated to proactively check that the p2p data store files have
been properly synchronized via LFS, and if not, the build fails with a
helpful error message. The docs/build.md instructions have also been
updated accordingly.
It is important that we make this change now, not only to avoid growing
the repository in the way described above as we have been doing now for
many releases, but also because we are now considering adding yet more
binary objects to the repository, as proposed at
https://github.com/bisq-network/projects/issues/25.
[1]: https://git-lfs.github.com
[2]: https://github.com/cbeams/lfs-test
[3]: https://docs-staging.travis-ci.com/user/customizing-the-build/#git-lfs
[4]: https://github.com/travis-ci/travis-ci/issues/8787#issuecomment-394202791
Previously, start scripts were generated for both *nix and Windows
platforms, resulting in an unnecessarily cluttered root directory.
With this change, both types of script are still generated, but Windows
.bat scripts are deleted immediately afterward if the user is running a
non-Windows OS (unfortunately, there was no clean way to suppress the
generation of these scripts in the Gradle StartScripts API).
See #1956
This change configures the Gradle build to generate "start scripts" for
each Bisq executable (e.g. Bisq Desktop, Bisq Seednode, etc) in the root
project directory, such that after invoking `./gradle build`, the
following executable scripts become available:
~/Work/bisq-network/bisq
$ ls -1 | egrep '(bisq*|lib)'
bisq-desktop
bisq-desktop.bat
bisq-monitor
bisq-monitor.bat
bisq-relay
bisq-relay.bat
bisq-seednode
bisq-seednode.bat
bisq-statsnode
bisq-statsnode.bat
lib
This makes it possible for users (developers) to easily discover and use
these scripts in an idiomatic and platform-agnostic way as opposed to
the previous situation where we would advise users to run e.g.
java -jar desktop/build/libs/desktop-0.8.0-SNAPSHOT-all.jar
This approach works, but is cumbersome and focuses unnecessarily on the
Java-based nature of the project. Now, with the changes in this commit,
the user would simply run:
./bisq-desktop
The 'lib' directory shown above contains all the jar files necessary to
construct classpaths for these various scripts. The 'cleanInstallDist'
task deletes the 'bisq-*' files and the 'lib' directory, and the default
'clean' task has been configured to depend on the 'cleanInstallDist'
task to ensure this cleanup happens automatically when most users would
expect it.
In the future, these same scripts can be used when installing Bisq
executables properly on users' systems via package managers like Brew
and Apt. The goal is to have the user experience around running
`bisq-desktop` (and more importantly, the forthcoming `bisqd`) be
similar in every way to installing and using `bitcoind`, `lnd` and other
idiomatic *nix-style utilities, be they Bitcoin-related or not.
See the changes in docs/build.md and docs/dev-setup.md for a further
sense of the how this change impacts the developer experience.
