Provide a new 'BitcoindDaemon' block notification socket server, to
replace 'com.neemre.btcdcli4j.daemon.BtcdDaemonImpl'. This starts a
single service thread to listen for raw block hashes on localhost port
512*, sent by the specified 'blocknotify' shell/batch script, delegating
to a pool of worker threads to run the supplied BlockListener handler.
Unlike the original BtcdDaemonImpl class, a call to the 'getblock' RPC
method is not made automatically to supply a complete block to the
handler, instead requiring a separate, manual BitcoindClient.getBlock
invocation from within RpcService.
Also provide unit tests using a mock ServerSocket + Socket.
TODO: Use the new Bitcoind(Client|Daemon) implementations in RpcService,
in place of btcdcli4j Btcd(Client|Daemon)Impl & remove the old library.
Seems the persistence at shutdown is too unsafe and we got bug reports where data was missing.
https://github.com/bisq-network/bisq/issues/4806
Use millisec instead of sec for delay
Rename delayInSec to delay
(which can happen in rare cases) and add guards that we never create multiple instances
for a given file as well not call initialize or other API methods after shutdown was started.
Before we use a thread in readFromResources and readAllPersisted. To avoid that client code need to deal with
threading we moved that to the PersistenceManager and changed the API accordingly so it will not return the persisted object but calls a consumer once it is completed with reading.
This fields contains the versions where we have tagged a resource file.
It does not need to be the latest version in case we have not added a new resource file.
This fields contains the versions where we have tagged a resource file.
It does not need to be the latest version in case we have not added a new resource file.
We have written all files each time at each new block which accumulated to about
1 GB of data, this led to stability issues and very high disk IO for the explorer nodes.
Now we write only the data of the new block. We also remove the deletion of all
files at startup.
The dao state is still written in a monolithic file so that cannot be optimized
but we added a new directly where we add each block named by block height.
Looking up the latest written file tells us our last persisted block.
So all that data is equivalent to the monolithic daoState data which should
be removed once the webapp devs have implemented the change to use the blocks directory.
Add shutdown method to TradeStatisticsConverter and call it via TradeStatisticsManager. Now as we have a reference to TradeStatisticsConverter in we don't need the hack anymore in applyInjector.
Set log level for com.neemre.btcdcli4j.core.client.ClientConfigurator to error as there is an expected warn log because of the outdated version.
Rename:
LOW to NETWORK
MID to PRIVATE_LOW_PRIO
HIGH to PRIVATE
Change delay of MID/PRIVATE_LOW_PRIO from 30 min to 2 hours (we had different datastores before using it, now its only real low prio stores)
Add comment to each enum
Make initializePersistenceManager in StorageService abstract to enforce in concrete class to define priority.
Change priorities for future renaming to a different meaning. instead of priority we want to describe the category: private data, public data,.... will come in next commit
Add comment
Use decrementAndGet instead of getAndDecrement
Tested cases when there is an exception at write to disk, but as we call the result handler in the finally clause we get always called onWriteCompleted, so it cannot happen that we get stuck.
at restore from seed and create a backup copy.
I don't know what was the motivation of removing the original file
(probably the use case that the seeds are different and thus there
is no match of existing entries with the new addresses) as
that breaks the internal association of addresses with trades/offers and
the reserved balances display.
This should be tested well though with the related use cases
(diff. offer and trade states, wallet restore with same seed of with
foreign seed -> in that case we need to recreated the AddressEntryList
but that should be handled in the related domain)
- Add copyFile method
In cased new fields are added to PB this case is expected as well in
development if protbuf definitions have changed over time and persisted
data do not match the new definition. We don't want this log to be
that verbose.
The access modifier on CompositeOptionSet was changed to public
in commit c402386, to make it visible to the apitest config
class. The annotation added in this change concisely explains
why the class is not package protected.
Use now the complete object data for signature creation. We use the protobuf data for creating the signature with DER encoding. We restict ourself more regarding backward compatibility but I think it is not a big problem.
If a new field is added to Filter and deployed the maintainer needs to publishes a new filter object. By using the new version he cannot remove the old filter from the network as the protobuf data is different and sig verification on the P2P datastorage level will fail. This is intended to keep the old filter alive for some time to support not updated users. We do not remove invalid filters anymore from our local storage to enable seed nodes to support also old filter objects.
For not updated users the new filter will fail at sig verification because the protobuf data is different. So they ignore the new filter and still use the old filter. For updated users the old filter will fail and the new filter is accepted. As it has a newer date it would also replace the old filter anyway. If the maintainer wants to delete the old filter from the network he can use the old app version and remove the filter. It is recommended to keep a copy of the data directory before the update so that the removal of the older filter is possible.
Refacorings:
- Rename isPeersPaymentAccountDataAreBanned to arePeersPaymentAccountDataBanned
- Rename isSignerPubKeyBanned to isWitnessSignerPubKeyBanned
This commit changes the access modifier on
bisq.common.config.CompositeOptionSet to make
it visible to bisq.apitest.config.ApiTestConfig,
and the duplicate bisq.apitest.config.ApiTestConfig
was deleted from 'apitest'.
Don't include outliers (20% deviation from moving average) in moving
average calculation. It's quite likely that low liquidity markets or
markets with large spreads can't calculate deposit suggestion and will
then suggest deposit from preferences.
Added test for moving average class
We add a second button (displayed as first) to the preferences UI for
resync from latest resources. We add a warning to the resync from
genesis as it takes very long time and causes heavy network load for
seeds.
The resync button at the stateMonitor does now a resync from resources,
not from genesis.
We add handling of the UnconfirmedBsqChangeOutputList file as well.
In Filemanager we add a check if file exists.
We add a second button (displayed as first) to the preferences UI for
resync from latest resources. We add a warning to the resync from
genesis as it takes very long time and causes heavy network load for
seeds.
The resync button at the stateMonitor does now a resync from resources,
not from genesis.
We add handling of the UnconfirmedBsqChangeOutputList file as well.
In Filemanager we add a check if file exists.
There is actually no use case for both username and password forming the
authentication token. This change simplifies the arrangement such that a
single token is passed.
--rpcUser and --rpcPassword are no longer used and are replaced by
--apiToken on both cli and daemon sides.
* Renamed CliCommand to RpcCommand, differentiating it from cmd
string token(s)
* Added new CliConfig, based on :common Config
* Injected Config into CoreApi to make bisq.properties and cmd line
args available to server
* Added cleartext username:password BisqCallCredentials to :cli, and
an AuthenticationInterceptor to :core (server)
* Moved :daemon resources folder to expected location under src/main
* Duplicated CompositeOptionSet, ConfigException and BisqException
in :cli because they are not visible from :common. (TODO refactor?)
CompositeOptionSet will be used to let command line parameters
override params set in config file.
* Removed outdated references to :cli in a couple of comments in
:common Config
* gRPC parameters & command names are lowercase to mimic bitcoind/rpc
TBD
* Decide what rpc auth param names to use, to differentiate them from
bitcoind rpc user/passord param names
Remove unused functionality to pass a custom 'toProto' function via a
constructor or setter, as it was unused and all subclasses override the
toProtoMessage() method anyway. In this way, the method may be removed
and the class (+ UserThreadMappedPersistableList) made abstract.
Also it appears PersistableHashMap was never used, so remove it.
Remove an unused PersistableEnvelope interface from the following five
PersistableNetworkPayload implementations:
AccountAgeWitness, BlindVotePayload, ProposalPayload,
SignedWitness, TradeStatistics2
These already have corresponding *Store envelope classes which correctly
implement the interface.
Protobuf definition files were moved from common and core to a new
protodefinition subproject.
The two main reasons for doing this are to speed up builds by not
having to regenerate common and core protobuf classes
every time a change is made in those subprojects, and to remove
the grpc cli's direct dependency on core, and the transitive dependency
on common.
In order to accomplish this, cli's BisqCliMain was stripped of
its dependencies on common and core. Cli can only get the version
and balance now.
gRPC stub boilerplate was moved from BisqCliMain to a CliCommand
class to avoid some of the bloat that is going to happen as the
read-response loop supports more rpc commands.
Provide UserThreadMappedPersistableList subclass for persistable lists
which need to implement UserThreadMappedPersistableEnvelope, instead of
putting the interface on the base class.
Make the (non-storage) classes MeritList and VoteWithProposalTxIdList
keep the original PersistableList superclass, deriving the remaining
subclasses of PersistableList from the new class instead. In this way,
further persistence-related changes are less likely to inadvertently
alter the behaviour of those two consensus-critical classes.
Removing the superfluous PersistableEnvelope interface from the two
classes (via the base class) will be done in a separate PR.
Make the default toPersistableMessage() method of PersistableEnvelope
simply delegate to Proto.toProtoMessage for speed, so that stores can
explicitly implement (Threaded|UserThreadMapped)PersistableEnvelope if
they actually need concurrency control.
As part of this, make PeerList implement PersistableEnvelope directly
instead of extending PersistableList, as it is non-critical & cloned on
the user thread prior to storage anyway, so doesn't need be thread-safe.
In this way, only PaymentAccountList & small DAO-related stores extend
PersistableList, so they can all be made user-thread-mapped.
After this change, the only concrete store classes not implementing
(Threaded|UserThreadMapped)PersistableEnvelope are:
AccountAgeWitness, BlindVotePayload, ProposalPayload, SignedWitness,
TradeStatistics2, NavigationPath & PeerList
The first five appear to erroneously implement PersistableEnvelope and
can be cleaned up in a separate commit. The last two are non-critical.
(Make NavigationPath.path an immutable list, for slightly better thread
safety anyway - that way it will never be observed half-constructed.)
There are circumstances where input via --appDataDir
will lead to the hiddenservice-directory to be deleted.
Successfully reproduced using
--appDataDir=~/foo
Although the "~" does not get interpreted correctly on
my linux system, it does manage to throw off the
mechanics of sparing the hiddenservice-directory from
being deletd.
Add toProtoMessageSynchronized() default method to PersistableEnvelope,
which performs (blocking) protobuf serialisation in the user thread,
regardless of the calling thread. This should prevent data races like
the ConcurrentModificationException observed in #3752, under the
reasonable assumption that shared persistable objects are only mutated
in the user thread.
Also add a ThreadedPersistableEnvelope sub-interface overriding the
default method above, to let objects which are expensive to serialise
(like DaoStateStore) be selectively serialised in the 'save-file-task-X'
thread as before, but directly synchronised with each mutating op. As
most objects are cheap to serialise, this avoids a noticeable perf drop
without having to track down every mutating method for each store.
In all cases but one, classes implementing ThreadedPersistableEnvelope
are stores like TradeStatistic2Store, with a single ConcurrentHashMap
field. These require no further serialisation, since the map entries are
immutable, so the only mutating operations are map.put(..) calls which
are already synchronised with map reads. (Even if map.values().stream()
sees updates @ different keys happen out-of-order, it should be benign.)
The remaining case is DaoStateStore, which is only ever reset or
modified via a single persist(..) call with a cloned DaoState instance
and hash chain from DaoStateSnapshotService, so there is no aliasing
risk from the various DAO state mutations done in DaoStateService and
elsewhere.
This should fix#3752.
Minor change for consistency: narrow the signature of some remaining
such methods, which have return type 'PersistableEnvelope'.
(This excludes some other cases with return type 'NetworkEnvelope'.)
This change fixes#3998 by avoiding the attempt to create a user's
appDataDir if it already exists. Normally this attempt is not a problem,
as Files#createDirectories does nothing if the target is a directory
that already exists, but in the case that the target is a symbolic link,
Files#createDirectories throws a FileAlreadyExistsException. This change
prevents the call to Files#createDirectories if the target already
exists, and if in fact the target is a symbolic link pointing to an
existing directory, this is not a problem and everything proceeds as per
usual. This mimics the behavior in Bisq v1.2.5 and prior, where
File#mkdirs was used instead of Files#createDirectories. File#mkdirs
does not throw an exception when the target directory is a symlink, it
just returns false.
This is a pure refactoring that renames and inlines variables to tighten
up and make more consistent the implementation of these two methods. It
is done in preparation for a subsequent substantive change that fixes a
bug.
Previously, Config#mkdir and #mkAppDir threw ConfigException if any
underlying IOException was thrown from Files#createDirectory or
Files#createDirectories respectively. This resulted in only a simple
error message being reported at the command line and the details of the
underlying exception's stack trace were lost. This change wraps and
rethrows the underlying IOException as an UncheckedIOException such that
it gets caught by the catch(Throwable) clause in BisqExecutable, which
does print a stacktrace to stderr. This is the way it always should have
worked; throwing ConfigException in these cases was an oversight in the
original implementation. This change also removes the ConfigException
constructor that allows for passing a nested exception to be wrapped as
there is actually no use case for this now that these two anomalies have
been corrected.
This change was made in the process of attempting to reproduce #3998.
Prior to this commit, Bisq's new `Config` infrastructure would throw an
exception on encountering an unrecognized option in the
`bisq.properties` config file and the Bisq application would handle this
by reporting an error at the command line indicating that the given
option was not recognized.
For example, given a bisq.properties file that reads as follows:
$ cat $APP_DATA_DIR/bisq.properties
bogus=42
running Bisq would exit with the following error message:
$ java -jar bisq.jar
error: problem parsing option 'bogus': bogus is not a recognized option.
This was reasonable enough behavior, but it had two problems. The first
is that the error message did not indicate to the user that the
unrecognized option was found in the config file as opposed to the
command line. Users unfamiliar with the config file might not know to
look there to eliminate or fix the offending option. The second problem
surfaced when testing the Windows executable build for the v1.2.6
release. When such an error was encountered, the executable would just
fail silently, reporting nothing to the user.
Both of these problems should be addressed on their own, i.e. option
parsing errors should report to the user whether the offending option
was at the command line or in the config file, and our packaged
executables should never just fail silently if we can avoid it.
However, this change does not address either of these problems directly.
It rather avoids the problems completely by relaxing config file
parsing to silently allow unrecognized options. This behavior mimics
prior (pre-`Config`) Bisq versions, and it also follows suit with
Bitcoin Core's own config file processing.
This fixes#3966, which surfaced this problem. In that particular issue,
an option line had somehow been added to the user's bisq.properties
config file that consisted of an option key containing 452 octal null
characters (\u0000) followed by an equals sign and an empty option
value. It is unknown how this bogus option was ever added to the user's
config file in the first place, but on the chance that previous Bisq
versions somehow added this, the changes in this commit ensure that such
entries will not cause affected users' Bisq applications to silently
fail in versions 1.2.6 and beyond.
Prior to this commit, Bisq's new `Config` infrastructure would throw an
exception on encountering an unrecognized option in the
`bisq.properties` config file and the Bisq application would handle this
by reporting an error at the command line indicating that the given
option was not recognized.
For example, given a bisq.properties file that reads as follows:
$ cat $APP_DATA_DIR/bisq.properties
bogus=42
running Bisq would exit with the following error message:
$ java -jar bisq.jar
error: problem parsing option 'bogus': bogus is not a recognized option.
This was reasonable enough behavior, but it had two problems. The first
is that the error message did not indicate to the user that the
unrecognized option was found in the config file as opposed to the
command line. Users unfamiliar with the config file might not know to
look there to eliminate or fix the offending option. The second problem
surfaced when testing the Windows executable build for the v1.2.6
release. When such an error was encountered, the executable would just
fail silently, reporting nothing to the user.
Both of these problems should be addressed on their own, i.e. option
parsing errors should report to the user whether the offending option
was at the command line or in the config file, and our packaged
executables should never just fail silently if we can avoid it.
However, this change does not address either of these problems directly.
It rather avoids the problems completely by relaxing config file
parsing to silently allow unrecognized options. This behavior mimics
prior (pre-`Config`) Bisq versions, and it also follows suit with
Bitcoin Core's own config file processing.
This fixes#3966, which surfaced this problem. In that particular issue,
an option line had somehow been added to the user's bisq.properties
config file that consisted of an option key containing 452 octal null
characters (\u0000) followed by an equals sign and an empty option
value. It is unknown how this bogus option was ever added to the user's
config file in the first place, but on the chance that previous Bisq
versions somehow added this, the changes in this commit ensure that such
entries will not cause affected users' Bisq applications to silently
fail in versions 1.2.6 and beyond.
* Close stream writer prior to deleting temp file
On Windows, instantiating the stream writer appears to lock the file,
preventing the file from being deleted. This was causing the assertion
that the file is deleted to fail.
* Use proper file path on Windows
On Windows, the exception message contained backwards slashes causing
the test to fail.
* Resolve codacy issues
* Add keyboard shortcut for trade process refresh, fix#3905
Trades have been getting stuck in the `Wait for payment` state, perhaps
due to lost mailbox messages, but it's hard to know for sure. There is
currently no way to get out of this state except going to mediation.
With ctrl+R the seller can ask the buyer to refresh the current trade
state and the buyer will resend the
`CounterCurrencyTransferStartedMessage` if they are in the phase
FIAT_SENT.
* Disallow more than one trade refresh per day
* Add refresh button for seller step 2, fix#3905
A seller can ask to refresh the trade process once every 24 hours. This
step has been a problem causing a lot of mediation lately so this is a
way to ask the buyer to resend the CounterCurrencyTransferStartedMessage
This fixes the problem when a mailbox message was lost. To test the
seller need to not get the first CounterCurrencyTransferStartedMessage
sent by the buyer, for example by letting the seednode drop it instead
of sending to the seller. When button is pressed
- a RefreshTradeStateRequest is sent from seller to buyer
- the buyer receives the RefreshTradeStateRequest and
- ignores it if it's not in FIAT_SENT phase
- responds with a CounterCurrencyTransferStartedMessage if in FIAT_SENT
phase and has already sent a CounterCurrencyTransferStartedMessage
* Fix codacy remarks
Move incoming message handling method to the right section
* Add refresh button info text
Hide refresh section when not available rather than graying out
Added info text:
Sometimes P2P network messages acknowledging payment are not delivered,
causing trades to get stuck. Hit the button below to make your peer
resend the last message.
* Fix codacy issues
Replace tail recursion of the play() method with an ordinary loop, to
prevent a new open JAR resource InputStream + sound file OutputStream
(which were created every 4 minute playback) from accumulating on the
stack, closing them inside the loop instead. (This also prevents
eventual stack overflow.)
Also tidy up FileUtil.resourceToFile and put the JAR URL InputStream in
a try-with-resources block, to ensure that it doesn't leak either.
There is currently no explicit rule for how option-related elements are
ordered in the code, but it is important to understand that the order in
which options are registered via parser.accept() calls is the order in
which they appear in --help output.
It would be nice to group options together into sections and separate
them in the --help output with section headers similar to the way that
Bitcoin Core's help output does it, but this is not a built-in option
with the JOpt Simple library, and not worth trying to hack into place at
the moment.
Previously ConfigTests constructed Config instances with string-based
options, e.g.:
Config config = new Config("--appName=My-Bisq");
The advantage here is clarity, but the downside is repetition of the
option names without any reference to their corresponding Config.*
constants.
One solution to the problem would be to format the option strings using
constants declared in the Config class, e.g.:
Config config = new Config(format("--%s=My-Bisq", APP_NAME));
but this is verbose, cumbersome to read and write and requires repeating
he '--' and '=' option syntax.
This commit introduces the Opt class and the opt() and configWithOpts()
methods to ConfigTests to make testing easier while using constant
references and preserving readability. e.g.:
Config config = configWithOpts(opt(APP_NAME, "My-Bisq"));
In the process of making these changes a bug was discovered in the
monitor submodule's P2PNetworkLoad class and that has been fixed here as
well.
This change also required introducing several option name constants that
had not previously been extracted in order to be referenced within
ConfigTests. For consistency and completeness, all additional option
names that did not previously have a contstant now have one.
Previously (as of the prior commit), a warning was issued if a
non-default config file path was specified at the command line, and then
the default config file path was used as a fallback. On review, however,
it would be better to halt execution immediately if the config file does
not exist. There is no risk of breaking backward compatibility by doing
this as Bisq never had a --configFile option before the recent commits
that introduce it. Furthermore, there is no clear benefit to the
fallback approach. If the user specifies a given config file and it does
not exist, they may not see the warning message in the log, and they may
be left with the impression that they are running against their custom
config file when in fact they are running against the default (which may
be empty or non-existent itself). Thus throwing an exception as is now
done in this commit should make everything more explicit and clear.
This behavior had already been implemented prior to this commit, but has
now been tested and improved with refactoring and logging messages.
Note that this approach emulates Bitcoin Core's own behavior. When
running, for example, `bitcoind -conf=rel/path/to/bitcoin.conf`, the
relative path is prefixed / fully qualified by the value of the
`datadir` option. So if `datadir` equals `~/Library/Application
Support/Bitcoin`, then the `conf` option value above would be fully
qualified as
~/Library/Application Support/Bitcoin/rel/path/to/bitcoin.conf
If the argument to `-conf` is an absolute path, e.g.
`/tmp/bitcoin.conf`, then that absolute path is used without further
modification or qualification. It is assumed that the rationale for this
behavior is to avoid accidentally running against the wrong conf file
because `bitcoind` was invoked in a different directory than usual or
because a malicious actor replaced the relative conf file with their own
version.
Bisq's new `--configFile` option works (and is now tested to work) in
the same way: relative paths get prefixed by the value of
Config.getAppDataDir(), and absolute paths are processed unmodified.
This new class encapsulates all functionality related to detecting a
local Bitcoin node and reporting whether or not it was detected.
Previously this functionality was spread across the Config class
(formerly BisqEnvironment) with its mutable static
isLocalBitcoinNodeRunning property and the BisqSetup class with its
checkIfLocalHostNodeIsRunning method. All of this functionality now
lives within the LocalBitcoinNode class, an instance of which is wired
up via Guice and injected wherever necessary.
Note that the code for detecting whether the node is running has been
simplified, in that it is no longer wrapped in its own dedicated Thread.
There appears to be no performance benefit from doing so, and leaving it
in place would have made testing more difficult than necessary.
Several methods in BisqSetup have also been refactored to accept
callbacks indicating which step should be run next. This has the effect
of clarifying when the step2()-step5() methods will be called.
Previously this static property had been managed within
BaseCurrencyNetwork itself and was accessed directly by callers. Now it
is managed within Config, made private and accessed only via the
new and well-documented baseCurrencyNetwork() method. The same goes for
baseCurrencyNetworkParameters().
It is unfortunate that we must retain these mutable static fields and
accessors, but after trying to eliminate them entirely, this approach is
the lesser of two evils; attempting to use a Config instance and
instance methods only ends up being quite cumbersome to implement,
requiring Config to be injected into many more classes than it currently
is. Getting access to the BaseCurrencyNetwork is basically a special
case, and treating it specially as a static field is in the end the most
pragmatic approach.
Prior to this commit, the way that the appDataDir and its subdirectories
were created was a haphazard process that worked but in a fragile and
non-obvious way. When Config was instantiated, an attempt to call
btcNetworkDir.mkdir() was made, but if appDataDir did not already exist,
this call would always fail because mkdir() does not create parent
directories. This problem was never detected, though, because the
KeyStorage class happened to call mkdirs() on its 'keys' subdirectory,
which, because of the plural mkdirs() call ended up creating the whole
${appDataDir}/${btcNetworkDir}/keys hierarchy. Other btcNetworkDir
subdirectories such as tor/ and db/ then benefited from the hierarchy
already existing when they attempted to call mkdir() for their own dirs.
So the whole arrangement worked only because KeyStorage happened to make
a mkdirs() call and because that code in KeyStorage happened to get
invoked before the code that managed the other subdirectories.
This change ensures that appDataDir and all its subdirectories are
created up front, such that they are guaranteed to exist by the time
they are injected into Storage, KeyStorage, WalletsSetup and TorSetup.
The hierarchy is unchanged, structured as it always has been:
${appDataDir}
└── btc_mainnet
├── db
├── keys
├── wallet
└── tor
Note that the tor/ subdirectory actually gets deleted and re-created
within the TorSetup infrastructure regardless of whether the directory
exists beforehand.
Option handling is now the responsibility of the Config class. JOpt's
OptionParser is no longer passed down to BisqExecutable subclasses'
doExecute method, as they can now rely on the Config abstraction.
In previous commits, BisqEnvironment functionality has been fully ported
to the new, simpler and more type-safe Config class. This change removes
BisqEnvironment and all dependencies on the Spring Framework Environment
interface that it implements.
The one exception is the pricenode module, which is separate and apart
from the rest of the codebase in that it is a standalone, Spring-based
HTTP service.
