Provide a new 'BitcoindDaemon' block notification socket server, to
replace 'com.neemre.btcdcli4j.daemon.BtcdDaemonImpl'. This starts a
single service thread to listen for raw block hashes on localhost port
512*, sent by the specified 'blocknotify' shell/batch script, delegating
to a pool of worker threads to run the supplied BlockListener handler.
Unlike the original BtcdDaemonImpl class, a call to the 'getblock' RPC
method is not made automatically to supply a complete block to the
handler, instead requiring a separate, manual BitcoindClient.getBlock
invocation from within RpcService.
Also provide unit tests using a mock ServerSocket + Socket.
TODO: Use the new Bitcoind(Client|Daemon) implementations in RpcService,
in place of btcdcli4j Btcd(Client|Daemon)Impl & remove the old library.
Seems the persistence at shutdown is too unsafe and we got bug reports where data was missing.
https://github.com/bisq-network/bisq/issues/4806
Use millisec instead of sec for delay
Rename delayInSec to delay
(which can happen in rare cases) and add guards that we never create multiple instances
for a given file as well not call initialize or other API methods after shutdown was started.
Before we use a thread in readFromResources and readAllPersisted. To avoid that client code need to deal with
threading we moved that to the PersistenceManager and changed the API accordingly so it will not return the persisted object but calls a consumer once it is completed with reading.
This fields contains the versions where we have tagged a resource file.
It does not need to be the latest version in case we have not added a new resource file.
This fields contains the versions where we have tagged a resource file.
It does not need to be the latest version in case we have not added a new resource file.
We have written all files each time at each new block which accumulated to about
1 GB of data, this led to stability issues and very high disk IO for the explorer nodes.
Now we write only the data of the new block. We also remove the deletion of all
files at startup.
The dao state is still written in a monolithic file so that cannot be optimized
but we added a new directly where we add each block named by block height.
Looking up the latest written file tells us our last persisted block.
So all that data is equivalent to the monolithic daoState data which should
be removed once the webapp devs have implemented the change to use the blocks directory.
Add shutdown method to TradeStatisticsConverter and call it via TradeStatisticsManager. Now as we have a reference to TradeStatisticsConverter in we don't need the hack anymore in applyInjector.
Set log level for com.neemre.btcdcli4j.core.client.ClientConfigurator to error as there is an expected warn log because of the outdated version.
Rename:
LOW to NETWORK
MID to PRIVATE_LOW_PRIO
HIGH to PRIVATE
Change delay of MID/PRIVATE_LOW_PRIO from 30 min to 2 hours (we had different datastores before using it, now its only real low prio stores)
Add comment to each enum
Make initializePersistenceManager in StorageService abstract to enforce in concrete class to define priority.
Change priorities for future renaming to a different meaning. instead of priority we want to describe the category: private data, public data,.... will come in next commit
Add comment
Use decrementAndGet instead of getAndDecrement
Tested cases when there is an exception at write to disk, but as we call the result handler in the finally clause we get always called onWriteCompleted, so it cannot happen that we get stuck.
at restore from seed and create a backup copy.
I don't know what was the motivation of removing the original file
(probably the use case that the seeds are different and thus there
is no match of existing entries with the new addresses) as
that breaks the internal association of addresses with trades/offers and
the reserved balances display.
This should be tested well though with the related use cases
(diff. offer and trade states, wallet restore with same seed of with
foreign seed -> in that case we need to recreated the AddressEntryList
but that should be handled in the related domain)
- Add copyFile method
In cased new fields are added to PB this case is expected as well in
development if protbuf definitions have changed over time and persisted
data do not match the new definition. We don't want this log to be
that verbose.
The access modifier on CompositeOptionSet was changed to public
in commit c402386, to make it visible to the apitest config
class. The annotation added in this change concisely explains
why the class is not package protected.
Use now the complete object data for signature creation. We use the protobuf data for creating the signature with DER encoding. We restict ourself more regarding backward compatibility but I think it is not a big problem.
If a new field is added to Filter and deployed the maintainer needs to publishes a new filter object. By using the new version he cannot remove the old filter from the network as the protobuf data is different and sig verification on the P2P datastorage level will fail. This is intended to keep the old filter alive for some time to support not updated users. We do not remove invalid filters anymore from our local storage to enable seed nodes to support also old filter objects.
For not updated users the new filter will fail at sig verification because the protobuf data is different. So they ignore the new filter and still use the old filter. For updated users the old filter will fail and the new filter is accepted. As it has a newer date it would also replace the old filter anyway. If the maintainer wants to delete the old filter from the network he can use the old app version and remove the filter. It is recommended to keep a copy of the data directory before the update so that the removal of the older filter is possible.
Refacorings:
- Rename isPeersPaymentAccountDataAreBanned to arePeersPaymentAccountDataBanned
- Rename isSignerPubKeyBanned to isWitnessSignerPubKeyBanned
This commit changes the access modifier on
bisq.common.config.CompositeOptionSet to make
it visible to bisq.apitest.config.ApiTestConfig,
and the duplicate bisq.apitest.config.ApiTestConfig
was deleted from 'apitest'.
Don't include outliers (20% deviation from moving average) in moving
average calculation. It's quite likely that low liquidity markets or
markets with large spreads can't calculate deposit suggestion and will
then suggest deposit from preferences.
Added test for moving average class
We add a second button (displayed as first) to the preferences UI for
resync from latest resources. We add a warning to the resync from
genesis as it takes very long time and causes heavy network load for
seeds.
The resync button at the stateMonitor does now a resync from resources,
not from genesis.
We add handling of the UnconfirmedBsqChangeOutputList file as well.
In Filemanager we add a check if file exists.
We add a second button (displayed as first) to the preferences UI for
resync from latest resources. We add a warning to the resync from
genesis as it takes very long time and causes heavy network load for
seeds.
The resync button at the stateMonitor does now a resync from resources,
not from genesis.
We add handling of the UnconfirmedBsqChangeOutputList file as well.
In Filemanager we add a check if file exists.
There is actually no use case for both username and password forming the
authentication token. This change simplifies the arrangement such that a
single token is passed.
--rpcUser and --rpcPassword are no longer used and are replaced by
--apiToken on both cli and daemon sides.
* Renamed CliCommand to RpcCommand, differentiating it from cmd
string token(s)
* Added new CliConfig, based on :common Config
* Injected Config into CoreApi to make bisq.properties and cmd line
args available to server
* Added cleartext username:password BisqCallCredentials to :cli, and
an AuthenticationInterceptor to :core (server)
* Moved :daemon resources folder to expected location under src/main
* Duplicated CompositeOptionSet, ConfigException and BisqException
in :cli because they are not visible from :common. (TODO refactor?)
CompositeOptionSet will be used to let command line parameters
override params set in config file.
* Removed outdated references to :cli in a couple of comments in
:common Config
* gRPC parameters & command names are lowercase to mimic bitcoind/rpc
TBD
* Decide what rpc auth param names to use, to differentiate them from
bitcoind rpc user/passord param names
Remove unused functionality to pass a custom 'toProto' function via a
constructor or setter, as it was unused and all subclasses override the
toProtoMessage() method anyway. In this way, the method may be removed
and the class (+ UserThreadMappedPersistableList) made abstract.
Also it appears PersistableHashMap was never used, so remove it.
Remove an unused PersistableEnvelope interface from the following five
PersistableNetworkPayload implementations:
AccountAgeWitness, BlindVotePayload, ProposalPayload,
SignedWitness, TradeStatistics2
These already have corresponding *Store envelope classes which correctly
implement the interface.
Protobuf definition files were moved from common and core to a new
protodefinition subproject.
The two main reasons for doing this are to speed up builds by not
having to regenerate common and core protobuf classes
every time a change is made in those subprojects, and to remove
the grpc cli's direct dependency on core, and the transitive dependency
on common.
In order to accomplish this, cli's BisqCliMain was stripped of
its dependencies on common and core. Cli can only get the version
and balance now.
gRPC stub boilerplate was moved from BisqCliMain to a CliCommand
class to avoid some of the bloat that is going to happen as the
read-response loop supports more rpc commands.
Provide UserThreadMappedPersistableList subclass for persistable lists
which need to implement UserThreadMappedPersistableEnvelope, instead of
putting the interface on the base class.
Make the (non-storage) classes MeritList and VoteWithProposalTxIdList
keep the original PersistableList superclass, deriving the remaining
subclasses of PersistableList from the new class instead. In this way,
further persistence-related changes are less likely to inadvertently
alter the behaviour of those two consensus-critical classes.
Removing the superfluous PersistableEnvelope interface from the two
classes (via the base class) will be done in a separate PR.
