* WIP
* Modify DataMessageHandler to use Set for filter batch, fix test case
* revert log level
* Remove duplicate filters test
* Re-add test to make sure we don't throw an exception when processing a filter we've seen before
* Empty commit to re-run CI
* Empty commit to re-run CI
* Empty commit to re-run CI
* Fix NeutrinoNodeTest
* Empty commit to re-run CI
* Empty commit to re-run CI
* Segregate P2PClientTest and P2PClientActorTest to make it easier to test the actor
* Fix cleanup code
* Clean up tables for postgres test case
* Fix BitcoinSAppConfigBitcoinFixtureStarted.afterAll()
* Empty commit
* Add forceNamedWallet parameter to for postgres test cases where we need multiple wallets. This prevents them from writing to the same postgres schema by making unique wallet names
* Cleanup:
* Empty commit
* Make PeerManager.finder private
* Make PeerManager.supervisor private
* Write unit test to see if default peer is added to PeerManager
* Use withNeutrinoNodeUnstarted fixture
* Get unit tests passing
* Add println to try and debug CI
* Add more println
* more println
* Adjust nodeTest / parallelExecution = false
* Try to make error logged to figure out why failure is happening on CI
* Adjust log level to ERROR
* Add catch to try and log failure messages
* Add log for sending messages to peers
* Try to increase threadpool size to see if we are deadlocking
* Add more logs
* Add another log to detect where failure is
* Try using Actor.tell() with explicit ActorRef.noSender
* Very detailed logging
* Remove duplicate method from rebase
* Revert things
* Revert build.sbt
* Empty commit
* Bump timeout
* Empty commit to run CI
* Revert more files
* Move handle network received message into awaitNetworkRequest() context
* Fix bug where I wasn't wrapping in NetworkMessageReceived
* revert logging
* remove more uncessary logs
* Empty commit
* Pull things over from 4950 that shouldn't cause issues to reduce size of 4950
* Reduce timeout for processing a batch of messages from 1000 seconds -> 60 seconds
* scalafmt
* Empty commit
* WIP
* WIP2
* Get all DataMessageHandlerTest passing
* Get NeutrinoNodeTest passing
* Fix NeutrinoNodeWithUncachedBitcoindTest unit test
* Use chainApi to detect if we are in IBD
* Empty commit
* Revert logging levels
* add header sync validation
* fix docs, minor fixes
* Refactor to us InvalidBlockHeader ChainException, also refactor recovery to private helper method
* changes from comments
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* Implement isinitialblockdownload flag in databaes
* Change to IsInitialBlockDownload to be less confusing
* Fix missing refactors, add ChainApi.isIBD()
* Implement logic for ChainApi.isIBD(), add configuration flag to toggle whether to emit websocket events while IBD is ongoing or not, add isinitialblockdownload field to getinfo endpoint
* Remove confusing ChainApi.isIBD() that didn't accurately account for the case where IBD wasn't set in the database
* Fix bug around emitting the websocket events
* Add some documentation
* removed logger parameter from where it's used with callbacks
* removed logger from onTXRecived... and onBlockRecived...
* removed unused imports
* removed logger from walletCallbacks
* Replace BoundedSourceQueueWithComplete with SourceQueueWithComplete so we can use the returned Future
* Recover the expected exceptions in CallBackUtilTest
* add P2PClientSupervisor
* changes from comments: made P2PClient Future
* empty commit to see if mac failures are consistent on ci
* changes from comments
* changes from comments
* Move pollBitcoind out of startBitcoindBlockPolling
* Rework startBitcoindBlockPolling to not return Future[Cancellable]
* Stop NodeCallbacksStreamManager inside of NodeAppConfig.stop()
* Make BitcoindSyncState which encapsulates syncing with bitcoind and polling of mempool
* Fix bug where processingBitcoindBlocks cannot be top level val as that method may be used by different parts of the codebase
* Rename isPolling -> processingBitcoindBlocks
* Cleanup more resources in appServerTest/test
* Complete polling stream regardless if we synced blocks are not to complete the Future returned by pollBitcoind
* Empty commit to run CI
* Move MockChainQueryApi/NodeApi out of BaseWalletTest
* refactor BitcoinSServerMain.start() to return WalletHolder
* Add walletConfig to WalletWithBitcoind
* Move findOutputsBeingSpent into WalletApi
* Add WalletApi.clearAllAddresses(), HDWalletApi.{findAccount, getnewaddress(account)}
* Add HDWalletApi.fundRawTransaction() with an account as parameter
* Add WalletApi.findByScriptPubKey()
* Fix lots of tests in WalletIntegrationTest
* Create WalletApi.processOurTransaction()
* Get things compiling
* Fix tag integration test
* Refactor AnyDLCHDWalletApi -> DLCNeutrinoHDWalletApi
* Fix docs
* Get postgres tests passing locally
* Move initBalance map before calling callback in CallBackUtilTest
* Get compile working, rename to destroyOnlyWalletWithBitcoindCached
* Fix docs
* Fix missing destroyWalletAppConfig
* Fix scalafmt
* Fix bug in dlcWalletTest where wallet db thread pools weren't being shutdown after unit test completes
* Empty commit
* Implement akka stream proxy for nodecallbacks
Add killswitch to createBitcoindNodeCallbacksForWallet
Add unit test for killswitch in createBitcoindNodeCallbacksForWallet
Add delays to make sure callbacks are executed
Fix rebase
Move killswitch out of methods into class level val
* Bump timeout
* Attempt to implement NodeCallbacks proxy with akka streams
* Refactor CallbackUtil to use NodeCallbackStreamManager
* Rebase & remove killswitch
* Implement NodeCallbackStreamManager.+ method
* Add `sync` JSON field to the `getinfo` endpoint output
* improve test coverage
* update docs
* rename sync flag
* WebSocket notifications
* fix unit tests
* fix unit tests
* increase test timout
* add support to find and switch peers
* fix compile on 2.12
* allow empty config peers in regtest
* fix test
* minor fixes
changes timeouts, fix issue with ipv6 dns seeds, initDisconnect when disconnected bug, dbPeers order fix
* fix: not removing peers on initialization timeout
* fix: query again when previous failed
* fix: wrong condition for deferred peers
* restore log levels
* clean up
* add PeerStack to allow trying peers based on priority values
* fix migrations
* changes from comments
* use StartStopAsync
* changes from comments
* fix switch if peer down test
* changes from comments
* Get things compiling and tests passing
* Fix bug where we weren't using filter height correctly
* Fix docs
* Move creationTime into KeyManagerAppConfig
* Fix nodeTest/test test fixtures
* Refactor testkit tests to use same nodeCallbacks that BitcoinSServerMain uses for neutrino
* Add test case
* Add logs for callbacks
* Cleanup
* Add test case for when funds are spent when we are offline
* Turn off logging again
* Cleanup more logs and comments
* WIP
* Finish implementing {CRUD,CRUDAction}.createAllAction()
* Try to make database inserts more atomic
* Consolidate more database calls in DLCWallet.initDLCForAccept()
* WIP2
* More refactor and using actions
* Add DLCIdDaoUtilNoPK for querying when dlcId is not a primary key
* Add DLCDataManagement.deleteDLC()
* make DLCWalletApi.createDLCOffer() use actions
* Rework DLCWallet.createNewDLCAccept() to use DBIOAction
* Use actions in DLCWallet.registerDLCAccept() a bit
* Create DLCActionBuilder, move actions to there
* Move more actions into ActionBuilder
* Implement transactionally on database writes for actions
* Fix case where we forget to correctly assign currentPeerMsgHandlerRecv when we disconnect
* Make handleNodeCommand() so we handle P2PClient.CloseCommand
* Enable DEBUG logging to see what is happening on CI
* Add PeerMessageReceiverTest
* Add InitializeDisconnectDone
* Bump timeout on condition
* P2P reconnect logic
* some fixes
* more changes
* fix DuplicateFilters exception in case of disconnect in the middle of syncing filters
* more fixes
* Repurpose maxconnections test to be just a generic reconnection test
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* DLC P2P WIP
* P2PClient refactor (#10)
* Add to CI
* Remove unused
* Attempt to create DLCNode and Tests
* Full Tor support
* Get DLCNegotiationTest passing
* Config options, connect & send func
* Test for DLCNode
* Add createDLCNode to config
* Fix formatting
* Update DLC state after all other data is set
* Remove unneeded line
* Respond to some review
* 2021 07 26 dlc node code review (#13)
* WIP
* WIP2
* Rewrite tests not use Await.result()
* Skip Tor test on CI
* Cleanup threadpool leaks in tests
* Handle actor pattern matching better
* Respond to review
* Implement DLCNode.stop
* sock5 -> socks5
* Use Tcp.Unbind
* Respond to review
* Implement postStop
* Switch to unbind
Co-authored-by: rorp <rorp@users.noreply.github.com>
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* relay flag in version message made configurable
* modified test classes as well
* Update configuration.md
* reverted few files and cleaned up format
* fixed mistake
* Try to add block generate to address in fixture setup to get around the case where block headers & filter headers are synced but compact filters are not
* Push to github to force re-run of CI 2
* Add ChainApi.getBestFilter(), use it in sync to determine if filter headers & filters are in sync
* Push to github to force re-run of CI 3
* Simplify getBestFilter, add unit test
* Add more comments, clean up some code
* Give working code for example on issue
* Fix race condition where both NeutrinoNode.sync() and our data message handler would request the same set of compact filters causing database errors for uniquness violations
* Fix missing super.stop() to shutdown DbAppConfig db connection pool inside of all DbAppConfig subclasses that override stop()
* Fix import
* Remove implicit modifier to NodeUnitTest.getFreshConfig(), make sure we are shutting down chainAppConfig inside NodeUnitTest
* Remove chainTest subclasses implementing ActorSystem, make sure we do correct cleanup in ChainUnitTest
* Decouple CachedChainAppConfig & CachedAppConfig. When using only one sub module in the CachedAppConfig, teardown may fail as the other modules might not be started such as using chainAppConfig in chainTest, BitcoinSAppConfig.stop() requires us to call nodAppConfig.stop() first, which isn't started.
* Push to github to force re-run of CI 4
* Wallet Rebroadcast Logic
* Use wallet scheduler for address queue rather than it's own thread
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* Bump default
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* Refactor AsyncUtil to just use an execution context rather than an actor system
* Create async util project, use it in various projects that depend on it
* Give connection pool example config
* Get hikari logging working on postgres
* fix 2.12.x
* Bump metrics core to a supported version
* Add hikari-logging and hikari-logging-interval configuration options to turn off/on hikari and schedule how often logs appear
* Turn off hikari logging in the db-commons/reference.conf and add it to appserver/reference.conf
* default the logging to off in scala code
* Make sure connection pool is enabled in db-commons, add default database configuration to website
* Address code review, fix things that didn't need to be changed
* Add walletinfo rpc
* Move rootXpub to key manager obj
* Make configuration keys for getinfo the same as the ones in our configuration files
* Cleanup
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* Clean up fixture shutdown code a bit to try and see if this resolves issue on CI
* Rework fixtures so we aren't starting app configs in multiple places
* Fix scaladoc
* WIP
* Get neutrino node with wallet 'receive information about received payments' working again
* Fix compile
* Remove initial sync logic from test case
* Remove sync logic in NeutrinoNodeWithWallet test cases
* Improve logging and rename a few things
* WIP2
* WIP3
* Get NeutrinoNodeWithWallet tests working
* Implement WalletSync, which allows you to sync a wallet from a arbitrary data source
* Get all tests passing again
* Use spv.appConfig in DataMessageHandlerTest rather than caching the config
* Modify cleanup to hopefully get CI passing
* Fix postgres tests by cleaning the table during the destroy phase of the test fixture. This is needed because the same postgres database is shared between tests in the same test suite
* Revert logback-test.xml
* Get sqlite/postgres tests passing pt 2
* syncHeight -> syncDescriptorOpt()
* Add case for genesis block hash in WalletSync
* Fix SpvNodeWithWallet test case to actually test spv functionality
* Remove nodeCallbacks parameters, callbacks should be registered on nodeAppConfig
* Fail broadcasting transaction when disconnected
* Wait until disconnected
* Move test
* Move to separate file
* Attempt fix
* Make test never have a peer
* Improve reliablity
* Multi Wallet support
* Fix Oracle tests
* Fix some CI errors
* Increase KeyManagerAppConfig code coverage
* Add test for multiple seeds in seed folder
* Move things in reference.conf files
* Lower key manager coverage requirement
* Fix postgres test issues
* Create DbAppConfig
* Add some docs, fix hardcoded postgres db name
* Clarify db vs schema
* Add character types
* Fix chaindb in conf
* Always copy default seed
* Remove need for driverName config
* Introduce ChainHandlerCached which behaves like the old ChainHandler. Now Chainhandler.getBestBlockHeader() will read headers from the database
* Remove ChainHandler.blockchains field, now it's only available in ChainHandlerCached
* De-futurify ChainHandler.fromDatabase()
* Adjust logging
* Patch test case
* Use BlockHeaderDAO.chainTips when getting best header rather thean BlockHeaderDAO.getBlockchains(). Implement a helper method ChainHandler.toChainHandlerCached()
* Fix chain.md,wallet.md
* Make ChainHandler.getBestBlockHeader() consider time of header if chainwork is the same. Make test cases less strict on what header is the best header when both chainwork and time are the same on the eader
* Only execute callbacks on headers that are going to be created in the database, not all headers passed into ChainHandler.processHeadersWithChains()
* Turn up log level again
* Small optimizations, check if we have seen a header before before processing it in ChainHandler.processHeadersWithChains(). Fix FilterSyncMarker.toString(). Use ChainHandlerCached in Node
* Remove ChainHandlerCached in appServer, re-add it in Node.scala
* Cleanup a bunch of imports in test
* Use @nowarn annotation for testing things that are deprecated
* Fix base58 test
* Cleanup crypto-test
* Use scala collection compat dependency to get access to the @nowarn annotation on scala 2.12.x
* Rework more scala 2.13.x compile failures in chain-test and key-manager-test
* Optimize imports for entire project
* Fix nits
* Use absolute paths for resolving configurations
Make log level OFF again
WIP: Move resolution of config into AppConfig.start(), things are totally broken
WIP: Make AppConfig.config private
Get all tests passing again but using absolute path in configuration
* Cherry-pick onto master
* Fix BitcoinSAppConfig configuration keys to use the full path
* Apply code review
* Fix log, move invariant for requiredConfirmations back into the lazy val
* Fix DlcOracleAppConfig to use absolute path
* Introduce 'FilterSyncMarker' to ChainApi, make it clearier what exactly the (Int,DoubleSha256Digest) tuple is returned from ChainApi.nextBlockHeaderRange()
* Fix doc
* Add scaladoc to FilterSyncMarker
* Rebase onto master, fix conflicts to use FilterSyncMarker
* Pull over test cases, WIP
* Rework ChainHandler.nextBlockHeaderRange() to use hashes rather than heights to very integrity of header chain, add test case
* Modify test case to make sure we are generating the 'heaviest chain work' header as first in the tuple that is used in reorg test cases
* Refactor to helper method called 'findNextHeader()' to make things simpler
* Add scaladoc to ChainHandler.findNextHeader()
* WIP
* Fix bugs in ChainApi.nextBlockHeaderRange(), now return (startHeight,stopHash) rather than (stopHeight,stopHash)
* Get chain handler tests passing, remove genesis filter header/fitler from default chain project fixture, make it a specialized one to have genesis filter header/filter in the database. Use that specialized fixture in ChainHandlerTest for now
* FilterSync work without having the genesis filter and filter header inserted into the database
* fix bug in DataMessageHandler where we were using sendNextGetCompactFilterHeadersCommand rather than using peerMsgSender.sendGetCompactFilterHeadersMessage
* Refactor ChainHandler.findNextHeader() to centralize where chains are fetched
* Address ben's code review
* Implement getBestFilterHeader based on a number of block headers that can be passed in as a parameter. These headers can be used to indicate what your current best chain is
* Bring back compiler opts
* Fix compiler error for maxByOption as it isn't in the 2.12 std library
* Implement a context free best filter headers search. The basic strategy is to look at headers in the _future_ of our current best filter header
* Fix bug in sql query were we were doing max chainWork too early on block headers, we needed to filter out headers in our set and _then_ we get the max chain work
* Add more unit tests
