* Fix bug where we just ned to awaitAllSync() rather than attempt to sync()
* Use awaitSyncAndIBD, mv awaitSyncAndIBD to NodeTestUtil
* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD
* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD
* Remove explicit call to sync()
* Remove extra call to NodeUnitTest.syncNeutrinoNode() in test fixture
* Add NeutrinoNodeNotConnnectedWithBitcoinds
* Don't switch sync peer if its the same peer we are currently syncing from it
* Fix another test
* Remove another sync() call
* Add carve out with MisBehavingPeers for the case where the badPeer was our last peer
* Remove more .sync() calls
* Defensively try to sync everytime a peer is a initialized
* Turn logging OFF
* Fix DataMessagehandlerTest, remove calls to NeutrinoNode.sync() in test fixtures
* First attempt at implementing inactivityChecks()
* Move lastParsedMessage logic into PeerMessageSender
* Add bitcoin-s.node.inactivity-timeout config option
* implement unit test
* Fix CommonSettings
* scalafmt
* scalafmt
* Rename lastSuccessfulParsedMsg -> lastSuccessfulParsedMsgOpt
* make sure we disconnect in the case the stream terminates with an error
* Reduce log level
* Make PeerData.peerMessageSender not wrapped in Future
* Remove peerMessageSenderApi param
* Small cleanups
* Rename DataMessageHandlerState -> NodeState
* Cleanups in DataMessageHandler
* Refactor StreamDataMessageWrapper -> NodeStreamMessage
* More small fixes
* Make handleDataPayload() take PeerData rather than Peer
* Move peers into NodeState and remove as param in DataMessageHandler
* replacePeers whenever PeerManager.{onP2PClientDisconnected,onInitialization} is called
* rework ValidatingHeaders.toString()
* Empty commit to run CI
* WIP: Try to move byte streaming/parsing of p2p messages out of P2PClient
* WIP2: Work on killing the actor, replace it with a steram
* Get basic ability to send/receive version message working
* Transition PeerMessageReceiverState to Initializing inside of PeerMessagesender.connect()
* Refactor things out of PeerMessageSender.connect(), add some flow logs
* Get NeutrinoNodeTest be able to sync passing
* Fix some bugs, create ConnectionGraph helper class
* Use killswitch rather than Source.maybe to disconnect peer
* WIP: Debug
* Switch halfClose to false on Tcp.outgoingConnection() to not keep lingering connections
* Delete P2PClientActorTest
* Delete all P2PClient stuff
* Attempt implementing reconnection logic in PeerMessageSender
* remove supervisor
* Empty commit to re-run CI
* Small cleanups
* Implement sendResponseTimeout()
* Restore logback-test.xml
* Add callback to log error message on reconnect
* Increase queueSize/maxConcurrentOffers size
* WIP
* WIP2
* Rebase with P2PClientCallbacks
* scalafmt
* Rework stream materialization in PeerManager to have DataMessageHandler encapsulated in the stream
* WIP: Fix compile
* Get everything compiling, ignore Uncached tests for now
* Increase queue size
* WIP: Make queue re-usable based on PeerManager.{start()/stop()}
* Get things compiling after rebase
* Try to handle case where we have SendToPeer in queue with peer that has been disconnected
* Empty commit to re-run CI
* Add sleep to let version/verack handshake complete in P2PClientActorTest
* Empty commit to re-run CI
* Reduce usage of bitcoind in P2PClientActorTest from 3 bitcoinds -> 2 bitcoinds
* Add error message to PeerFinder.stop() so we know what peer was not getting removed
* Cleanup error message
* Fix scalafmt, add state to log message
* Fix bug PeerMessageReceiverState.stopReconnect() which didn't send DisconnectedPeer() to queue
* Empty commit to re-run CI
* Empty commit to re-run CI
* Reduce log level of onP2PClientDisconnected
* Empty commit to re-run CI
* Small cleanup
* scalafmt
* Get new reference to ChainHandler in more places node.syncFromNewPeer() is called
* Fix rebase
* Commit to run on CI
* Empty commit to run CI
* Empty commit to run CI
* Empty commit to re-run CI
* Empty commit to re-run CI
* Try to reproduce with logs on CI
* Empty commit to re-run CI
* WIP
* Rework onP2PClientDisconnected to return new DataMessagehandlerState
* Save comment about bug
* Add a helper method switchSyncToPeer to take into account the previous DataMessagehandlerState if we need to start a new sync because of disconnection
* Empty commit to re-run CI
* Empty commit to re-run CI
* Cleanup
* Fix case where we weren't sending getheaders to new peer when old peer was disconnected when in state DoneSyncing
* Revert logback-test.xml
* remove comment
* Try using syncHelper() rather than getHeaderSyncHelper() to make sure we sync filters as well if needed
* Re-add log
* Fix bug where we weren't starting to sync filter headers
* Tighten dmhState type to SyncDataMessageHandler on syncFilters(), clean up uncessary code
* Empty commit to re-run CI
* Empty commit to re-run CI
* Add PeerMessageSenderApi.gossipGetHeadersMessage(), use it in Node.sync()
* Rework invalid headers test case to not need to reach into internals of akka stream
* Rework re-query headers test case to not need to reach into internals of akka stream
* Rework switch peers test case to not need to reach into internals of akka stream
* Use peerManager.offer() rather than reaching into DataMessageHandler to send messages to stream
* use gossipGetHeadersMessage() after getting done with IBD to query all peers instead of just one
* Empty commit to re-run CI
* Empty commit to re-run CI
* Empty commit to re-run CI
* Get PeerMessageSenderApi using akka streams for outbound p2p messages
* Use offer method rather than accessing queue directly
* Fix flaky unit test
* Empty commit to re-run CI
* Move methods for requesting filterheaders/filters into PeerManager, now use akka stream for those outbound p2p message
* Move sendInventoryMessage to PeerManager
* Move sendGetHeadersMessage() methods to PeerManager
* WIP: move more methods to PeerMessageSenderApi
* WIP2
* initialize stream before calling PeerFinder.start() so oubound messages get processed
* Rebase
* Make queue buffer size dependent on maxConnectedPeers
* Change state to HeaderSync() if we are re-querying for block headers
* Empty commit to re-run CI
* Remove PeerMessageSender from handleDataPayload()
* Limit access to PeerManager.peerMessageSenders
* revert a few things
* Fix rebase issues
* Fix rebase
* Turn down logging
* Rebase
* remove guard that checks peer size before labeling as MisBehavingPeer
* Fix small bug where we needed to switch syncPeer and we weren't
* Empty commit to run CI
* Empty commit to re-run CI
* Fix test case where we weren't awaiting for Node.sync() to return Some(peer)
* Empty commit to re-run CI
* Fix another reference where we were calling Node.sync() too soon after Node.start()
* scalafmt
* Add another retryUntilSatisfied() on NeutrinoNode.sync()
* Remove awaitPeerWithServices()
* Empty commit to run CI
* Rework Node.sync() to return Future[Option[Peer]] rather than Future[Unit]. This returns the peer we are syncing with, if we could find one to sync with
* Turn logging OFF again
* Empty commit to re-run CI
* Use AsyncUtil.retryUntilSatisfied() when calling node.sync() after starting node to make sure we have a peer to sync from in a test case
* Await on re-started node not stale reference in NeutrinoNodeWithWalletTest
* Fix second reference
* Empty commit to re-run CI
* Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop()
* Rename onStop -> onDisconnect
* Add reconnect flag to P2PCallbacks.onDisconnect so we don't attempt to reconnect when not necessary
* Rename flag to forceReconnect, check getPeerConnectionCount in PeerManager.onP2PClientDisconnect to see if we have 0 connections. If we do reconnect
* Add PeerManager.isStarted flag, guard reconnection logic with this flag in onP2PClientDisconnected()
* Clear PeerFinder._peerData when PeerFinder.stop() is called
* WIP: Move disconnection logic into stream
* Rework ordering of PeerManager.stop() to shutdown queue after peers are removed
* Empty commit to re-run CI
* Await for getConnectionCount async in test case
* Try increasing queue size
* Bump queue size to 16
* Put initialization, initialization timeout logic in queue rather than callbacks
* Make messages that are sent to the queue rather than callbacks for various control messages
* Empty commit to re-run CI
* Empty commit to re-run CI
* Empty commit to re-run CI
* Remove P2PCallbacks all together
* Re-add PeerMessageReceiverTest
* WIP
* Reset PeerManager.dataMessageHandler to None on PeerManager.stop()
* WIP2
* Always set dataMessageHandler.peerData we process a new message in the stream
* remove PeerManager from PeerDatta
* Make PeerFinder mutable inside of PeerManager, pass the queue into PeerFinder to decouple PeerManager and PeerFinder
* Don't verify actor system shutdown for now
* Make ControlMessageHandler take PeerManager rather than Node as a param
* refactor PeerData to not take a reference to Node
* Move ControlMessageHandler out of {Node,NeutrinoNode}
* Fix ReConnectionTest
* Cleanup
* Revert logback-test.xml
* Fix connectioncount test case
* Get P2PClientTest passing consistently
* Empty commit to re-run CI
* Make PeerManager.peerDataMap private
* Remove PeerMessageSender as param to DataMessageHandler.addToStream()
* Remove PeerMessageSender parameter from DataMessageWrapper
* Add PeerMessageSenderApi
* Try adding a supervision strategy to the stream
* Empty commit to re-run CI
* Adjust log level down to try and get a better idea of whats happening on IC
* Add commandName to exception when we cannot find peerMessageSender in stream
* Try decreasing queue size to reduce async processing between being stream processing and actor receiving external p2p message
* Empty commit to re-run CI
* Increase max concurrent offers to nodeAppConfig.maxConnectedPeers
* Revert logging
* WIP: Add new states for FilterHeaderSync, FilterSync, DoneSyncing
* Get DataMessageHandler compiling
* Get src compiling
* Get tests compiling
* Remove syncPeer from DoneSyncing
* WIP: Get first NeutrinoNodeTest working
* Get first NeutrinoNodeTest working for basic sync
* Fix small bug
* Fix bug where we transition to HeaderSync when we receive HeadersMessage with 0 headers. Also reduce conflicting states in DataMessageHandler.getHeaders()
* Add caveats for empty HeadersMessage
* WIP: Get NeutrinoNodeTest syncing compact filters during IBD passing consistently
* Remove generate block delay
* Empty commit to re-run CI
* Fix bug in onP2PClientStopped()
* Fix handling of DoneSyncing state when we receive headers messages
* Remove impossible case
* Remove node parameter to PeerMessageReceiver, pass the ControlMessageHandler and DataMessageHandler as parameters
* Remove node reference in DataMessageHandler, just pass PeerManager as parameter
* Fix pattern match to be exhaustive
* Move fetchCompactFilterHeaders, sendFirstGetCompactFilterHeadersCommand out of DataMessagehandler to decouple DataMessageHandler, PeerManager
* scalafmt
* WIP: Refactor PeerMessageReceiver methods into PeerMessageReceiverState
* Remove state helper methods from PeerMessageReceiver
* WIP: Remove PeerMessageReceiverState from PeerMessageReceiver
* Get things compiling and tests passing
* Remove currentPeerMessageHandlerReceiver
* Refactor PeerMessageReceiverTest to not use bitcoind
* Add logic to fetch block headers after compact filters are synced in IBD to avoid a stale tip
* Restore logging, remove println
* Add comment linking to issue
* Cleanup logging
* Only send getheaders message after filter sync if we are in IBD
* WIP
* Modify DataMessageHandler to use Set for filter batch, fix test case
* revert log level
* Remove duplicate filters test
* Re-add test to make sure we don't throw an exception when processing a filter we've seen before
* Empty commit to re-run CI
* Empty commit to re-run CI
* Empty commit to re-run CI
* Fix NeutrinoNodeTest
* Empty commit to re-run CI
* Empty commit to re-run CI
* Segregate P2PClientTest and P2PClientActorTest to make it easier to test the actor
* Fix cleanup code
* Clean up tables for postgres test case
* Fix BitcoinSAppConfigBitcoinFixtureStarted.afterAll()
* Empty commit
* Add forceNamedWallet parameter to for postgres test cases where we need multiple wallets. This prevents them from writing to the same postgres schema by making unique wallet names
* Cleanup:
* Empty commit
* Make PeerManager.finder private
* Make PeerManager.supervisor private
* Write unit test to see if default peer is added to PeerManager
* Use withNeutrinoNodeUnstarted fixture
* Get unit tests passing
* Add println to try and debug CI
* Add more println
* more println
* Adjust nodeTest / parallelExecution = false
* Try to make error logged to figure out why failure is happening on CI
* Adjust log level to ERROR
* Add catch to try and log failure messages
* Add log for sending messages to peers
* Try to increase threadpool size to see if we are deadlocking
* Add more logs
* Add another log to detect where failure is
* Try using Actor.tell() with explicit ActorRef.noSender
* Very detailed logging
* Remove duplicate method from rebase
* Revert things
* Revert build.sbt
* Empty commit
* Bump timeout
* Empty commit to run CI
* Revert more files
* Pull things over from 4950 that shouldn't cause issues to reduce size of 4950
* Reduce timeout for processing a batch of messages from 1000 seconds -> 60 seconds
* scalafmt
* Empty commit
* WIP
* WIP2
* Get all DataMessageHandlerTest passing
* Get NeutrinoNodeTest passing
* Fix NeutrinoNodeWithUncachedBitcoindTest unit test
* Use chainApi to detect if we are in IBD
* Empty commit
* Revert logging levels
* Break versioning on V19 of bitcoind rpc client in chain handler, use BitcoindRpcClient with V19BlockFilterRpc
* Make NodeTestWithCachedBitcoindNoP2pBlockFilters for test case with unsupported peers
* Fix docs
* Empty commit
* Empty commit
* Turn logging off again
* add header sync validation
* fix docs, minor fixes
* Refactor to us InvalidBlockHeader ChainException, also refactor recovery to private helper method
* changes from comments
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* Embedded Pg v1.0.1
* fix GitHub action evironment
* fix Windows CI tests
* test db-commons in different environments
* remove dbCommontTest from Windows and Mac Github actions
* cleanup
* configurable postgres startup wait
* Empty commit to run CI
* Bump default timeout to 60
* Empty commit
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* add P2PClientSupervisor
* changes from comments: made P2PClient Future
* empty commit to see if mac failures are consistent on ci
* changes from comments
* changes from comments
* fix node test shared fixtures bug
The cached bitcoind fixtures were used and stopped in UnsyncedNeutrinoNodeTest which causes an error if NeutrinoNodeTest is run at the same time on high performant systems, which is why it escaped CI. Merges NeutrinoNodeTest and UnsyncedNeutrinoNodeTest
* fix possible issues with PeerMessageReceiverTest
This reverts commit 55e7cafee6c6fd8515650cf65d96f9d92a2a8812.
* fix filter sync issue when wallet creation time indicates already synced
* move switch peer test to NeutrinoNodeWithUnachedBitcoindTest
* add support to find and switch peers
* fix compile on 2.12
* allow empty config peers in regtest
* fix test
* minor fixes
changes timeouts, fix issue with ipv6 dns seeds, initDisconnect when disconnected bug, dbPeers order fix
* fix: not removing peers on initialization timeout
* fix: query again when previous failed
* fix: wrong condition for deferred peers
* restore log levels
* clean up
* add PeerStack to allow trying peers based on priority values
* fix migrations
* changes from comments
* use StartStopAsync
* changes from comments
* fix switch if peer down test
* changes from comments
* Fix bug where we prune unused addreses on rescan. This doesn't let us discover funds for addresses sent that were not paid to until after the rescan was completed
* Add test case
* Fix neutrino node test case
* Some cleanup
* Get things compiling and tests passing
* Fix bug where we weren't using filter height correctly
* Fix docs
* Move creationTime into KeyManagerAppConfig
* Fix nodeTest/test test fixtures
* Refactor testkit tests to use same nodeCallbacks that BitcoinSServerMain uses for neutrino
* Add test case
* Add logs for callbacks
* Cleanup
* Add test case for when funds are spent when we are offline
* Turn off logging again
* Cleanup more logs and comments
* Remove download binary scripts from CI, try to make binary downloads contingent on tests being run in a project
* Cleanup docs to remove unecessary downloadBitcoind
* Fix case where we forget to correctly assign currentPeerMsgHandlerRecv when we disconnect
* Make handleNodeCommand() so we handle P2PClient.CloseCommand
* Enable DEBUG logging to see what is happening on CI
* Add PeerMessageReceiverTest
* Add InitializeDisconnectDone
* Bump timeout on condition
* Tor packaged for testkit
* Add CI row
* Small fixes
* More fixes
* Keep tor running
* Skip tor test for old version of bitcoind
* Respond to review
* Fix with master
* Bump tor start time to same as app config
* Use cached tor for DLCNodeTest
* Disable parallel execution for dlc node test with tor
* Fix DLCNodeTest for tor and clearnet
* Fix name of CI rows
* P2P reconnect logic
* some fixes
* more changes
* fix DuplicateFilters exception in case of disconnect in the middle of syncing filters
* more fixes
* Repurpose maxconnections test to be just a generic reconnection test
Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
* relay flag in version message made configurable
* modified test classes as well
* Update configuration.md
* reverted few files and cleaned up format
* fixed mistake
* enabled neutrino node to receive transactions as a callback
* added txid to log message
* changed txId to txIdBE
* unit test added for checking if we're getting transaction as a callback
* Refactor MultiWalletTest to actually shutdown the walletAppConfigs
* Remove 'implicit' modifier from getFreshWalletConfig. This is a problem as it can be used implicitly to bind resources (threads, files etc) that need to be cleaned up. Since it is implicitly passed, it is very hard to remember to clean up these resources
* Cleanup TrezorAddressTest
* Remove unecessary BitcoinSWalletTest.afterAll()
* Fix docs
* Removal of datadir race condition
* Cleanup nodeTest to shutdown the transitive chainAppConfig required by NodeAppConfig
* remove experimental tag on NeutrinoNodeTest
* Remove extra line
* Cleanup after BitcoindChainhandlerViaZmqTest
* Push to github to force re-run of CI 2
* Fix missing super.stop() to shutdown DbAppConfig db connection pool inside of all DbAppConfig subclasses that override stop()
* Fix import
* Remove implicit modifier to NodeUnitTest.getFreshConfig(), make sure we are shutting down chainAppConfig inside NodeUnitTest
* Remove chainTest subclasses implementing ActorSystem, make sure we do correct cleanup in ChainUnitTest
* Decouple CachedChainAppConfig & CachedAppConfig. When using only one sub module in the CachedAppConfig, teardown may fail as the other modules might not be started such as using chainAppConfig in chainTest, BitcoinSAppConfig.stop() requires us to call nodAppConfig.stop() first, which isn't started.
* Push to github to force re-run of CI 4
* Create CachedBitcoind, implement it in FundTransactionHandlingTest
* Add BaseWalletTest, extend it with BitcoinSWalletTest & BitcoinSWalletTestCachedBitcoind, add CachedBitcoinV19 and use it RescanHandlingTest
* Make ProcessBlockTest work with cached bitcoind
* Make trait for CachedBitcoindNewest for the newest version of bitcoind
* Make UTXOLifeCycleTest use cached bitcoind
* Add WalletBloom, WalletSyncTest to use cached bitcoinds
* Add WalletIntegrationTest
* Rework beforeAll() and afterAll() into the super trait like BaseWalletTest
* Add standlone BitcoindFixtures, use it in BitcoindBackendTest
* Use new BitcoindFixtures in BitcoindBlockPollingTest
* Introduce BaseNodeTest, start implementing the usage of cached bitcoinds in the nodeTest project
* Use cached bitcoind's with SpvNodeTest & SpvNodeWithWalletTest
* Fix bug on postgres with reusing database, upsert the genesis header rather than create it
* Get NeutrinoNode tests workign with cached bitcoinds
* Fix NeutrinoNodeWithWallet by destroying wallet state for Postgres
* Add teardown helper method for bitcoind
* Teardown chain project when using node fixtures since node is dependent upon the chain project.
* Turn off parallelExecution again
* Switch the parallelExecution flag to only be set on CI, so we can get better performance when running locally
* Start implementing BitcoindFixtures, use BitcoindFixturesCachedTriple on TestUtilRpcTest
* Fix compiler errors, begin implementing NodePair
* Refactor TestRpcUtilTest to use 2 bitcoinds rather than 2
* Reduce the number of bitcoinds that MultiWalletRpcTest needs from 3 -> 1
* Reduce number of bitcoinds used in WalletRpcTest from 3 -> 2
* Add some documentation
* Try to re-add parallelExecution
* Reduce the number of bitcoinds used in PsbtRpcTest from 3 -> 2
* Disable parallelExecution in Test again
* Make BitcoindV21RpcClientTest & BitcoindV20RpcClientTest reduce bitcoind usage from 2 -> 1
* Make BitcoindV19RpcClienttest reduce bitcoind usage from 2 -> 1
* Rework MempoolRpcTest to use fixtures, add BitcoindVersion to CachedBitcoindCollection
* Make sure clientAccumm has to be specified as a paramter now rather than filling in by default
* Begin parameterizing NodePair/NodeTriple to retain type information for the specific version of bitcoind that was used
* Don't implement version in super trait
* Fix docs
* Fix async issue in V21 test suite
* Append to vectors in CachedBitcoinCollection rather than replace
* Fix rebase issues
* Add scaladocs
* Fix BitcoindV18RpcClient address info test
* Implement fixtures in BitcoindV17RpcClientTest fixtures
* Cleanup v17 PsbtRpcTest
* Reduce bitcoind usage from 3 -> 1 in BitcoindV18RpcClientTest
* Remove abandon transaction test, this allows us to reduce the number of bitcoind's used in MempoolRpcTest from 3 -> 2
* Remove the requirement to inject BitcoinSAsyncFixtureTest, add it in the test traits explicitly to make things easier. Also add explicit afterAll() method to tear down both the CachedBitcoind & BitcoinSAsyncFixtureTest
* Fix missing Await.result() in BitcoindRpcTest.afterAll()
* Rework MultiWalletRpcTest to use a NodePair
* Rework BlockchainRpcTest to use fixtures
* Rework Client start()/stop() methods. Now use an AtomicBoolean to indicate when a user has requested a client to start/stop rather than sending pings to bitcoind that can fail because the conneciton pool has been shutdown in test cases
* Try my luck with turning on parallelExecution in CI again
* Revert parallelExecution, now testes do not run in parallel on CI
* Only turn off parallelExecution for bitcoindRpcTest
* Adjust build to only have bitcoindRpcTest NOT in run parallel on mac, reduce number of blocks used in BitcoindRpcTestUtil.createNodeSequence
* Run less tests in the rpc test suite as that takes the longest, move them over to node/wallet/dlc test suite on mac osx CI
* Don't run eclair tests in parallel either
* Remove CachedBitcoind from BitcoinSWalletTest
* Fix async bug in test case
* Push to github to force re-run of CI
* Push to github to force re-run of CI
* Push to github to force re-run of CI
* Refactor AsyncUtil to just use an execution context rather than an actor system
* Create testkit-core, start moving data structures over to testkit-core
* WIP, getting rid of the hard dependency on BitcoinSAsyncTest and akka
* Rework test traits to not be dependnent on akka, move BaseAsyncTest into testkitCore, add a test trait that just uses the default scala executionContext
* Get everything compiling
* Move logback-test.xml to testkit-core so it applies to coreTest,cryptoTest as well
* Cleanup rebase issues
* Fix Deps.scala
* Address ben's code review
* Segregate jvm settings and compiler settings
* Filter out -Xfatal-warning for scalajs for now since there is a ton of errors
* Move maintainer settings into CommonSettings.settings
* Refactor AsyncUtil to just use an execution context rather than an actor system
* Create async util project, use it in various projects that depend on it
