Commit graph

186 commits

Author SHA1 Message Date
Chris Stewart
aa02683db9
Cleanup use of syncPeerOpt as a param in parts of PeerManager (#5209) 2023-08-26 12:37:27 -05:00
Chris Stewart
147f7782e5
2023 07 28 cleanup tests (#5172)
* simplify re-query invalid headers test case

* Cleanup test

* Cleanup another test

* Fix re-query invalid headers unti test

* fix unit test

* Empty commit to run CI

* Empty commit to re-run CI

* Empty commit to run CI
2023-08-01 11:26:51 -05:00
Chris Stewart
9b85838823
Implement logic to restart PeerManager in inactivity checks when we have 0 peers (#5171)
* Implement logic to restart PeerManager in inactivity checks when we have 0 peers

* Revert logback-test.xml

* cleanup
2023-08-01 10:07:40 -05:00
Chris Stewart
c2902c1146
Fix onBlockReceived unit test (#5166)
* Fix onBlockReceived unit test

* scalafmt
2023-07-25 16:20:39 -05:00
Chris Stewart
2d57ff6a3e
Refactor PeerManager methods to be connectPeer(), disconnectPeer() (#5163)
* Refactor PeerManager.{removePeer,waitingForDeletion} -> PeerManager.{disconnectPeer,waitingForDisconnection}

* PeerManager.addPeer() -> PeerManager.connectPeer()

* scalafmt
2023-07-25 11:59:58 -05:00
Chris Stewart
878dce7945
Use Await.result() in DataMessageHandlerTest to avoid test cases that never complete (#5159) 2023-07-24 07:57:54 -05:00
Chris Stewart
2032b16620
Fix bug where we just need to awaitSyncAndIBD() rather than attempt to sync() (#5158)
* Fix bug where we just ned to awaitAllSync() rather than attempt to sync()

* Use awaitSyncAndIBD, mv awaitSyncAndIBD to NodeTestUtil

* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD

* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD
2023-07-21 17:08:01 -05:00
Chris Stewart
1ccc6a9230
2023 07 19 cleanup nodetest (#5156)
* Remove explicit call to sync()

* Remove extra call to NodeUnitTest.syncNeutrinoNode() in test fixture

* Add NeutrinoNodeNotConnnectedWithBitcoinds

* Don't switch sync peer if its the same peer we are currently syncing from it

* Fix another test

* Remove another sync() call

* Add carve out with MisBehavingPeers for the case where the badPeer was our last peer

* Remove more .sync() calls
2023-07-21 09:18:48 -05:00
Chris Stewart
44190a535c
Defensively try to sync everytime a peer is a initialized (#5154)
* Defensively try to sync everytime a peer is a initialized

* Turn logging OFF

* Fix DataMessagehandlerTest, remove calls to NeutrinoNode.sync() in test fixtures
2023-07-19 09:43:23 -05:00
Chris Stewart
e66c078863
First attempt at implementing inactivityChecks() (#5144)
* First attempt at implementing inactivityChecks()

* Move lastParsedMessage logic into PeerMessageSender

* Add bitcoin-s.node.inactivity-timeout config option

* implement unit test

* Fix CommonSettings

* scalafmt

* scalafmt

* Rename lastSuccessfulParsedMsg -> lastSuccessfulParsedMsgOpt

* make sure we disconnect in the case the stream terminates with an error

* Reduce log level
2023-07-16 16:03:16 -05:00
Chris Stewart
b1e6488bb7
Move Socks5ProxyParams, Credentials into core (#5138) 2023-07-08 13:57:44 -05:00
Chris Stewart
ee619051a3
2023 07 04 node refactors (#5130)
* Make PeerData.peerMessageSender not wrapped in Future

* Remove peerMessageSenderApi param

* Small cleanups

* Rename DataMessageHandlerState -> NodeState

* Cleanups in DataMessageHandler

* Refactor StreamDataMessageWrapper -> NodeStreamMessage

* More small fixes

* Make handleDataPayload() take PeerData rather than Peer

* Move peers into NodeState and remove as param in DataMessageHandler

* replacePeers whenever PeerManager.{onP2PClientDisconnected,onInitialization} is called

* rework ValidatingHeaders.toString()

* Empty commit to run CI
2023-07-07 08:11:39 -05:00
Chris Stewart
6befad2dd3
Remove offer(SendToPeer) from PeerManager queue (#5119)
* Remove offer(SendToPeer) from PeerManager queue

* scalafmt

* Remove comments
2023-06-28 07:58:34 -05:00
Chris Stewart
1d82ed04a7
2023 06 18 implement tcp connection using akka streams (#5111)
* WIP: Try to move byte streaming/parsing of p2p messages out of P2PClient

* WIP2: Work on killing the actor, replace it with a steram

* Get basic ability to send/receive version message working

* Transition PeerMessageReceiverState to Initializing inside of PeerMessagesender.connect()

* Refactor things out of PeerMessageSender.connect(), add some flow logs

* Get NeutrinoNodeTest be able to sync passing

* Fix some bugs, create ConnectionGraph helper class

* Use killswitch rather than Source.maybe to disconnect peer

* WIP: Debug

* Switch halfClose to false on Tcp.outgoingConnection() to not keep lingering connections

* Delete P2PClientActorTest

* Delete all P2PClient stuff

* Attempt implementing reconnection logic in PeerMessageSender

* remove supervisor

* Empty commit to re-run CI

* Small cleanups

* Implement sendResponseTimeout()

* Restore logback-test.xml

* Add callback to log error message on reconnect

* Increase queueSize/maxConcurrentOffers size
2023-06-24 07:53:44 -05:00
Chris Stewart
5ae4993bed
Move PeerMessageReceiverState into PeerMessageReceiver (#5110)
* Move PeerMessageReceiverState into PeerMessageReceiver

* Delete PeerMessageReceiverTest
2023-06-19 15:04:35 -05:00
Chris Stewart
30b06b1131
Simplify DataMessageHandlerTest so we don't have duplicate callback execution (#5108)
* Simplify DataMessageHandlerTest so we don't have duplicate callback execution

* Turn logs OFF again
2023-06-16 13:34:45 -05:00
Chris Stewart
c3eed1e92b
2023 05 05 Make DataMessageHandler be accummulated in our akka stream (#5098)
* WIP

* WIP2

* Rebase with P2PClientCallbacks

* scalafmt

* Rework stream materialization in PeerManager to have DataMessageHandler encapsulated in the stream

* WIP: Fix compile

* Get everything compiling, ignore Uncached tests for now

* Increase queue size

* WIP: Make queue re-usable based on PeerManager.{start()/stop()}

* Get things compiling after rebase

* Try to handle case where we have SendToPeer in queue with peer that has been disconnected

* Empty commit to re-run CI

* Add sleep to let version/verack handshake complete in P2PClientActorTest

* Empty commit to re-run CI

* Reduce usage of bitcoind in P2PClientActorTest from 3 bitcoinds -> 2 bitcoinds

* Add error message to PeerFinder.stop() so we know what peer was not getting removed

* Cleanup error message

* Fix scalafmt, add state to log message

* Fix bug PeerMessageReceiverState.stopReconnect() which didn't send DisconnectedPeer() to queue

* Empty commit to re-run CI

* Empty commit to re-run CI

* Reduce log level of onP2PClientDisconnected

* Empty commit to re-run CI

* Small cleanup

* scalafmt

* Get new reference to ChainHandler in more places node.syncFromNewPeer() is called

* Fix rebase

* Commit to run on CI

* Empty commit to run CI

* Empty commit to run CI

* Empty commit to re-run CI

* Empty commit to re-run CI

* Try to reproduce with logs on CI

* Empty commit to re-run CI

* WIP

* Rework onP2PClientDisconnected to return new DataMessagehandlerState

* Save comment about bug

* Add a helper method switchSyncToPeer to take into account the previous DataMessagehandlerState if we need to start a new sync because of disconnection

* Empty commit to re-run CI

* Empty commit to re-run CI

* Cleanup

* Fix case where we weren't sending getheaders to new peer when old peer was disconnected when in state DoneSyncing

* Revert logback-test.xml

* remove comment

* Try using syncHelper() rather than getHeaderSyncHelper() to make sure we sync filters as well if needed

* Re-add log

* Fix bug where we weren't starting to sync filter headers

* Tighten dmhState type to SyncDataMessageHandler on syncFilters(), clean up uncessary code

* Empty commit to re-run CI

* Empty commit to re-run CI
2023-06-16 08:08:11 -05:00
Chris Stewart
e3f8eb2cc6
Add PeerMessageSenderApi.gossipGetHeadersMessage(), use it in Node.sync() (#5100)
* Add PeerMessageSenderApi.gossipGetHeadersMessage(), use it in Node.sync()

* Rework invalid headers test case to not need to reach into internals of akka stream

* Rework re-query headers test case to not need to reach into internals of akka stream

* Rework switch peers test case to not need to reach into internals of akka stream

* Use peerManager.offer() rather than reaching into DataMessageHandler to send messages to stream

* use gossipGetHeadersMessage() after getting done with IBD to query all peers instead of just one

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI
2023-06-09 15:34:29 -05:00
Chris Stewart
ae8c97a4d1
Rework Node.sync() to return Future[Peer] rather than Future[Option[Peer]] (#5096) 2023-06-08 10:30:26 -05:00
Chris Stewart
295be36d63
Get PeerMessageSenderApi using akka streams for outbound p2p messages (#5069)
* Get PeerMessageSenderApi using akka streams for outbound p2p messages

* Use offer method rather than accessing queue directly

* Fix flaky unit test

* Empty commit to re-run CI

* Move methods for requesting filterheaders/filters into PeerManager, now use akka stream for those outbound p2p message

* Move sendInventoryMessage to PeerManager

* Move sendGetHeadersMessage() methods to PeerManager

* WIP: move more methods to PeerMessageSenderApi

* WIP2

* initialize stream before calling PeerFinder.start() so oubound messages get processed

* Rebase

* Make queue buffer size dependent on maxConnectedPeers

* Change state to HeaderSync() if we are re-querying for block headers

* Empty commit to re-run CI

* Remove PeerMessageSender from handleDataPayload()

* Limit access to PeerManager.peerMessageSenders

* revert a few things

* Fix rebase issues

* Fix rebase

* Turn down logging

* Rebase

* remove guard that checks peer size before labeling as MisBehavingPeer

* Fix small bug where we needed to switch syncPeer and we weren't

* Empty commit to run CI

* Empty commit to re-run CI

* Fix test case where we weren't awaiting for Node.sync() to return Some(peer)

* Empty commit to re-run CI

* Fix another reference where we were calling Node.sync() too soon after Node.start()

* scalafmt

* Add another retryUntilSatisfied() on NeutrinoNode.sync()
2023-06-08 08:47:24 -05:00
Chris Stewart
abeaaa05de
Remove awaitPeerWithServices() (#5093)
* Remove awaitPeerWithServices()

* Empty commit to run CI

* Rework Node.sync() to return Future[Option[Peer]] rather than Future[Unit]. This returns the peer we are syncing with, if we could find one to sync with

* Turn logging OFF again

* Empty commit to re-run CI

* Use AsyncUtil.retryUntilSatisfied() when calling node.sync() after starting node to make sure we have a peer to sync from in a test case

* Await on re-started node not stale reference in NeutrinoNodeWithWalletTest

* Fix second reference

* Empty commit to re-run CI
2023-06-07 04:47:39 -05:00
Chris Stewart
e74af8bf24
Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop() (#5089)
* Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop()

* Rename onStop -> onDisconnect

* Add reconnect flag to P2PCallbacks.onDisconnect so we don't attempt to reconnect when not necessary

* Rename flag to forceReconnect, check getPeerConnectionCount in PeerManager.onP2PClientDisconnect to see if we have 0 connections. If we do reconnect

* Add PeerManager.isStarted flag, guard reconnection logic with this flag in onP2PClientDisconnected()

* Clear PeerFinder._peerData when PeerFinder.stop() is called

* WIP: Move disconnection logic into stream

* Rework ordering of PeerManager.stop() to shutdown queue after peers are removed

* Empty commit to re-run CI

* Await for getConnectionCount async in test case

* Try increasing queue size

* Bump queue size to 16

* Put initialization, initialization timeout logic in queue rather than callbacks

* Make messages that are sent to the queue rather than callbacks for various control messages

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI

* Remove P2PCallbacks all together

* Re-add PeerMessageReceiverTest
2023-06-05 10:39:02 -05:00
Chris Stewart
61e142a631
2023 05 29 peermanager peerfinder refactor (#5086)
* WIP

* Reset PeerManager.dataMessageHandler to None on PeerManager.stop()

* WIP2

* Always set dataMessageHandler.peerData we process a new message in the stream

* remove PeerManager from PeerDatta

* Make PeerFinder mutable inside of PeerManager, pass the queue into PeerFinder to decouple PeerManager and PeerFinder

* Don't verify actor system shutdown for now
2023-05-31 13:35:46 -05:00
Chris Stewart
9202e63c90
Create P2PClientCallbacks to encapsulate callbacks (#5084) 2023-05-30 07:40:52 -05:00
Chris Stewart
34df4ccbb1
Make ControlMessageHandler take PeerManager rather than Node as a param (#5081)
* Make ControlMessageHandler take PeerManager rather than Node as a param

* refactor PeerData to not take a reference to Node

* Move ControlMessageHandler out of {Node,NeutrinoNode}
2023-05-29 09:01:06 -05:00
Chris Stewart
777743989a
Remove PeerHandler as it isn't used in src (#5075)
* Remove PeerHandler as it isn't used in src

* Fix rebase
2023-05-22 11:11:03 -05:00
Chris Stewart
8e4aa49aef
Fix ReConnectionTest part 2 (#5074)
* Fix ReConnectionTest

* Cleanup

* Revert logback-test.xml

* Fix connectioncount test case

* Get P2PClientTest passing consistently

* Empty commit to re-run CI
2023-05-22 08:11:57 -05:00
Chris Stewart
8dfb7d091f
2023 05 05 encapsulate peermanager peermsgsender (#5066)
* Make PeerManager.peerDataMap private

* Remove PeerMessageSender as param to DataMessageHandler.addToStream()

* Remove PeerMessageSender parameter from DataMessageWrapper

* Add PeerMessageSenderApi

* Try adding a supervision strategy to the stream

* Empty commit to re-run CI

* Adjust log level down to try and get a better idea of whats happening on IC

* Add commandName to exception when we cannot find peerMessageSender in stream

* Try decreasing queue size to reduce async processing between being stream processing and actor receiving external p2p message

* Empty commit to re-run CI

* Increase max concurrent offers to nodeAppConfig.maxConnectedPeers

* Revert logging
2023-05-08 15:17:36 -05:00
Chris Stewart
4c6090207a
Add MisbehavingPeer state (#5065)
* Add MisbehavingPeer state

* Remove PeerManager.syncFromNewPeer()
2023-05-05 16:11:25 -05:00
Chris Stewart
2fdd237e20
2023 04 26 dmh state refactor (#5062)
* WIP: Add new states for FilterHeaderSync, FilterSync, DoneSyncing

* Get DataMessageHandler compiling

* Get src compiling

* Get tests compiling

* Remove syncPeer from DoneSyncing

* WIP: Get first NeutrinoNodeTest working

* Get first NeutrinoNodeTest working for basic sync

* Fix small bug

* Fix bug where we transition to HeaderSync when we receive HeadersMessage with 0 headers. Also reduce conflicting states in DataMessageHandler.getHeaders()

* Add caveats for empty HeadersMessage

* WIP: Get NeutrinoNodeTest syncing compact filters during IBD passing consistently

* Remove generate block delay

* Empty commit to re-run CI

* Fix bug in onP2PClientStopped()

* Fix handling of DoneSyncing state when we receive headers messages

* Remove impossible case
2023-05-04 12:45:40 -05:00
Chris Stewart
1fc6edf825
Remove initialSyncDone Promise as it isn't used (#5058) 2023-04-25 06:45:15 -05:00
Chris Stewart
ce6d2212c1
2023 04 22 peermanager dmh refactor (#5057)
* WIP: Move DataMessageHandler into PeerManager

* Get things compiling

* Turn off logging
2023-04-24 08:14:02 -05:00
Chris Stewart
18482c7e44
2023 04 20 decouple node (#5049)
* Remove node parameter to PeerMessageReceiver, pass the ControlMessageHandler and DataMessageHandler as parameters

* Remove node reference in DataMessageHandler, just pass PeerManager as parameter

* Fix pattern match to be exhaustive

* Move fetchCompactFilterHeaders, sendFirstGetCompactFilterHeadersCommand out of DataMessagehandler to decouple DataMessageHandler, PeerManager

* scalafmt
2023-04-20 18:39:30 -05:00
Chris Stewart
5f9c89820f
Add wait before checking connection count test (#5050) 2023-04-20 17:20:24 -05:00
Chris Stewart
447c6d03de
Implement getconnectioncount rpc (#5048)
* Implement getconnectioncount rpc

* Reduce waits in test, add documentation

* Fix docs

* Empty commit to re-run CI
2023-04-19 16:47:54 -05:00
Chris Stewart
51429a7d68
2023 04 17 refactor peer message receiver (#5045)
* WIP: Refactor PeerMessageReceiver methods into PeerMessageReceiverState

* Remove state helper methods from PeerMessageReceiver

* WIP: Remove PeerMessageReceiverState from PeerMessageReceiver

* Get things compiling and tests passing

* Remove currentPeerMessageHandlerReceiver

* Refactor PeerMessageReceiverTest to not use bitcoind
2023-04-18 10:25:24 -05:00
Chris Stewart
f95360f8ba
Add logic to fetch block headers after compact filters are synced in IBD to avoid a stale tip (#5037)
* Add logic to fetch block headers after compact filters are synced in IBD to avoid a stale tip

* Restore logging, remove println

* Add comment linking to issue

* Cleanup logging

* Only send getheaders message after filter sync if we are in IBD
2023-04-08 16:12:03 -05:00
Chris Stewart
31e1bd79e9
Speed up DataMessageHandlerTest dramatically by using CachedBitcoind (#5038) 2023-04-07 11:12:06 -05:00
Chris Stewart
e791932f99
Refactor peerData -> peerDataMap, try to fix #4955 (#5030)
* Refactor peerData -> peerDataMap, try to fix #4955

* scalafmt

* Empty commit to re-run CI
2023-04-03 06:35:39 -05:00
Chris Stewart
df5472f263
Generate blocks in two rpc calls (#5011)
* Generate blocks in two rpc calls

* Empty commit to re-run CI

* Empty commit to re-run CI
2023-03-15 07:29:53 -05:00
Chris Stewart
875a67a73c
2023 03 03 rm DataMessageHandler filter params (#5009)
* Remove DataMessageHandler filter params, tests not passing

* Adjust logging levels

* WIP

* Get tests passing

* Cleanup

* Fix logging level

* Invert isSynced flag

* Rework BaseBlockchain.toString

* Fix BlockchainTest

* Fix syncing of compact filters when walletCreationTime is passed into dmh

* Cleanup
2023-03-14 11:45:21 -05:00
Chris Stewart
de0e892b3e
2023 02 27 Fix bug where duplicate filters caused node not to sync (#5008)
* WIP

* Modify DataMessageHandler to use Set for filter batch, fix test case

* revert log level

* Remove duplicate filters test

* Re-add test to make sure we don't throw an exception when processing a filter we've seen before

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI

* Fix NeutrinoNodeTest

* Empty commit to re-run CI

* Empty commit to re-run CI
2023-03-02 06:46:08 -06:00
Chris Stewart
58fbcf41e3
Bump timeout in P2PClientActorTest (#4985) 2023-02-12 17:20:21 -06:00
Chris Stewart
c7febc1ef0
Segregate P2PClientTest and P2PClientActorTest (#4974)
* Segregate P2PClientTest and P2PClientActorTest to make it easier to test the actor

* Fix cleanup code

* Clean up tables for postgres test case

* Fix BitcoinSAppConfigBitcoinFixtureStarted.afterAll()

* Empty commit

* Add forceNamedWallet parameter to for postgres test cases where we need multiple wallets. This prevents them from writing to the same postgres schema by making unique wallet names

* Cleanup:

* Empty commit
2023-02-06 15:07:18 -06:00
Chris Stewart
7e346e58fe
Refactor NodeTestUtil.awaitBestHash() to take a reference to bitcoind (#4973)
* Refactor NodeTestUtil.awaitBestHash() to take a reference to bitcoind

* Empty commit
2023-02-05 08:59:09 -06:00
Chris Stewart
1391c9497e
Segregate handling of exceptions in fixtures (#4971)
* Segregate handling of exceptions being thrown in test cases vs fixture setup code

* Add log message
2023-02-04 11:56:23 -06:00
Chris Stewart
c5983730bb
2023 01 10 refactor peermanager (#4950)
* Make PeerManager.finder private

* Make PeerManager.supervisor private

* Write unit test to see if default peer is added to PeerManager

* Use withNeutrinoNodeUnstarted fixture

* Get unit tests passing

* Add println to try and debug CI

* Add more println

* more println

* Adjust nodeTest / parallelExecution = false

* Try to make error logged to figure out why failure is happening on CI

* Adjust log level to ERROR

* Add catch to try and log failure messages

* Add log for sending messages to peers

* Try to increase threadpool size to see if we are deadlocking

* Add more logs

* Add another log to detect where failure is

* Try using Actor.tell() with explicit ActorRef.noSender

* Very detailed logging

* Remove duplicate method from rebase

* Revert things

* Revert build.sbt

* Empty commit

* Bump timeout

* Empty commit to run CI

* Revert more files
2023-01-27 10:43:50 -06:00
Chris Stewart
dd08dedb41
2023 01 24 small changes from pr 4950 (#4958)
* Pull things over from 4950 that shouldn't cause issues to reduce size of 4950

* Reduce timeout for processing a batch of messages from 1000 seconds -> 60 seconds

* scalafmt

* Empty commit
2023-01-25 08:31:57 -06:00
Chris Stewart
9646994a99
Encapsulate PeerData.client (#4944) 2023-01-09 11:56:38 -06:00
Chris Stewart
8f4ed4ac83
Refactor to use NodeTestUtil.awaitSync() (#4942) 2023-01-06 15:57:07 -06:00