Commit graph

529 commits

Author SHA1 Message Date
Chris Stewart
3177ee405f
Move away from suredbits provided peers by default (#5390)
* Move away from suredbits provided peers by default

* Remove setting bitcoin-s.node.use-default-peers

* Don't allow peer to stay connected if it doesn't support compact filters

* Move disconnection logic into managePeerAfterInitialization()

* Remove reconnect() when we have no other peers, send getaddr message after we know we are keeping the peer connection

* Remove hasCf flag check, revert log

* Reduce inactivity-timeout to 5 minutes by default, shuffle peers in PeerFinder beforing pushing onto stack

* Add logic to disconnect AttemptToConnect peers after addr mesage is received

* Create PeerFinder.queryForPeerConnections(), call it directly from PeerManager on health checks rather than restarting PeerFinder

* scalafmt

* Move more logic into managePeerAfterInitalization()

* Don't set isPersistent for db peers by default

* Add call to queryForPeerConnections() inside of PeerManager.onDisconnect() to try to attempt to more peers when we have no more connections, fix MisBehavingPeer logic to not start sync, starting sync should happen in onDisconnect()

* Revert logback-test.xml

* Sort peers we are attempting to connect to by lastSeen parameter

* Refactor DoneSyncing.toHeaderSync() to return Option[HeaderSync] to represent the case where we don't have a peer to sync with

* scalafmt

* Remove duplicate PeerFinder.buildPeerData()
2024-02-20 12:12:57 -06:00
Chris Stewart
e4fbfe73e6
2024 02 16 fix get filter sync marker from stop block header bug (#5401)
* Fix bug in getFilterSyncMarkerFromStopBlockHeader where we were previously querying for children rather than the block hashes at the given height for building the FilterSyncMarker for getcfilter request. Also introduce delay in sending getcfheader because of https://github.com/bitcoin/bitcoin/issues/27085

* Only fallback to reorg scenario if we find no headers at height + 1

* Remove the best filter's block header from the candidate headers in reorg scenarios
2024-02-17 06:26:55 -06:00
Chris Stewart
1685c84eea
2024 01 31 Refactor BitcoindRpcTestUtil.{startSevers(),stopSevers()} to use Future.traverse() (#5382)
* Refactor BitcoindRpcTestUtil.startServers() to use Future.traverse()

* Refactor BitcoindRpcTestUtil.stopServers() to use Future.traverse()
2024-01-31 15:20:36 -06:00
Chris Stewart
018631dad4
Create NodeTestUtil.awaitConnectionCount() (#5378)
Refactor ReConnectionTest to use NodeTestUtil.awaitConnectionCount()

Refactor more of codebase to use NodeTestUtil.awaitConnectionCount()

Refactor PeerManagerTest to use NodeTestUtil.awaitConnectionCount()

Refactor more nodeTest to use NodeTestUtil.awaitConnectionCount()

use bitcoinds.size for expectedConnectionCount
2024-01-31 07:39:20 -06:00
Chris Stewart
76e468c5c4
2024 01 28 Refactor PeerFinder.start() to avoid initializing connections (#5376)
* Conslidate PeerFinder.start() logic into peerConnectionScheduler, add bitcoins-s.node.try-peers-start-delay config settig to indicate how long until we start attempting to connect to peers in PeerFinder

* Cleanup

* Revert logback-test.xml

* Add documentation

* Empty commit to run CI

* Rework NodeAppConfig.peers to return Vector[Peer] rather than Vector[String]

* Add higher priority to paramPeers
2024-01-29 09:46:31 -06:00
Chris Stewart
3a927edee1
Fix bug when requesting FilterSyncMarker for filter headers (#5344)
* Fix bug when requesting FilterSyncMarker for filter headers outside of our in memory blockchain's range

* Rework buildNHeaders to be faster, move it to ChainUnitTest
2024-01-06 09:59:26 -06:00
Chris Stewart
b39736fb8d
Rework NodeTestUtil to use a specific bestBlockHash (#5332)
* Rework NodeTestUtil to use a specific bestBlockHash, this is useful in reorg situations

Use param in reorg test, modify scaladoc

WIP

* Cleanup

* Try to fix usage of stopHashBE

* Cleanup

* WIP: Fix getCompactFilterStartHeight()

* Revert logback-test.xml

* Fix bug with isFiltersSynced() in reorg situations

* scalafmt, fix compile

* Fix another bug with isFiltersSynced()

* Fix compile
2023-12-31 17:38:12 -06:00
Chris Stewart
5e81ec5ed2
Add ChainApi.getBestChainTips() (#5338)
* Add ChainApi.getBestChainTips()

* Implement BitcoindRpcClient.getBestChainTips()

* Revert NodeTestUtil

* Use ChainApi.getBestChainTips() throughout codebase rather than blockHeaderDAO.getBestChainTips()
2023-12-31 16:54:40 -06:00
Chris Stewart
d983a1bac4
2023 12 24 Make filter headers / filters sync work with block header reorgs (#5333)
* Rework ChainApi.nextBlockHeaderBatchRange() to take stopHash parameter, get chainTest/test passing

* WIP: Refactor ChainApi.nextFilterHeaderBatchRange()

* Finish ChainApi.nextFilterHeaderBatchRange() refactor, get all tests passing except reorg related tests in nodeTest

* Get NeutrinoNodeTest reorg test case working

* Improve flaky test

* Cleanup

* Switch sync check to bitcoinds(1)

* Empty commit to run CI

* Implement batchSize tests in ChainHandlerTest

* Rework ChainHandlerTest to not assume we have the genesis filter header / filter inserted into the database already

* Cleanup println

* Fix bug with nextFilterHeaderBatchRange() wrt to startHeightOpt parameter

* Fix off by one bug with compact filter sync

* Add check for connectionCount

* Add longer timeout

* scalafmt
2023-12-28 10:43:01 -06:00
Chris Stewart
16fb5d2dad
2023 12 19 compiler opts (#5330)
* Enable more compiler options

* enable -Xlint:valpattern, fix bug with out Test / scalacOptions were set with incorrect source code compiler flags

* Enable -Xlint:eta-sam

* Fix cryptoTestJVM

* Fix cryptoTestJS/test

* Fix asyncUtilTest

* Fix asyncUtilTest
2023-12-20 05:42:01 -06:00
Chris Stewart
01537185bd
2023 12 13 mv connectioncount check fixtures (#5324)
* Move peer connection count checks into test fixtures

* Add connection count check in another test fixture

* scalafmt
2023-12-14 06:19:47 -06:00
Chris Stewart
b252c2d6a2
2023 10 16 Implement WalletCallbackStreamManager, DLCWalletCallbackStreamManager (#5263)
* WIP: Implement WalletCallbackStreamManager

* Implement stopping of WalletCallbackStreamManager in WalletAppConfig

* WIP: DLCWalletCallbackStreamManager

* Refactor DLCWalletCallbacks to be a trait

* Integrate DLCWalletCallbackStreamManager into WebsocketUtil

* Use  java.util.concurrent.atomic.AtomicBoolean rather than monix

* Remove logger param from dlc callbacks

* fix docs

* Call super.stop() in DLCappConfig

* Empty commit to run CI

* Try to debug

* Empty commit to run CI

* Revert things
2023-10-19 11:52:07 -05:00
Chris Stewart
531c909597
Use BitcoindRpcBackendUtil.buildBitcoindNodeApi() in wallet test fixtures, re-implement getConfirmationsForBlocks() to use akka streams to avoid max-open-requests limit inside of akka (#5259) 2023-10-12 18:02:06 -05:00
Chris Stewart
12e5dfc021
Fix chain fixtures to use ChainHandler rather than always using ChainHandlerCached (#5243)
* Fix chain fixtures to use CHainHandler rather than always using ChainHandlerCached

* Remove unecessary ChainAppConfig.start()
2023-09-25 11:13:59 -05:00
Scala Steward
d7037ede28
Update scala-library to 2.13.12 (#5235)
* Update scala-library to 2.13.12

* Fix compiler errors on scala 2.13.12

---------

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2023-09-12 10:45:17 -05:00
Chris Stewart
b13e0565af
Remove PeerMessageReceiver (#5182)
* Remove PeerMessageReceiver

* Dont use QueueOfferResult, use akka.Done
2023-08-06 13:10:11 -05:00
Chris Stewart
c238191209
2023 08 04 rm peermsgrecvstate (#5177)
* WIP: Remove PeerMessageReceiverState from PeerMessageReceiver

* Get tests passing
2023-08-06 11:57:04 -05:00
Chris Stewart
2032b16620
Fix bug where we just need to awaitSyncAndIBD() rather than attempt to sync() (#5158)
* Fix bug where we just ned to awaitAllSync() rather than attempt to sync()

* Use awaitSyncAndIBD, mv awaitSyncAndIBD to NodeTestUtil

* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD

* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD
2023-07-21 17:08:01 -05:00
Chris Stewart
1ccc6a9230
2023 07 19 cleanup nodetest (#5156)
* Remove explicit call to sync()

* Remove extra call to NodeUnitTest.syncNeutrinoNode() in test fixture

* Add NeutrinoNodeNotConnnectedWithBitcoinds

* Don't switch sync peer if its the same peer we are currently syncing from it

* Fix another test

* Remove another sync() call

* Add carve out with MisBehavingPeers for the case where the badPeer was our last peer

* Remove more .sync() calls
2023-07-21 09:18:48 -05:00
Chris Stewart
e08469901b
Make PeerMessageSender.reconnect() return a Future that is completed when the connection is established (#5155)
* Make PeerMessageSender.reconnect() return a Future that is completed when the connection is established

* Replace hard require() with AsyncUtil.retryUntilSatisfiedF()
2023-07-19 13:10:15 -05:00
Chris Stewart
44190a535c
Defensively try to sync everytime a peer is a initialized (#5154)
* Defensively try to sync everytime a peer is a initialized

* Turn logging OFF

* Fix DataMessagehandlerTest, remove calls to NeutrinoNode.sync() in test fixtures
2023-07-19 09:43:23 -05:00
Chris Stewart
e66c078863
First attempt at implementing inactivityChecks() (#5144)
* First attempt at implementing inactivityChecks()

* Move lastParsedMessage logic into PeerMessageSender

* Add bitcoin-s.node.inactivity-timeout config option

* implement unit test

* Fix CommonSettings

* scalafmt

* scalafmt

* Rename lastSuccessfulParsedMsg -> lastSuccessfulParsedMsgOpt

* make sure we disconnect in the case the stream terminates with an error

* Reduce log level
2023-07-16 16:03:16 -05:00
Chris Stewart
b1e6488bb7
Move Socks5ProxyParams, Credentials into core (#5138) 2023-07-08 13:57:44 -05:00
Chris Stewart
1d82ed04a7
2023 06 18 implement tcp connection using akka streams (#5111)
* WIP: Try to move byte streaming/parsing of p2p messages out of P2PClient

* WIP2: Work on killing the actor, replace it with a steram

* Get basic ability to send/receive version message working

* Transition PeerMessageReceiverState to Initializing inside of PeerMessagesender.connect()

* Refactor things out of PeerMessageSender.connect(), add some flow logs

* Get NeutrinoNodeTest be able to sync passing

* Fix some bugs, create ConnectionGraph helper class

* Use killswitch rather than Source.maybe to disconnect peer

* WIP: Debug

* Switch halfClose to false on Tcp.outgoingConnection() to not keep lingering connections

* Delete P2PClientActorTest

* Delete all P2PClient stuff

* Attempt implementing reconnection logic in PeerMessageSender

* remove supervisor

* Empty commit to re-run CI

* Small cleanups

* Implement sendResponseTimeout()

* Restore logback-test.xml

* Add callback to log error message on reconnect

* Increase queueSize/maxConcurrentOffers size
2023-06-24 07:53:44 -05:00
Chris Stewart
5ae4993bed
Move PeerMessageReceiverState into PeerMessageReceiver (#5110)
* Move PeerMessageReceiverState into PeerMessageReceiver

* Delete PeerMessageReceiverTest
2023-06-19 15:04:35 -05:00
Chris Stewart
458a80d854
Destroy wallet config when tearing down node (#5105) 2023-06-15 13:39:37 -05:00
Chris Stewart
bed670fb6f
Remove chainApi parameter from NeutrinoNode (#5101)
* Remove chainApi parameter from NeutrinoNode

* Fix doc
2023-06-13 12:07:04 -05:00
Chris Stewart
ae8c97a4d1
Rework Node.sync() to return Future[Peer] rather than Future[Option[Peer]] (#5096) 2023-06-08 10:30:26 -05:00
Chris Stewart
ebe98959ed
Upgrade to scalac 2.13.11 (#5097)
* Upgrade to scalac 2.13.11

* Fix compile issues with scalac 2.13.11
2023-06-08 10:27:39 -05:00
Chris Stewart
295be36d63
Get PeerMessageSenderApi using akka streams for outbound p2p messages (#5069)
* Get PeerMessageSenderApi using akka streams for outbound p2p messages

* Use offer method rather than accessing queue directly

* Fix flaky unit test

* Empty commit to re-run CI

* Move methods for requesting filterheaders/filters into PeerManager, now use akka stream for those outbound p2p message

* Move sendInventoryMessage to PeerManager

* Move sendGetHeadersMessage() methods to PeerManager

* WIP: move more methods to PeerMessageSenderApi

* WIP2

* initialize stream before calling PeerFinder.start() so oubound messages get processed

* Rebase

* Make queue buffer size dependent on maxConnectedPeers

* Change state to HeaderSync() if we are re-querying for block headers

* Empty commit to re-run CI

* Remove PeerMessageSender from handleDataPayload()

* Limit access to PeerManager.peerMessageSenders

* revert a few things

* Fix rebase issues

* Fix rebase

* Turn down logging

* Rebase

* remove guard that checks peer size before labeling as MisBehavingPeer

* Fix small bug where we needed to switch syncPeer and we weren't

* Empty commit to run CI

* Empty commit to re-run CI

* Fix test case where we weren't awaiting for Node.sync() to return Some(peer)

* Empty commit to re-run CI

* Fix another reference where we were calling Node.sync() too soon after Node.start()

* scalafmt

* Add another retryUntilSatisfied() on NeutrinoNode.sync()
2023-06-08 08:47:24 -05:00
Chris Stewart
abeaaa05de
Remove awaitPeerWithServices() (#5093)
* Remove awaitPeerWithServices()

* Empty commit to run CI

* Rework Node.sync() to return Future[Option[Peer]] rather than Future[Unit]. This returns the peer we are syncing with, if we could find one to sync with

* Turn logging OFF again

* Empty commit to re-run CI

* Use AsyncUtil.retryUntilSatisfied() when calling node.sync() after starting node to make sure we have a peer to sync from in a test case

* Await on re-started node not stale reference in NeutrinoNodeWithWalletTest

* Fix second reference

* Empty commit to re-run CI
2023-06-07 04:47:39 -05:00
Chris Stewart
e74af8bf24
Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop() (#5089)
* Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop()

* Rename onStop -> onDisconnect

* Add reconnect flag to P2PCallbacks.onDisconnect so we don't attempt to reconnect when not necessary

* Rename flag to forceReconnect, check getPeerConnectionCount in PeerManager.onP2PClientDisconnect to see if we have 0 connections. If we do reconnect

* Add PeerManager.isStarted flag, guard reconnection logic with this flag in onP2PClientDisconnected()

* Clear PeerFinder._peerData when PeerFinder.stop() is called

* WIP: Move disconnection logic into stream

* Rework ordering of PeerManager.stop() to shutdown queue after peers are removed

* Empty commit to re-run CI

* Await for getConnectionCount async in test case

* Try increasing queue size

* Bump queue size to 16

* Put initialization, initialization timeout logic in queue rather than callbacks

* Make messages that are sent to the queue rather than callbacks for various control messages

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI

* Remove P2PCallbacks all together

* Re-add PeerMessageReceiverTest
2023-06-05 10:39:02 -05:00
Chris Stewart
61e142a631
2023 05 29 peermanager peerfinder refactor (#5086)
* WIP

* Reset PeerManager.dataMessageHandler to None on PeerManager.stop()

* WIP2

* Always set dataMessageHandler.peerData we process a new message in the stream

* remove PeerManager from PeerDatta

* Make PeerFinder mutable inside of PeerManager, pass the queue into PeerFinder to decouple PeerManager and PeerFinder

* Don't verify actor system shutdown for now
2023-05-31 13:35:46 -05:00
Chris Stewart
9202e63c90
Create P2PClientCallbacks to encapsulate callbacks (#5084) 2023-05-30 07:40:52 -05:00
Chris Stewart
34df4ccbb1
Make ControlMessageHandler take PeerManager rather than Node as a param (#5081)
* Make ControlMessageHandler take PeerManager rather than Node as a param

* refactor PeerData to not take a reference to Node

* Move ControlMessageHandler out of {Node,NeutrinoNode}
2023-05-29 09:01:06 -05:00
Chris Stewart
777743989a
Remove PeerHandler as it isn't used in src (#5075)
* Remove PeerHandler as it isn't used in src

* Fix rebase
2023-05-22 11:11:03 -05:00
Chris Stewart
ce6d2212c1
2023 04 22 peermanager dmh refactor (#5057)
* WIP: Move DataMessageHandler into PeerManager

* Get things compiling

* Turn off logging
2023-04-24 08:14:02 -05:00
Chris Stewart
18482c7e44
2023 04 20 decouple node (#5049)
* Remove node parameter to PeerMessageReceiver, pass the ControlMessageHandler and DataMessageHandler as parameters

* Remove node reference in DataMessageHandler, just pass PeerManager as parameter

* Fix pattern match to be exhaustive

* Move fetchCompactFilterHeaders, sendFirstGetCompactFilterHeadersCommand out of DataMessagehandler to decouple DataMessageHandler, PeerManager

* scalafmt
2023-04-20 18:39:30 -05:00
Chris Stewart
447c6d03de
Implement getconnectioncount rpc (#5048)
* Implement getconnectioncount rpc

* Reduce waits in test, add documentation

* Fix docs

* Empty commit to re-run CI
2023-04-19 16:47:54 -05:00
Chris Stewart
51429a7d68
2023 04 17 refactor peer message receiver (#5045)
* WIP: Refactor PeerMessageReceiver methods into PeerMessageReceiverState

* Remove state helper methods from PeerMessageReceiver

* WIP: Remove PeerMessageReceiverState from PeerMessageReceiver

* Get things compiling and tests passing

* Remove currentPeerMessageHandlerReceiver

* Refactor PeerMessageReceiverTest to not use bitcoind
2023-04-18 10:25:24 -05:00
benthecarman
08757d536f
Lnd 0.16.0 rpc (#5005)
* Lnd 0.16.0 rpc

* Add 2 second delay for after channels are open
2023-03-30 14:43:26 -05:00
Chris Stewart
dc0e3645d0
2023 03 24 is tip stale (#5024)
* WIP

* Fix test

* Add more tests for stale header

* remove uncessary comments
2023-03-26 08:20:27 -05:00
Chris Stewart
d53ed7d487
Bump clightning version to v23.02.2 (#5025)
* Bump version to v23.02.2

* Fix hash

* Reduce timeout on clightning startup, its absurdly long causing slow test iteration

* Try to specify the OS version clightning tests run on to ubuntu-20.04

* Rework downloadClightning to be conscious of the ubuntu OS version

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI
2023-03-26 06:26:46 -05:00
Chris Stewart
3db663a387
Remove v20 bitcoind support (#4992)
* Remove v20 bitcoind support

* Remove v20 bitcoind from downloadBitcoind task

* Remove duplicate test
2023-02-19 13:28:46 -06:00
Chris Stewart
af349d2179
Remove support for v19 of bitcoind (#4991)
* Remove support for v19 of bitcoind

* Remove downloading v19 bitcoind inside of bitcoind-rpc.sbt

* Clean up docs

* Move unit tests in BitcoindV19RpcClientTest into other test files so we don't remove them, these rpcs are supported in future versions of bitcoind
2023-02-16 10:27:56 -06:00
Chris Stewart
b329c3670e
Add rpc for syncwithvalidationinterfacequeue, use it in NodeUnitTest.syncNeutrinoNode (#4983) 2023-02-11 14:32:59 -06:00
Chris Stewart
c7febc1ef0
Segregate P2PClientTest and P2PClientActorTest (#4974)
* Segregate P2PClientTest and P2PClientActorTest to make it easier to test the actor

* Fix cleanup code

* Clean up tables for postgres test case

* Fix BitcoinSAppConfigBitcoinFixtureStarted.afterAll()

* Empty commit

* Add forceNamedWallet parameter to for postgres test cases where we need multiple wallets. This prevents them from writing to the same postgres schema by making unique wallet names

* Cleanup:

* Empty commit
2023-02-06 15:07:18 -06:00
Chris Stewart
7e346e58fe
Refactor NodeTestUtil.awaitBestHash() to take a reference to bitcoind (#4973)
* Refactor NodeTestUtil.awaitBestHash() to take a reference to bitcoind

* Empty commit
2023-02-05 08:59:09 -06:00
Chris Stewart
1391c9497e
Segregate handling of exceptions in fixtures (#4971)
* Segregate handling of exceptions being thrown in test cases vs fixture setup code

* Add log message
2023-02-04 11:56:23 -06:00
Chris Stewart
76fc3cdc41
Refactor to not throw an exception is node is already syncing (#4970)
* Refactor to not throw an exception is node is already syncing

* Refactor method name
2023-02-03 12:57:19 -06:00