Commit graph

578 commits

Author SHA1 Message Date
Chris Stewart
e4fbfe73e6
2024 02 16 fix get filter sync marker from stop block header bug (#5401)
* Fix bug in getFilterSyncMarkerFromStopBlockHeader where we were previously querying for children rather than the block hashes at the given height for building the FilterSyncMarker for getcfilter request. Also introduce delay in sending getcfheader because of https://github.com/bitcoin/bitcoin/issues/27085

* Only fallback to reorg scenario if we find no headers at height + 1

* Remove the best filter's block header from the candidate headers in reorg scenarios
2024-02-17 06:26:55 -06:00
Chris Stewart
1685c84eea
2024 01 31 Refactor BitcoindRpcTestUtil.{startSevers(),stopSevers()} to use Future.traverse() (#5382)
* Refactor BitcoindRpcTestUtil.startServers() to use Future.traverse()

* Refactor BitcoindRpcTestUtil.stopServers() to use Future.traverse()
2024-01-31 15:20:36 -06:00
Chris Stewart
018631dad4
Create NodeTestUtil.awaitConnectionCount() (#5378)
Refactor ReConnectionTest to use NodeTestUtil.awaitConnectionCount()

Refactor more of codebase to use NodeTestUtil.awaitConnectionCount()

Refactor PeerManagerTest to use NodeTestUtil.awaitConnectionCount()

Refactor more nodeTest to use NodeTestUtil.awaitConnectionCount()

use bitcoinds.size for expectedConnectionCount
2024-01-31 07:39:20 -06:00
Chris Stewart
76e468c5c4
2024 01 28 Refactor PeerFinder.start() to avoid initializing connections (#5376)
* Conslidate PeerFinder.start() logic into peerConnectionScheduler, add bitcoins-s.node.try-peers-start-delay config settig to indicate how long until we start attempting to connect to peers in PeerFinder

* Cleanup

* Revert logback-test.xml

* Add documentation

* Empty commit to run CI

* Rework NodeAppConfig.peers to return Vector[Peer] rather than Vector[String]

* Add higher priority to paramPeers
2024-01-29 09:46:31 -06:00
Chris Stewart
3a927edee1
Fix bug when requesting FilterSyncMarker for filter headers (#5344)
* Fix bug when requesting FilterSyncMarker for filter headers outside of our in memory blockchain's range

* Rework buildNHeaders to be faster, move it to ChainUnitTest
2024-01-06 09:59:26 -06:00
Chris Stewart
b39736fb8d
Rework NodeTestUtil to use a specific bestBlockHash (#5332)
* Rework NodeTestUtil to use a specific bestBlockHash, this is useful in reorg situations

Use param in reorg test, modify scaladoc

WIP

* Cleanup

* Try to fix usage of stopHashBE

* Cleanup

* WIP: Fix getCompactFilterStartHeight()

* Revert logback-test.xml

* Fix bug with isFiltersSynced() in reorg situations

* scalafmt, fix compile

* Fix another bug with isFiltersSynced()

* Fix compile
2023-12-31 17:38:12 -06:00
Chris Stewart
5e81ec5ed2
Add ChainApi.getBestChainTips() (#5338)
* Add ChainApi.getBestChainTips()

* Implement BitcoindRpcClient.getBestChainTips()

* Revert NodeTestUtil

* Use ChainApi.getBestChainTips() throughout codebase rather than blockHeaderDAO.getBestChainTips()
2023-12-31 16:54:40 -06:00
Chris Stewart
d983a1bac4
2023 12 24 Make filter headers / filters sync work with block header reorgs (#5333)
* Rework ChainApi.nextBlockHeaderBatchRange() to take stopHash parameter, get chainTest/test passing

* WIP: Refactor ChainApi.nextFilterHeaderBatchRange()

* Finish ChainApi.nextFilterHeaderBatchRange() refactor, get all tests passing except reorg related tests in nodeTest

* Get NeutrinoNodeTest reorg test case working

* Improve flaky test

* Cleanup

* Switch sync check to bitcoinds(1)

* Empty commit to run CI

* Implement batchSize tests in ChainHandlerTest

* Rework ChainHandlerTest to not assume we have the genesis filter header / filter inserted into the database already

* Cleanup println

* Fix bug with nextFilterHeaderBatchRange() wrt to startHeightOpt parameter

* Fix off by one bug with compact filter sync

* Add check for connectionCount

* Add longer timeout

* scalafmt
2023-12-28 10:43:01 -06:00
Chris Stewart
16fb5d2dad
2023 12 19 compiler opts (#5330)
* Enable more compiler options

* enable -Xlint:valpattern, fix bug with out Test / scalacOptions were set with incorrect source code compiler flags

* Enable -Xlint:eta-sam

* Fix cryptoTestJVM

* Fix cryptoTestJS/test

* Fix asyncUtilTest

* Fix asyncUtilTest
2023-12-20 05:42:01 -06:00
Chris Stewart
01537185bd
2023 12 13 mv connectioncount check fixtures (#5324)
* Move peer connection count checks into test fixtures

* Add connection count check in another test fixture

* scalafmt
2023-12-14 06:19:47 -06:00
Chris Stewart
b252c2d6a2
2023 10 16 Implement WalletCallbackStreamManager, DLCWalletCallbackStreamManager (#5263)
* WIP: Implement WalletCallbackStreamManager

* Implement stopping of WalletCallbackStreamManager in WalletAppConfig

* WIP: DLCWalletCallbackStreamManager

* Refactor DLCWalletCallbacks to be a trait

* Integrate DLCWalletCallbackStreamManager into WebsocketUtil

* Use  java.util.concurrent.atomic.AtomicBoolean rather than monix

* Remove logger param from dlc callbacks

* fix docs

* Call super.stop() in DLCappConfig

* Empty commit to run CI

* Try to debug

* Empty commit to run CI

* Revert things
2023-10-19 11:52:07 -05:00
Chris Stewart
531c909597
Use BitcoindRpcBackendUtil.buildBitcoindNodeApi() in wallet test fixtures, re-implement getConfirmationsForBlocks() to use akka streams to avoid max-open-requests limit inside of akka (#5259) 2023-10-12 18:02:06 -05:00
Chris Stewart
12e5dfc021
Fix chain fixtures to use ChainHandler rather than always using ChainHandlerCached (#5243)
* Fix chain fixtures to use CHainHandler rather than always using ChainHandlerCached

* Remove unecessary ChainAppConfig.start()
2023-09-25 11:13:59 -05:00
Scala Steward
d7037ede28
Update scala-library to 2.13.12 (#5235)
* Update scala-library to 2.13.12

* Fix compiler errors on scala 2.13.12

---------

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2023-09-12 10:45:17 -05:00
Chris Stewart
b13e0565af
Remove PeerMessageReceiver (#5182)
* Remove PeerMessageReceiver

* Dont use QueueOfferResult, use akka.Done
2023-08-06 13:10:11 -05:00
Chris Stewart
c238191209
2023 08 04 rm peermsgrecvstate (#5177)
* WIP: Remove PeerMessageReceiverState from PeerMessageReceiver

* Get tests passing
2023-08-06 11:57:04 -05:00
Chris Stewart
2032b16620
Fix bug where we just need to awaitSyncAndIBD() rather than attempt to sync() (#5158)
* Fix bug where we just ned to awaitAllSync() rather than attempt to sync()

* Use awaitSyncAndIBD, mv awaitSyncAndIBD to NodeTestUtil

* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD

* Replace more NodeUnitTest.syncNeutrinoNode() with NodeTestUtil.awaitSyncAndIBD
2023-07-21 17:08:01 -05:00
Chris Stewart
1ccc6a9230
2023 07 19 cleanup nodetest (#5156)
* Remove explicit call to sync()

* Remove extra call to NodeUnitTest.syncNeutrinoNode() in test fixture

* Add NeutrinoNodeNotConnnectedWithBitcoinds

* Don't switch sync peer if its the same peer we are currently syncing from it

* Fix another test

* Remove another sync() call

* Add carve out with MisBehavingPeers for the case where the badPeer was our last peer

* Remove more .sync() calls
2023-07-21 09:18:48 -05:00
Chris Stewart
e08469901b
Make PeerMessageSender.reconnect() return a Future that is completed when the connection is established (#5155)
* Make PeerMessageSender.reconnect() return a Future that is completed when the connection is established

* Replace hard require() with AsyncUtil.retryUntilSatisfiedF()
2023-07-19 13:10:15 -05:00
Chris Stewart
44190a535c
Defensively try to sync everytime a peer is a initialized (#5154)
* Defensively try to sync everytime a peer is a initialized

* Turn logging OFF

* Fix DataMessagehandlerTest, remove calls to NeutrinoNode.sync() in test fixtures
2023-07-19 09:43:23 -05:00
Chris Stewart
e66c078863
First attempt at implementing inactivityChecks() (#5144)
* First attempt at implementing inactivityChecks()

* Move lastParsedMessage logic into PeerMessageSender

* Add bitcoin-s.node.inactivity-timeout config option

* implement unit test

* Fix CommonSettings

* scalafmt

* scalafmt

* Rename lastSuccessfulParsedMsg -> lastSuccessfulParsedMsgOpt

* make sure we disconnect in the case the stream terminates with an error

* Reduce log level
2023-07-16 16:03:16 -05:00
Chris Stewart
b1e6488bb7
Move Socks5ProxyParams, Credentials into core (#5138) 2023-07-08 13:57:44 -05:00
Chris Stewart
1d82ed04a7
2023 06 18 implement tcp connection using akka streams (#5111)
* WIP: Try to move byte streaming/parsing of p2p messages out of P2PClient

* WIP2: Work on killing the actor, replace it with a steram

* Get basic ability to send/receive version message working

* Transition PeerMessageReceiverState to Initializing inside of PeerMessagesender.connect()

* Refactor things out of PeerMessageSender.connect(), add some flow logs

* Get NeutrinoNodeTest be able to sync passing

* Fix some bugs, create ConnectionGraph helper class

* Use killswitch rather than Source.maybe to disconnect peer

* WIP: Debug

* Switch halfClose to false on Tcp.outgoingConnection() to not keep lingering connections

* Delete P2PClientActorTest

* Delete all P2PClient stuff

* Attempt implementing reconnection logic in PeerMessageSender

* remove supervisor

* Empty commit to re-run CI

* Small cleanups

* Implement sendResponseTimeout()

* Restore logback-test.xml

* Add callback to log error message on reconnect

* Increase queueSize/maxConcurrentOffers size
2023-06-24 07:53:44 -05:00
Chris Stewart
5ae4993bed
Move PeerMessageReceiverState into PeerMessageReceiver (#5110)
* Move PeerMessageReceiverState into PeerMessageReceiver

* Delete PeerMessageReceiverTest
2023-06-19 15:04:35 -05:00
Chris Stewart
458a80d854
Destroy wallet config when tearing down node (#5105) 2023-06-15 13:39:37 -05:00
Chris Stewart
bed670fb6f
Remove chainApi parameter from NeutrinoNode (#5101)
* Remove chainApi parameter from NeutrinoNode

* Fix doc
2023-06-13 12:07:04 -05:00
Chris Stewart
ae8c97a4d1
Rework Node.sync() to return Future[Peer] rather than Future[Option[Peer]] (#5096) 2023-06-08 10:30:26 -05:00
Chris Stewart
ebe98959ed
Upgrade to scalac 2.13.11 (#5097)
* Upgrade to scalac 2.13.11

* Fix compile issues with scalac 2.13.11
2023-06-08 10:27:39 -05:00
Chris Stewart
295be36d63
Get PeerMessageSenderApi using akka streams for outbound p2p messages (#5069)
* Get PeerMessageSenderApi using akka streams for outbound p2p messages

* Use offer method rather than accessing queue directly

* Fix flaky unit test

* Empty commit to re-run CI

* Move methods for requesting filterheaders/filters into PeerManager, now use akka stream for those outbound p2p message

* Move sendInventoryMessage to PeerManager

* Move sendGetHeadersMessage() methods to PeerManager

* WIP: move more methods to PeerMessageSenderApi

* WIP2

* initialize stream before calling PeerFinder.start() so oubound messages get processed

* Rebase

* Make queue buffer size dependent on maxConnectedPeers

* Change state to HeaderSync() if we are re-querying for block headers

* Empty commit to re-run CI

* Remove PeerMessageSender from handleDataPayload()

* Limit access to PeerManager.peerMessageSenders

* revert a few things

* Fix rebase issues

* Fix rebase

* Turn down logging

* Rebase

* remove guard that checks peer size before labeling as MisBehavingPeer

* Fix small bug where we needed to switch syncPeer and we weren't

* Empty commit to run CI

* Empty commit to re-run CI

* Fix test case where we weren't awaiting for Node.sync() to return Some(peer)

* Empty commit to re-run CI

* Fix another reference where we were calling Node.sync() too soon after Node.start()

* scalafmt

* Add another retryUntilSatisfied() on NeutrinoNode.sync()
2023-06-08 08:47:24 -05:00
Chris Stewart
abeaaa05de
Remove awaitPeerWithServices() (#5093)
* Remove awaitPeerWithServices()

* Empty commit to run CI

* Rework Node.sync() to return Future[Option[Peer]] rather than Future[Unit]. This returns the peer we are syncing with, if we could find one to sync with

* Turn logging OFF again

* Empty commit to re-run CI

* Use AsyncUtil.retryUntilSatisfied() when calling node.sync() after starting node to make sure we have a peer to sync from in a test case

* Await on re-started node not stale reference in NeutrinoNodeWithWalletTest

* Fix second reference

* Empty commit to re-run CI
2023-06-07 04:47:39 -05:00
Chris Stewart
e74af8bf24
Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop() (#5089)
* Move P2PClientCallbacks.onStop() into disconnection logic rather than Actor.postStop()

* Rename onStop -> onDisconnect

* Add reconnect flag to P2PCallbacks.onDisconnect so we don't attempt to reconnect when not necessary

* Rename flag to forceReconnect, check getPeerConnectionCount in PeerManager.onP2PClientDisconnect to see if we have 0 connections. If we do reconnect

* Add PeerManager.isStarted flag, guard reconnection logic with this flag in onP2PClientDisconnected()

* Clear PeerFinder._peerData when PeerFinder.stop() is called

* WIP: Move disconnection logic into stream

* Rework ordering of PeerManager.stop() to shutdown queue after peers are removed

* Empty commit to re-run CI

* Await for getConnectionCount async in test case

* Try increasing queue size

* Bump queue size to 16

* Put initialization, initialization timeout logic in queue rather than callbacks

* Make messages that are sent to the queue rather than callbacks for various control messages

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI

* Remove P2PCallbacks all together

* Re-add PeerMessageReceiverTest
2023-06-05 10:39:02 -05:00
Chris Stewart
61e142a631
2023 05 29 peermanager peerfinder refactor (#5086)
* WIP

* Reset PeerManager.dataMessageHandler to None on PeerManager.stop()

* WIP2

* Always set dataMessageHandler.peerData we process a new message in the stream

* remove PeerManager from PeerDatta

* Make PeerFinder mutable inside of PeerManager, pass the queue into PeerFinder to decouple PeerManager and PeerFinder

* Don't verify actor system shutdown for now
2023-05-31 13:35:46 -05:00
Chris Stewart
9202e63c90
Create P2PClientCallbacks to encapsulate callbacks (#5084) 2023-05-30 07:40:52 -05:00
Chris Stewart
34df4ccbb1
Make ControlMessageHandler take PeerManager rather than Node as a param (#5081)
* Make ControlMessageHandler take PeerManager rather than Node as a param

* refactor PeerData to not take a reference to Node

* Move ControlMessageHandler out of {Node,NeutrinoNode}
2023-05-29 09:01:06 -05:00
Chris Stewart
777743989a
Remove PeerHandler as it isn't used in src (#5075)
* Remove PeerHandler as it isn't used in src

* Fix rebase
2023-05-22 11:11:03 -05:00
Chris Stewart
ce6d2212c1
2023 04 22 peermanager dmh refactor (#5057)
* WIP: Move DataMessageHandler into PeerManager

* Get things compiling

* Turn off logging
2023-04-24 08:14:02 -05:00
Chris Stewart
18482c7e44
2023 04 20 decouple node (#5049)
* Remove node parameter to PeerMessageReceiver, pass the ControlMessageHandler and DataMessageHandler as parameters

* Remove node reference in DataMessageHandler, just pass PeerManager as parameter

* Fix pattern match to be exhaustive

* Move fetchCompactFilterHeaders, sendFirstGetCompactFilterHeadersCommand out of DataMessagehandler to decouple DataMessageHandler, PeerManager

* scalafmt
2023-04-20 18:39:30 -05:00
Chris Stewart
447c6d03de
Implement getconnectioncount rpc (#5048)
* Implement getconnectioncount rpc

* Reduce waits in test, add documentation

* Fix docs

* Empty commit to re-run CI
2023-04-19 16:47:54 -05:00
Chris Stewart
51429a7d68
2023 04 17 refactor peer message receiver (#5045)
* WIP: Refactor PeerMessageReceiver methods into PeerMessageReceiverState

* Remove state helper methods from PeerMessageReceiver

* WIP: Remove PeerMessageReceiverState from PeerMessageReceiver

* Get things compiling and tests passing

* Remove currentPeerMessageHandlerReceiver

* Refactor PeerMessageReceiverTest to not use bitcoind
2023-04-18 10:25:24 -05:00
benthecarman
08757d536f
Lnd 0.16.0 rpc (#5005)
* Lnd 0.16.0 rpc

* Add 2 second delay for after channels are open
2023-03-30 14:43:26 -05:00
Chris Stewart
dc0e3645d0
2023 03 24 is tip stale (#5024)
* WIP

* Fix test

* Add more tests for stale header

* remove uncessary comments
2023-03-26 08:20:27 -05:00
Chris Stewart
d53ed7d487
Bump clightning version to v23.02.2 (#5025)
* Bump version to v23.02.2

* Fix hash

* Reduce timeout on clightning startup, its absurdly long causing slow test iteration

* Try to specify the OS version clightning tests run on to ubuntu-20.04

* Rework downloadClightning to be conscious of the ubuntu OS version

* Empty commit to re-run CI

* Empty commit to re-run CI

* Empty commit to re-run CI
2023-03-26 06:26:46 -05:00
Chris Stewart
3db663a387
Remove v20 bitcoind support (#4992)
* Remove v20 bitcoind support

* Remove v20 bitcoind from downloadBitcoind task

* Remove duplicate test
2023-02-19 13:28:46 -06:00
Chris Stewart
af349d2179
Remove support for v19 of bitcoind (#4991)
* Remove support for v19 of bitcoind

* Remove downloading v19 bitcoind inside of bitcoind-rpc.sbt

* Clean up docs

* Move unit tests in BitcoindV19RpcClientTest into other test files so we don't remove them, these rpcs are supported in future versions of bitcoind
2023-02-16 10:27:56 -06:00
Chris Stewart
b329c3670e
Add rpc for syncwithvalidationinterfacequeue, use it in NodeUnitTest.syncNeutrinoNode (#4983) 2023-02-11 14:32:59 -06:00
Chris Stewart
c7febc1ef0
Segregate P2PClientTest and P2PClientActorTest (#4974)
* Segregate P2PClientTest and P2PClientActorTest to make it easier to test the actor

* Fix cleanup code

* Clean up tables for postgres test case

* Fix BitcoinSAppConfigBitcoinFixtureStarted.afterAll()

* Empty commit

* Add forceNamedWallet parameter to for postgres test cases where we need multiple wallets. This prevents them from writing to the same postgres schema by making unique wallet names

* Cleanup:

* Empty commit
2023-02-06 15:07:18 -06:00
Chris Stewart
7e346e58fe
Refactor NodeTestUtil.awaitBestHash() to take a reference to bitcoind (#4973)
* Refactor NodeTestUtil.awaitBestHash() to take a reference to bitcoind

* Empty commit
2023-02-05 08:59:09 -06:00
Chris Stewart
1391c9497e
Segregate handling of exceptions in fixtures (#4971)
* Segregate handling of exceptions being thrown in test cases vs fixture setup code

* Add log message
2023-02-04 11:56:23 -06:00
Chris Stewart
76fc3cdc41
Refactor to not throw an exception is node is already syncing (#4970)
* Refactor to not throw an exception is node is already syncing

* Refactor method name
2023-02-03 12:57:19 -06:00
Chris Stewart
c5983730bb
2023 01 10 refactor peermanager (#4950)
* Make PeerManager.finder private

* Make PeerManager.supervisor private

* Write unit test to see if default peer is added to PeerManager

* Use withNeutrinoNodeUnstarted fixture

* Get unit tests passing

* Add println to try and debug CI

* Add more println

* more println

* Adjust nodeTest / parallelExecution = false

* Try to make error logged to figure out why failure is happening on CI

* Adjust log level to ERROR

* Add catch to try and log failure messages

* Add log for sending messages to peers

* Try to increase threadpool size to see if we are deadlocking

* Add more logs

* Add another log to detect where failure is

* Try using Actor.tell() with explicit ActorRef.noSender

* Very detailed logging

* Remove duplicate method from rebase

* Revert things

* Revert build.sbt

* Empty commit

* Bump timeout

* Empty commit to run CI

* Revert more files
2023-01-27 10:43:50 -06:00