Commit Graph

111 Commits

Author SHA1 Message Date
Chris Stewart
52dcf51e82
Automatically download binaries if they are used in test suite (#4005)
* Remove download binary scripts from CI, try to make binary downloads contingent on tests being run in a project

* Cleanup docs to remove unecessary downloadBitcoind
2022-01-23 17:43:30 -06:00
Shreyansh
a58ef1cd02
Storing peers in database (#3773)
* rebase

* fix

* fix

* remove unwanted diffs

* fix

* formatting

* changes from comments

* Revert "changes from comments"

This reverts commit eb8a795718dc590802a19c7ce4cce4501bd2de1f.

* changes from comments

* add PeerManager

* add sha3-256

* add NetworkUtilTest

* formatting
2022-01-09 07:21:59 -06:00
rorp
41b96c4c7e
Add rescan field to walletinfo response (#3933) 2021-12-23 14:30:26 -08:00
rorp
4b07629d56
Add getmediantimepast RPC call (#3921)
* Add `getmediantimepast` RPC call

* update docs

* respond to the PR comments
2021-12-22 16:15:30 -06:00
rorp
a6898defe2
Support for Bitcoin Core v22 (#3834)
* Support for Bitcoin Core v22

* respond to the PR comments
2021-11-19 13:58:27 -06:00
benthecarman
92ab9faa45
Decide which coin selection solution to use based on waste metric (#3646)
* Decide which coin selection solution to use based on waste metric

* Fix test

* Fix NeutrinoNodeWithWalletTest
2021-10-19 09:47:28 -05:00
Shreyansh
99107b61ea
Add PeerDAO (#3738)
* add PeerDAO

* update DbManagementTest expected migration count
2021-10-10 07:50:11 -05:00
Chris Stewart
db46e35172
Fix case where we forget to correctly assign currentPeerMsgHandlerRecv (#3662)
* Fix case where we forget to correctly assign currentPeerMsgHandlerRecv when we disconnect

* Make handleNodeCommand() so we handle P2PClient.CloseCommand

* Enable DEBUG logging to see what is happening on CI

* Add PeerMessageReceiverTest

* Add InitializeDisconnectDone

* Bump timeout on condition
2021-09-14 06:38:38 -05:00
Chris Stewart
b6ef27eb3e
Move block generation and synchronization of th wallet before clearing utxo/addresses (#3623) 2021-08-29 14:39:13 -05:00
rorp
4ef1e6bcb6
CI fixes (#3597)
* CI fixes

* fix workflow names

* cleanup unused code

* fix ServerRunTest

* change number attempts to start Node
2021-08-25 15:07:27 -05:00
benthecarman
495ab62104
Tor in Testkit (#3484)
* Tor packaged for testkit

* Add CI row

* Small fixes

* More fixes

* Keep tor running

* Skip tor test for old version of bitcoind

* Respond to review

* Fix with master

* Bump tor start time to same as app config

* Use cached tor for DLCNodeTest

* Disable parallel execution for dlc node test with tor

* Fix DLCNodeTest for tor and clearnet

* Fix name of CI rows
2021-08-23 13:06:14 -05:00
Chris Stewart
ac8bdb120c
Implement PeerMessageReceiverState.InitializedDisconnect. This allows us to distinguish between disconnections we initalized and connections the peer intialized. This is needed for determining whether we should reconnect or not (#3583) 2021-08-21 17:04:50 -05:00
Chris Stewart
20575bcd68
Fix missing afterAll in ReConnectionTest (#3584) 2021-08-20 15:52:42 -05:00
rorp
54ce2ebeb2
P2P reconnect logic (#3572)
* P2P reconnect logic

* some fixes

* more changes

* fix DuplicateFilters exception in case of  disconnect in the middle of syncing filters

* more fixes

* Repurpose maxconnections test to be just a generic reconnection test

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2021-08-20 05:51:51 -05:00
Shreyansh
1426c31483
Multi peer: Sync with random peer (#3454)
* ensure random peer used for sync supports compact filters

* test file

* fix

* refactor

* rerun ci

* rerun ci

* changes from comments

* fix

* fix tests

* changes from comments

* refactor

* remove unintended diffs
2021-07-31 07:54:31 -05:00
Shreyansh
85975b8b07
complete handshake with peers (#3446)
fix
2021-07-25 11:11:01 -05:00
Shruthii RG
e2c2798e60
Implemented the ability to fetch unconfirmed transactions (#3429)
* relay flag in version message made configurable

* modified test classes as well

* Update configuration.md

* reverted few files and cleaned up format

* fixed mistake

* enabled neutrino node to receive transactions as a callback

* added txid to log message

* changed txId to txIdBE

* unit test added for checking if we're getting transaction as a callback
2021-07-23 13:01:07 -05:00
Shreyansh
4be2b2109b
[WIP] Adding multi-peer support (#3401)
* establish conn with multiple peers

squash 1

squash 2

squash 3

* add tests

* change log msg level

* minor changes

* fix

* formatting

* fix docs

* commit to rerun ci

* changes from comments

* fix bug

* rerun ci
2021-07-21 15:18:45 -05:00
rorp
8b663d91b6
Tor support for P2P (#3311)
* Tor support for P2P

* fix stuck connections

* unit tests

* docs

* update docs

* respond to the PR comments

* fix docs
2021-07-13 14:27:24 -05:00
Chris Stewart
77cd94ac41 2021 04 27 wallet fixtures config (#2980)
* Refactor MultiWalletTest to actually shutdown the walletAppConfigs

* Remove 'implicit' modifier from getFreshWalletConfig. This is a problem as it can be used implicitly to bind resources (threads, files etc) that need to be cleaned up. Since it is implicitly passed, it is very hard to remember to clean up these resources

* Cleanup TrezorAddressTest

* Remove unecessary BitcoinSWalletTest.afterAll()

* Fix docs

* Removal of datadir race condition

* Cleanup nodeTest to shutdown the transitive chainAppConfig required by NodeAppConfig

* remove experimental tag on NeutrinoNodeTest

* Remove extra line

* Cleanup after BitcoindChainhandlerViaZmqTest

* Push to github to force re-run of CI 2
2021-04-29 06:30:58 -05:00
benthecarman
e3017fd17d Peer Message Receiver Refactor (#2938) 2021-04-27 15:54:44 -05:00
Chris Stewart
16538980e3 Fix missing super.stop() to shutdown DbAppConfig db connection pool (#2943)
* Fix missing super.stop() to shutdown DbAppConfig db connection pool inside of all DbAppConfig subclasses that override stop()

* Fix import

* Remove implicit modifier to NodeUnitTest.getFreshConfig(), make sure we are shutting down chainAppConfig inside NodeUnitTest

* Remove chainTest subclasses implementing ActorSystem, make sure we do correct cleanup in ChainUnitTest

* Decouple CachedChainAppConfig & CachedAppConfig. When using only one sub module in the CachedAppConfig, teardown may fail as the other modules might not be started such as using chainAppConfig in chainTest, BitcoinSAppConfig.stop() requires us to call nodAppConfig.stop() first, which isn't started.

* Push to github to force re-run of CI 4
2021-04-23 08:21:56 -05:00
Chris Stewart
2287c6ced9 Implement caching of bitcoind in the walletTest,nodeTest, and partially bitcoindRpcTest project (#2792)
* Create CachedBitcoind, implement it in FundTransactionHandlingTest

* Add BaseWalletTest, extend it with BitcoinSWalletTest & BitcoinSWalletTestCachedBitcoind, add CachedBitcoinV19 and use it RescanHandlingTest

* Make ProcessBlockTest work with cached bitcoind

* Make trait for CachedBitcoindNewest for the newest version of bitcoind

* Make UTXOLifeCycleTest use cached bitcoind

* Add WalletBloom, WalletSyncTest to use cached bitcoinds

* Add WalletIntegrationTest

* Rework beforeAll() and afterAll() into the super trait like BaseWalletTest

* Add standlone BitcoindFixtures, use it in BitcoindBackendTest

* Use new BitcoindFixtures in BitcoindBlockPollingTest

* Introduce BaseNodeTest, start implementing the usage of cached bitcoinds in the nodeTest project

* Use cached bitcoind's with SpvNodeTest & SpvNodeWithWalletTest

* Fix bug on postgres with reusing database, upsert the genesis header rather than create it

* Get NeutrinoNode tests workign with cached bitcoinds

* Fix NeutrinoNodeWithWallet by destroying wallet state for Postgres

* Add teardown helper method for bitcoind

* Teardown chain project when using node fixtures since node is dependent upon the chain project.

* Turn off parallelExecution again

* Switch the parallelExecution flag to only be set on CI, so we can get better performance when running locally

* Start implementing BitcoindFixtures, use BitcoindFixturesCachedTriple on TestUtilRpcTest

* Fix compiler errors, begin implementing NodePair

* Refactor TestRpcUtilTest to use 2 bitcoinds rather than 2

* Reduce the number of bitcoinds that MultiWalletRpcTest needs from 3 -> 1

* Reduce number of bitcoinds used in WalletRpcTest from 3 -> 2

* Add some documentation

* Try to re-add parallelExecution

* Reduce the number of bitcoinds used in PsbtRpcTest from 3 -> 2

* Disable parallelExecution in Test again

* Make BitcoindV21RpcClientTest & BitcoindV20RpcClientTest reduce bitcoind usage from 2 -> 1

* Make BitcoindV19RpcClienttest reduce bitcoind usage from 2 -> 1

* Rework MempoolRpcTest to use fixtures, add BitcoindVersion to CachedBitcoindCollection

* Make sure clientAccumm has to be specified as a paramter now rather than filling in by default

* Begin parameterizing NodePair/NodeTriple to retain type information for the specific version of bitcoind that was used

* Don't implement version in super trait

* Fix docs

* Fix async issue in V21 test suite

* Append to vectors in CachedBitcoinCollection rather than replace

* Fix rebase issues

* Add scaladocs

* Fix BitcoindV18RpcClient address info test

* Implement fixtures in BitcoindV17RpcClientTest fixtures

* Cleanup v17 PsbtRpcTest

* Reduce bitcoind usage from 3 -> 1 in BitcoindV18RpcClientTest

* Remove abandon transaction test, this allows us to reduce the number of bitcoind's used in MempoolRpcTest from 3 -> 2

* Remove the requirement to inject  BitcoinSAsyncFixtureTest, add it in the test traits explicitly to make things easier. Also add explicit afterAll() method to tear down both the CachedBitcoind & BitcoinSAsyncFixtureTest

* Fix missing Await.result() in BitcoindRpcTest.afterAll()

* Rework MultiWalletRpcTest to use a NodePair

* Rework BlockchainRpcTest to use fixtures

* Rework Client start()/stop() methods. Now use an AtomicBoolean to indicate when a user has requested a client to start/stop rather than sending pings to bitcoind that can fail because the conneciton pool has been shutdown in test cases

* Try my luck with turning on parallelExecution in CI again

* Revert parallelExecution, now testes do not run in parallel on CI

* Only turn off parallelExecution for bitcoindRpcTest

* Adjust build to only have bitcoindRpcTest NOT in run parallel on mac, reduce number of blocks used in BitcoindRpcTestUtil.createNodeSequence

* Run less tests in the rpc test suite as that takes the longest, move them over to node/wallet/dlc test suite on mac osx CI

* Don't run eclair tests in parallel either

* Remove CachedBitcoind from BitcoinSWalletTest

* Fix async bug in test case

* Push to github to force re-run of CI

* Push to github to force re-run of CI

* Push to github to force re-run of CI
2021-03-19 06:37:53 -05:00
Chris Stewart
1959495cec Add testkit-core module (#2726)
* Refactor AsyncUtil to just use an execution context rather than an actor system

* Create testkit-core, start moving data structures over to testkit-core

* WIP, getting rid of the hard dependency on BitcoinSAsyncTest and akka

* Rework test traits to not be dependnent on akka, move BaseAsyncTest into testkitCore, add a test trait that just uses the default scala executionContext

* Get everything compiling

* Move logback-test.xml to testkit-core so it applies to coreTest,cryptoTest as well

* Cleanup rebase issues

* Fix Deps.scala

* Address ben's code review

* Segregate jvm settings and compiler settings

* Filter out -Xfatal-warning for scalajs for now since there is a ton of errors

* Move maintainer settings into CommonSettings.settings
2021-03-02 07:05:21 -06:00
Chris Stewart
7a068ac036 2021 02 25 async utils (#2725)
* Refactor AsyncUtil to just use an execution context rather than an actor system

* Create async util project, use it in various projects that depend on it
2021-02-25 13:26:38 -06:00
Aris
85087b0f70 Refactoring chain (#2662) 2021-02-13 14:10:40 -06:00
Chris Stewart
8f958bf5d9 Update scalafmt (#2619) 2021-02-03 16:37:12 -06:00
rorp
05e21b1200 Bump Eclair version (#2405)
* Bump Eclair version

* cleanup

* Bump JVM version for CI

* Update docs

* Fix docs
2021-01-22 10:47:19 -06:00
benthecarman
fcc258c5b2 Add unit test to make sure DataMessageHandler exception doesn't stop node (#2536) 2021-01-18 16:18:23 -06:00
Chris Stewart
abc0a2b459 Refactor test case to be more idiomatic in hopes this kills CI failures (#2524) 2021-01-16 14:26:51 -06:00
benthecarman
d69c60b8b2 Bitcoind v0.21.0 support (#2414)
* Bitcoind v0.21.0rc3 support

* Use v0.21.0 instead of Experimental in node tests

* Bump to rc5

* Use official release

* Fix rebase issues

* Remove DoNotDiscover tag

* Fix count for NeutrinoNodeTest
2021-01-14 14:01:44 -06:00
Chris Stewart
2048cc0bbb Make ChainApi.processHeaders() return a failed future in the case we … (#2436)
* Rename NodeUnitTest.confg -> NodeUnitTest.getFreshConfig()

* Rename CachedBitcoinSAppConfig.config -> CachedBitcoinSAppConfig.cachedConfig

* Make CachedChainAppConfig extend CachedBitcoinSAppConfig

* Make ChainApi.processHeaders() return a failed future in the case we have no valid headers

* Run scalafmt

* Fix test case to check if promise is completed yet

* WIP: Get something working that isn't network specific

Start putting things back in place

Add comment

Revert logback file

Remove BitcoinSLogger object

* Fix unused import

* Get rid of annoying diff

* Fix spacing nit
2021-01-12 06:11:19 -06:00
Chris Stewart
51e9796b39 2021 01 11 issue 2493 (#2503)
* Rename NodeUnitTest.confg -> NodeUnitTest.getFreshConfig()

* Rename CachedBitcoinSAppConfig.config -> CachedBitcoinSAppConfig.cachedConfig

* Make CachedChainAppConfig extend CachedBitcoinSAppConfig

* use cached configs in both wallet/chain project as well taking ben's suggestion to fix those
2021-01-11 16:45:10 -06:00
Chris Stewart
98ace6f14e 2021 01 02 issue 2457 (#2461)
* WIP

* Get neutrino node with wallet 'receive information about received payments' working again

* Fix compile

* Remove initial sync logic from test case

* Remove sync logic in NeutrinoNodeWithWallet test cases

* Improve logging and rename a few things

* WIP2

* WIP3

* Get NeutrinoNodeWithWallet tests working

* Implement WalletSync, which allows you to sync a wallet from a arbitrary data source

* Get all tests passing again

* Use spv.appConfig in DataMessageHandlerTest rather than caching the config

* Modify cleanup to hopefully get CI passing

* Fix postgres tests by cleaning the table during the destroy phase of the test fixture. This is needed because the same postgres database is shared between tests in the same test suite

* Revert logback-test.xml

* Get sqlite/postgres tests passing pt 2

* syncHeight -> syncDescriptorOpt()

* Add case for genesis block hash in WalletSync

* Fix SpvNodeWithWallet test case to actually test spv functionality

* Remove nodeCallbacks parameters, callbacks should be registered on nodeAppConfig
2021-01-09 09:33:37 -06:00
benthecarman
46c73970ff Remove callbacks param from DataMessageHandler & PeerMessageReceiver (#2476) 2021-01-06 13:23:47 -06:00
Chris Stewart
4e1a6c69f0 2020 12 18 enable lint options (#2454)
* Enable -Xlint:adapted-args,nullary-unit on our for compile scope

* Add Xlint:infer-any compiler option

* Fix build so test compiler options are actually enabled

* Enable more scalac linting options

* Add '-Xlint:eta-sam'

* Add 'Xlint:inaccessible,Xlint:missing-interpolator'
2021-01-03 08:01:26 -06:00
Chris Stewart
d204056ad8 2020 12 19 enable test compileropts (#2400)
* Fix bug where test compiler opts were not enabled

* Fix all exsting compiler warnings in test project

* Add scalaCollectionCompat dependency to bitcoindRpcTest so we can use @nowarn annotation on scala 2.12
2020-12-20 07:09:07 -06:00
Ben Carman
8912fcbc82 Fail broadcasting transaction when disconnected (#2336)
* Fail broadcasting transaction when disconnected

* Wait until disconnected

* Move test

* Move to separate file

* Attempt fix

* Make test never have a peer

* Improve reliablity
2020-12-17 13:28:32 -06:00
Ben Carman
dd26e51594 Setup Github Actions (#2319) 2020-12-07 13:35:22 -06:00
Ben Carman
9291d6ec3a Fix P2PClient parsing unknown messages (#2315)
* Fix P2PClient parsing unknown messages

* Use Vector, calculate correct checksum
2020-12-06 13:49:30 -06:00
Ben Carman
197367dc33 Request witness blocks from peers (#2289) 2020-11-23 22:08:03 -06:00
Chris Stewart
66ec96b1f5 2020 11 13 issue 2258 (#2260)
* Introduce ChainHandlerCached which behaves like the old ChainHandler. Now Chainhandler.getBestBlockHeader() will read headers from the database

* Remove ChainHandler.blockchains field, now it's only available in ChainHandlerCached

* De-futurify ChainHandler.fromDatabase()

* Adjust logging

* Patch test case

* Use BlockHeaderDAO.chainTips when getting best header rather thean BlockHeaderDAO.getBlockchains(). Implement a helper method ChainHandler.toChainHandlerCached()

* Fix chain.md,wallet.md

* Make ChainHandler.getBestBlockHeader() consider time of header if chainwork is the same. Make test cases less strict on what header is the best header when both chainwork and time are the same on the eader

* Only execute callbacks on headers that are going to be created in the database, not all headers passed into ChainHandler.processHeadersWithChains()

* Turn up log level again

* Small optimizations, check if we have seen a header before before processing it in ChainHandler.processHeadersWithChains(). Fix FilterSyncMarker.toString(). Use ChainHandlerCached in Node

* Remove ChainHandlerCached in appServer, re-add it in Node.scala
2020-11-17 06:19:07 -06:00
Chris Stewart
9b09b04230 Upgrade to scalac 2.13.3 (#2115) 2020-10-04 07:42:44 -05:00
Ben Carman
a694ef16b7 Add ability to fully spend utxos (#2063)
* Let RawTxSigner dummy sign transactions

* Add ability to fully spend utxos

* Fix NeutrinoNodeWithWalletTest

* Fix test, simplify call
2020-09-27 08:16:26 -05:00
Ben Carman
1e3aee55c3 Merge pull request #2019
* Refactor logging to only use grizzled slf4j

* Fix docs

* test travis config change
2020-09-19 14:09:12 -05:00
Chris Stewart
2980e3fc01 Resolve issues with reorgs and syncing filters (#1969)
* Pull over test cases, WIP

* Rework ChainHandler.nextBlockHeaderRange() to use hashes rather than heights to very integrity of header chain, add test case

* Modify test case to make sure we are generating the 'heaviest chain work' header as first in the tuple that is used in reorg test cases

* Refactor to helper method called 'findNextHeader()' to make things simpler

* Add scaladoc to ChainHandler.findNextHeader()

* WIP

* Fix bugs in ChainApi.nextBlockHeaderRange(), now return (startHeight,stopHash) rather than (stopHeight,stopHash)

* Get chain handler tests passing, remove genesis filter header/fitler from default chain project fixture, make it a specialized one to have genesis filter header/filter in the database. Use that specialized fixture in ChainHandlerTest for now

* FilterSync work without having the genesis filter and filter header inserted into the database

* fix bug in DataMessageHandler where we were using sendNextGetCompactFilterHeadersCommand rather than using peerMsgSender.sendGetCompactFilterHeadersMessage

* Refactor ChainHandler.findNextHeader() to centralize where chains are fetched

* Address ben's code review
2020-09-11 12:41:25 -05:00
Chris Stewart
39ef35b9ee Rename parameter names from 'duration' -> 'interval' just like our ActorSystem.schedule() names the parameter (#2005) 2020-09-11 11:10:59 -05:00
rorp
c62be8b5f6 Update callbacks for LN support (#1938)
Update callbacks for LN support
2020-09-09 11:02:31 -07:00
Ben Carman
10425760c3 Simplify Transaction Broadcast (#1872)
* Simplify transaction broadcast

* Assert tx in db, add note on privacy leak
2020-08-25 07:09:09 -05:00
Ben Carman
304896e171 Move WalletApi to Core (#1890) 2020-08-25 07:05:20 -05:00