Commit Graph

137 Commits

Author SHA1 Message Date
Chris Stewart
7ba7f8b9ba
Try to add block generate to address in fixture setup to get around compact filter sync edge case (#3231)
* Try to add block generate to address in fixture setup to get around the case where block headers & filter headers are synced but compact filters are not

* Push to github to force re-run of CI 2

* Add ChainApi.getBestFilter(), use it in sync to determine if filter headers & filters are in sync

* Push to github to force re-run of CI 3

* Simplify getBestFilter, add unit test

* Add more comments, clean up some code
2021-06-08 12:47:28 -05:00
Chris Stewart
41e22b3cbc
2021 05 23 Sync race condition (#3129)
* Give working code for example on issue

* Fix race condition where both NeutrinoNode.sync() and our data message handler would request the same set of compact filters causing database errors for uniquness violations
2021-05-24 12:30:13 -05:00
Chris Stewart
a104787985
Fix Node.sync() bug that was caused by not starting filter header sync from the current best filter headers block hash (#3092) 2021-05-17 06:04:04 -05:00
benthecarman
e3017fd17d Peer Message Receiver Refactor (#2938) 2021-04-27 15:54:44 -05:00
Chris Stewart
16538980e3 Fix missing super.stop() to shutdown DbAppConfig db connection pool (#2943)
* Fix missing super.stop() to shutdown DbAppConfig db connection pool inside of all DbAppConfig subclasses that override stop()

* Fix import

* Remove implicit modifier to NodeUnitTest.getFreshConfig(), make sure we are shutting down chainAppConfig inside NodeUnitTest

* Remove chainTest subclasses implementing ActorSystem, make sure we do correct cleanup in ChainUnitTest

* Decouple CachedChainAppConfig & CachedAppConfig. When using only one sub module in the CachedAppConfig, teardown may fail as the other modules might not be started such as using chainAppConfig in chainTest, BitcoinSAppConfig.stop() requires us to call nodAppConfig.stop() first, which isn't started.

* Push to github to force re-run of CI 4
2021-04-23 08:21:56 -05:00
benthecarman
d0629486ab Wallet Rebroadcast Transactions thread (#2711)
* Wallet Rebroadcast Logic

* Use wallet scheduler for address queue rather than it's own thread

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>

* Bump default

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2021-04-07 08:13:43 -05:00
benthecarman
fa80f36d2f Get all of Core working with JS (#2826)
* core-test-js-json

* Add SigPointComputer cross compile
Co-authored-by: Nadav Kohen <nadavk25@gmail.com>

* Fix java.time for scalajs

* InetAddress for scalajs

* Fix linking errors in coreTestJS

* Easy fixes

* InetAddress Tests

* Update micropickle, fix exact functions

* Get BigSizeUIntTest working with upickle

* Fix exact funcs

* Update Base58Test with upickle

* Update ScriptSignatureTest with upickle

* Update TransactionTest with upickle

* Update BlockFilterTest with upickle

* Update ScriptInterpreterTest with upickle

* Update MnemonicCodeTest with upickle

* Add InetAddress unit tests

* Fix compile errors

* Get core tests running, not passing

* Sign with js

* Fix PSBTUnitTest

* Fix PBKDF2 for JS

* Use bcrypto's secp256k1 instead of ecdsa

* Fix 2.12.12 compile

* Core tests passing :)

* Test corejs
2021-03-31 15:04:30 -05:00
benthecarman
07514e2348 Remove logging from core (#2810)
* Remove logging from core

* Remove slf4j from testkit core

* Remove logback from coretest
2021-03-21 19:11:55 -05:00
Chris Stewart
7764828b3a Bump timeout on bind to avoid spurious ci failures hopefully (#2791)
* Bump timeout on bind to avoid spurious ci failures hopefully

* Actually pass the timeout

* Bump to 30 seconds
2021-03-11 12:24:28 -06:00
Chris Stewart
7a068ac036 2021 02 25 async utils (#2725)
* Refactor AsyncUtil to just use an execution context rather than an actor system

* Create async util project, use it in various projects that depend on it
2021-02-25 13:26:38 -06:00
Aris
85087b0f70 Refactoring chain (#2662) 2021-02-13 14:10:40 -06:00
Aris
f8694eb097 Fix/typos (#2633)
* Fix typos

* Collection minor fixes

Removes redundant .headOption on options
Replaces .exists(_ == ???) with .contains(???)
Replaces option == None with option.isEmpty
2021-02-07 15:00:25 -06:00
Chris Stewart
8f958bf5d9 Update scalafmt (#2619) 2021-02-03 16:37:12 -06:00
benthecarman
03ca6f617e Optimize filter sync and fetching filter heights (#2568) 2021-01-30 14:56:47 -06:00
benthecarman
ff6b85d195 Ignore block messages while syncing (#2587) 2021-01-29 09:28:57 -06:00
Chris Stewart
756bfc4427 2021 01 27 conectionpool (#2578)
* Give connection pool example config

* Get hikari logging working on postgres

* fix 2.12.x

* Bump metrics core to a supported version

* Add hikari-logging and hikari-logging-interval configuration options to turn off/on hikari and schedule how often logs appear

* Turn off hikari logging in the db-commons/reference.conf and add it to appserver/reference.conf

* default the logging to off in scala code

* Make sure connection pool is enabled in db-commons, add default database configuration to website

* Address code review, fix things that didn't need to be changed
2021-01-28 09:44:03 -06:00
benthecarman
29bde74cb8 Fix all DAOs to use safeDatabase (#2556) 2021-01-22 16:13:45 -06:00
benthecarman
e10e278aa4 Add walletinfo rpc (#2546)
* Add walletinfo rpc

* Move rootXpub to key manager obj

* Make configuration keys for getinfo the same as the ones in our configuration files

* Cleanup

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2021-01-22 13:55:21 -06:00
benthecarman
0d07d558c4 Optimize node.start() and fetching filter & filter header heights (#2554)
* Make node.start() more async

* Add missing filter indexes

* Move queries to vals
2021-01-22 10:38:32 -06:00
benthecarman
03e36d5fa2 Recover errors in DataMessageHandler (#2460) 2021-01-10 12:17:51 -06:00
Chris Stewart
68af9760b3 Clean up fixture shutdown code a bit to try and see if this resolves … (#2498)
* Clean up fixture shutdown code a bit to try and see if this resolves issue on CI

* Rework fixtures so we aren't starting app configs in multiple places

* Fix scaladoc
2021-01-09 17:07:26 -06:00
Chris Stewart
98ace6f14e 2021 01 02 issue 2457 (#2461)
* WIP

* Get neutrino node with wallet 'receive information about received payments' working again

* Fix compile

* Remove initial sync logic from test case

* Remove sync logic in NeutrinoNodeWithWallet test cases

* Improve logging and rename a few things

* WIP2

* WIP3

* Get NeutrinoNodeWithWallet tests working

* Implement WalletSync, which allows you to sync a wallet from a arbitrary data source

* Get all tests passing again

* Use spv.appConfig in DataMessageHandlerTest rather than caching the config

* Modify cleanup to hopefully get CI passing

* Fix postgres tests by cleaning the table during the destroy phase of the test fixture. This is needed because the same postgres database is shared between tests in the same test suite

* Revert logback-test.xml

* Get sqlite/postgres tests passing pt 2

* syncHeight -> syncDescriptorOpt()

* Add case for genesis block hash in WalletSync

* Fix SpvNodeWithWallet test case to actually test spv functionality

* Remove nodeCallbacks parameters, callbacks should be registered on nodeAppConfig
2021-01-09 09:33:37 -06:00
benthecarman
46c73970ff Remove callbacks param from DataMessageHandler & PeerMessageReceiver (#2476) 2021-01-06 13:23:47 -06:00
benthecarman
ab55f5163f Request filters after processing (#2463) 2021-01-04 06:37:48 -06:00
Chris Stewart
4e1a6c69f0 2020 12 18 enable lint options (#2454)
* Enable -Xlint:adapted-args,nullary-unit on our for compile scope

* Add Xlint:infer-any compiler option

* Fix build so test compiler options are actually enabled

* Enable more scalac linting options

* Add '-Xlint:eta-sam'

* Add 'Xlint:inaccessible,Xlint:missing-interpolator'
2021-01-03 08:01:26 -06:00
Ben Carman
8912fcbc82 Fail broadcasting transaction when disconnected (#2336)
* Fail broadcasting transaction when disconnected

* Wait until disconnected

* Move test

* Move to separate file

* Attempt fix

* Make test never have a peer

* Improve reliablity
2020-12-17 13:28:32 -06:00
Ben Carman
1225584526 Multi Wallet support (#2345)
* Multi Wallet support

* Fix Oracle tests

* Fix some CI errors

* Increase KeyManagerAppConfig code coverage

* Add test for multiple seeds in seed folder

* Move things in reference.conf files

* Lower key manager coverage requirement

* Fix postgres test issues

* Create DbAppConfig

* Add some docs, fix hardcoded postgres db name

* Clarify db vs schema

* Add character types

* Fix chaindb in conf

* Always copy default seed

* Remove need for driverName config
2020-12-16 17:27:56 -06:00
Ben Carman
9291d6ec3a Fix P2PClient parsing unknown messages (#2315)
* Fix P2PClient parsing unknown messages

* Use Vector, calculate correct checksum
2020-12-06 13:49:30 -06:00
Ben Carman
8cdddfecde Implement BIP 155 addrv2 messages (#2321)
* Implement BIP 155 addrv2 messages

* Add unit tests
2020-12-06 08:07:57 -06:00
Ben Carman
197367dc33 Request witness blocks from peers (#2289) 2020-11-23 22:08:03 -06:00
Chris Stewart
66ec96b1f5 2020 11 13 issue 2258 (#2260)
* Introduce ChainHandlerCached which behaves like the old ChainHandler. Now Chainhandler.getBestBlockHeader() will read headers from the database

* Remove ChainHandler.blockchains field, now it's only available in ChainHandlerCached

* De-futurify ChainHandler.fromDatabase()

* Adjust logging

* Patch test case

* Use BlockHeaderDAO.chainTips when getting best header rather thean BlockHeaderDAO.getBlockchains(). Implement a helper method ChainHandler.toChainHandlerCached()

* Fix chain.md,wallet.md

* Make ChainHandler.getBestBlockHeader() consider time of header if chainwork is the same. Make test cases less strict on what header is the best header when both chainwork and time are the same on the eader

* Only execute callbacks on headers that are going to be created in the database, not all headers passed into ChainHandler.processHeadersWithChains()

* Turn up log level again

* Small optimizations, check if we have seen a header before before processing it in ChainHandler.processHeadersWithChains(). Fix FilterSyncMarker.toString(). Use ChainHandlerCached in Node

* Remove ChainHandlerCached in appServer, re-add it in Node.scala
2020-11-17 06:19:07 -06:00
Chris Stewart
581d46c2c8 2020 11 02 cleanup (#2233)
* Cleanup a bunch of imports in test

* Use @nowarn annotation for testing things that are deprecated

* Fix base58 test

* Cleanup crypto-test

* Use scala collection compat dependency to get access to the @nowarn annotation on scala 2.12.x

* Rework more scala 2.13.x compile failures in chain-test and key-manager-test

* Optimize imports for entire project

* Fix nits
2020-11-03 11:07:04 -06:00
Chris Stewart
ade2503702 2020 10 05 redo config (#2121)
* Use absolute paths for resolving configurations

Make log level OFF again

WIP: Move resolution of config into AppConfig.start(), things are totally broken

WIP: Make AppConfig.config private

Get all tests passing again but using absolute path in configuration

* Cherry-pick onto master

* Fix BitcoinSAppConfig configuration keys to use the full path

* Apply code review

* Fix log, move invariant for requiredConfirmations back into the lazy val

* Fix DlcOracleAppConfig to use absolute path
2020-10-05 18:00:30 -05:00
Chris Stewart
9b09b04230 Upgrade to scalac 2.13.3 (#2115) 2020-10-04 07:42:44 -05:00
Ben Carman
6bd12f1f19 Bitcoind backend on server start up (#2088) 2020-10-02 10:33:24 -05:00
Ben Carman
f19cba90d3 Bump user agent to new version (#2055) 2020-09-24 13:27:10 -05:00
Ben Carman
795d62c041 Change requesting filters log to use big endian hashes (#2048) 2020-09-23 12:02:34 -05:00
Ben Carman
1e3aee55c3 Merge pull request #2019
* Refactor logging to only use grizzled slf4j

* Fix docs

* test travis config change
2020-09-19 14:09:12 -05:00
Chris Stewart
76b9577181 Introduce 'FilterSyncMarker' to ChainApi, make it clearier what exact… (#2003)
* Introduce 'FilterSyncMarker' to ChainApi, make it clearier what exactly the (Int,DoubleSha256Digest) tuple is returned from ChainApi.nextBlockHeaderRange()

* Fix doc

* Add scaladoc to FilterSyncMarker

* Rebase onto master, fix conflicts to use FilterSyncMarker
2020-09-11 13:48:40 -05:00
Chris Stewart
2980e3fc01 Resolve issues with reorgs and syncing filters (#1969)
* Pull over test cases, WIP

* Rework ChainHandler.nextBlockHeaderRange() to use hashes rather than heights to very integrity of header chain, add test case

* Modify test case to make sure we are generating the 'heaviest chain work' header as first in the tuple that is used in reorg test cases

* Refactor to helper method called 'findNextHeader()' to make things simpler

* Add scaladoc to ChainHandler.findNextHeader()

* WIP

* Fix bugs in ChainApi.nextBlockHeaderRange(), now return (startHeight,stopHash) rather than (stopHeight,stopHash)

* Get chain handler tests passing, remove genesis filter header/fitler from default chain project fixture, make it a specialized one to have genesis filter header/filter in the database. Use that specialized fixture in ChainHandlerTest for now

* FilterSync work without having the genesis filter and filter header inserted into the database

* fix bug in DataMessageHandler where we were using sendNextGetCompactFilterHeadersCommand rather than using peerMsgSender.sendGetCompactFilterHeadersMessage

* Refactor ChainHandler.findNextHeader() to centralize where chains are fetched

* Address ben's code review
2020-09-11 12:41:25 -05:00
Chris Stewart
39ef35b9ee Rename parameter names from 'duration' -> 'interval' just like our ActorSystem.schedule() names the parameter (#2005) 2020-09-11 11:10:59 -05:00
rorp
c62be8b5f6 Update callbacks for LN support (#1938)
Update callbacks for LN support
2020-09-09 11:02:31 -07:00
Chris Stewart
98236835a5 Rename ChainApi.nextHeaderBatchRange -> ChainApi.nextBlockHeaderBatchrnage (#1957) 2020-09-02 13:19:20 -05:00
Chris Stewart
e7993890cb Make sure both filter ehaders and filters are empty before sending fi… (#1936)
* Make sure both filter ehaders and filters are empty before sending first filterheader sync message

* Address nit
2020-08-31 06:32:01 -05:00
Chris Stewart
d3af9c2ccb Implement getBestFilterHeader based on a number of block headers that… (#1926)
* Implement getBestFilterHeader based on a number of block headers that can be passed in as a parameter. These headers can be used to indicate what your current best chain is

* Bring back compiler opts

* Fix compiler error for maxByOption as it isn't in the 2.12 std library

* Implement a context free best filter headers search. The basic strategy is to look at headers in the _future_ of our current best filter header

* Fix bug in sql query were we were doing max chainWork too early on block headers, we needed to filter out headers in our set and _then_ we get the max chain work

* Add more unit tests
2020-08-29 08:13:38 -05:00
Ben Carman
ef4329d283 Warn if peer does not support services we need (#1921) 2020-08-27 13:57:58 -05:00
Chris Stewart
13fbb3f889 Improve logging in DataMessageHandler (#1922)
* Improve logging in DataMessageHandler

* Make hashes in logs big endian

* One more big endian conversion i forgot
2020-08-27 12:39:45 -05:00
Chris Stewart
3ab280a12b Remove AppConfig.initialize() in favor of AppConfig.start() (#1907)
* Remove AppConfig.initialize() in favor of AppConfig.start()

* Fix docs
2020-08-26 16:24:38 -05:00
rorp
23685f124e Add an ability to one Postgres database for all sub-projects (#1897)
* Add an ability to one Postgres database for all sub-projects
2020-08-26 12:20:18 -07:00
Ben Carman
581851f22c Clean up calls in Node and Chain (#1896)
* Clean up calls in Node and Chain

* Remove parens on non-modifying call
2020-08-26 13:04:34 -05:00