Commit Graph

244 Commits

Author SHA1 Message Date
Chris Stewart
bc79a24f53
Get both bundle and app server logging working (#3362)
* Get both bundle and app server logging working, oracleServer does not compile

* Get everythig compiling

* Fix bug where bundle wouldn't use command line param rpc port for ConsoleCli

* Add unit tests to make sure we are translating to config properly

* Fix bug where app config wasn't re-created after parsing bundle args

* Fix datadir resolution in BundleGUI

* Implement force-recalc-chainwork

* fix bug

* Try not using tilde on windows

* Try to print datadir on CI

* Try using stackoverflow answer

* Try escaping windows path

* Take ben's suggestion of using AppConfig.safePathToString

* Just check the paths are equal

* Fix nullpointerexceptions on startup

* Fix bug where bitcoin-s-bundle.conf was not being read

* Remove usage of --conf flag in bundle when starting backend server

* Fix compile

* Rename usedDir -> networkDir

* Fix passing in datadir
2021-07-07 10:59:11 -05:00
benthecarman
629c2a2c31
Set sync height on new wallet (#3368) 2021-07-06 12:33:38 -05:00
benthecarman
0d2bc7a927
Remove unneeded asInstanceOf calls in DLCWallet (#3345) 2021-06-28 22:34:42 -05:00
benthecarman
fdba5ad6be
Silence warning log if no error (#3314) 2021-06-22 12:03:36 -05:00
benthecarman
bd11c84462
Add ability to sweep wallet (#3274)
* Add ability to sweep wallet

* Respond to review

* empty
2021-06-16 14:37:40 -05:00
Chris Stewart
5caf7ee38b
2021 06 10 cache spendinginfodbs for block (#3245)
* Add mark as spent optimization to batch writes rather than write inside the method

* Scaladoc refactor

* Refactor received spendingInfoDbs to be cached outside of processTransactionImpl()

* Implement processBlockCachedUtxos() to improve performance

* Fix typo
2021-06-11 15:14:02 -05:00
Chris Stewart
af8aaa7bad
2021 06 07 markasspent optimization (#3244)
* Add mark as spent optimization to batch writes rather than write inside the method

* Scaladoc refactor
2021-06-11 13:26:24 -05:00
Chris Stewart
8574edede7
Move addressDb database read out of inner loop (#3239)
* Move addressDb database read out of inner loop and into processReceivedUtxo() to hopefully improve performance

* Cleanup
2021-06-09 12:18:57 -05:00
benthecarman
be8e965367
DLC Wallet pulldown (#3138)
* DLC Wallet pulldown

* Respond to review
2021-05-25 17:31:01 -05:00
Chris Stewart
cea8802c05
parallelize matching compact filters as this is a bottleneck during IBD (#3137)
* parallelize matching compact filters as this is a bottleneck during IBD

* Use FutureUtil.makeAsync() to actually make it async

* Small optimization, move asmVec outside of lambda so it doesn't need to be re-computed everytime
2021-05-25 12:39:39 -05:00
Chris Stewart
68c7bc1040
2021 05 21 fetch height parallel (#3124)
* Fetch block height in parallel for Wallet.processCompactFilters()

* Make implementation safe against Vector.empty being passed as a parameter
2021-05-23 18:30:19 -05:00
Chris Stewart
9c9e27a8f5
Add optimzations for IBD when the wallet is empty, basically skip all logic for matching filters since there is nothing to match against (#3121) 2021-05-21 15:08:25 -05:00
Chris Stewart
7468cbec23
Optimize updateUtxoConfirmedStates() to fetch confirmations for blocks in parallel (#3094) 2021-05-16 17:10:58 -05:00
Chris Stewart
72636b7180
2021 05 09 received utxos (#3063)
* Make TransactionProcessing.processTransactionImpl() a bit more parallel, fetch all relevant received and spent txos in parallel

* More refactors, make sure return types are Vector rather than Seq to make sure we don't have any pathlogical surprises with scala collections

* Fix type annoation to be Vector

* Fix remaining usage of Seq in TransactionProcessing

* Fix pattern match on Vector on 2.12.x

* Change input to Vector rather than Seq in processSpentUtxos()
2021-05-12 10:45:23 -05:00
Nadav Kohen
78f4dfb8c6
Pubkey Refactor (#2936)
* Removed ExecutionContext from ECKey

* Refactored ECPublicKey to remove compression state and introduced ECPublicKeyBytes to handle cases where serialization of input is important

* Fixed the rest of bitcoin-s so that it passes all tests

* Made all ECKeys into case classes

* Successfully added isFullyValid invariant to ECPublicKey!

* Fixed docs

* Added scaladocs and fixed a RpcPsbtResult bug

* Reject private keys of length < 32 and fix WIF parsing bug
2021-05-06 13:19:52 -05:00
benthecarman
682a759977
Fix wallet v11 postgres migration (#3009) 2021-05-02 10:27:07 -05:00
Chris Stewart
a2911f31ed
Fix race condition with BitcoindChainHandlerViaZmqTest (#2990)
Turn off logging again

Try to move things around even more to make sure zmq is started

Turn logging level down to ERROR to hopefully get some meaningful output

Adjust both valdiation & ChainHandler to be ERROR level logging

switch zmqpubrawblock to be the port we know is free

Reduce logging to WARN

Bump bitcoind max retries to 120

Fix rebase

Revert logging

Remove some noisy logs

Restore logs to trace level

Add explicit tag
2021-05-01 13:11:13 -05:00
Chris Stewart
73939a15fc Call WalletAppConfig.stop() when destroying wallet in test fixtures (#2975)
* Fix bug introduced in PR 2597, we need to explicitly call WalletAppConfig.stop() now rather than transitively calling it via Wallet.stop()

* Fix calls to wallet.stop() -> wallet.walletAppConfig.stop() in BitcoindBlockPolling,BitcoindBackend. Also add some explicit type annotations
2021-04-26 15:48:33 -05:00
Chris Stewart
27afb66220 2021 04 23 issue Move rebroadcast scheduling into WalletAppConfig (#2957)
* Move startRebroadcastScheduler() and stopRebroadcastRescheduler() into WalletAppConfig

* Add logs for wallet rebroadcast task
2021-04-24 08:57:36 -05:00
benthecarman
cbfbdd17ba Call .hex on all txIds and blockhashes in logs for TxProcessing (#2939) 2021-04-21 19:48:00 -05:00
Chris Stewart
c95c0f9706 Move wallet scheduler into WalletAppConfig (#2933)
* Move wallet scheuler into WalletAppConfig, use scheduler.shutdownNow() so we actually shutdown the scheduler rather than wait for all queued tasks to finish executing before shutting down

* Fix typo

* Move rescan thread pool in to walletAppConfig, shut it down on wallet shutdown

* Fix compile
2021-04-21 11:06:29 -05:00
Chris Stewart
13fc3c2b4e 2021 04 18 Reset txo state when overwriting spendingTxId (#2919)
* Add invariant to spendingInfoDb to that requires if the spendinginfodb is in a TxoState.spentStates, the SpendingInfoDb.spendingTxIdOpt is defined

* Remove unused SpendingInfoDAO.updateTxoState()

* Fix bug that doesn't revert TxoState to TxoState.BroadcastSpent when overwriting the spendingTxId

* Remove unecessary transition from TxoState.Reserved -> TxoState.PendingConfirmationsReceived inside of processSpentUtxos()

* Update TransactionProcessing.markAsSpent() to treat TxoState.PendingConfirmationsSpent and TxoState.ConfirmedSpent the same way

* Remove unecessary comment
2021-04-21 07:21:42 -05:00
benthecarman
38fdbb33c4 Add test for tx that doesn't originate from wallet (#2932) 2021-04-20 18:34:38 -05:00
benthecarman
105942efa2 Use filters for bitcoind backend syncing if available (#2926)
* Use filters for bitcoind backend syncing if available

* Start bitcoind polling after sync is done

* Fix compile issue and test
2021-04-20 17:28:23 -05:00
Chris Stewart
19319494cd 2021 04 19 Cleanup after ourselves in postgres tests (#2921)
* Fix missing database shutdown on WalletAppConfig.stop()

* Cleanup BitcoindBlockPollingTest

* Cleanup ProcessBlockTest and all tests that use CachedBitcoind

* Cleanup BitcoindBackendTest

* Cache rescan thread pool rather than creating a new one everytime
2021-04-19 13:58:00 -05:00
Chris Stewart
238c083aad 2021 04 18 wallet received txo state (#2914)
* Add invariant to spendingInfoDb to that requires if the spendinginfodb is in a TxoState.spentStates, the SpendingInfoDb.spendingTxIdOpt is defined

* Remove unused SpendingInfoDAO.updateTxoState()

* Tighten up TxoState -> ReceivedState types we are using in the wallet. The ensures we are talking specific class of states (receiving a txo) rather than accounting for the case of spending and receiving since there is different information required for the spending states

* Add TxoState.ImmatureCoinbase to ReceivedStates
2021-04-18 16:56:08 -05:00
Chris Stewart
a3954dbcae 2021 04 17 spendinfodb invariant (#2912)
* Add invariant to spendingInfoDb to that requires if the spendinginfodb is in a TxoState.spentStates, the SpendingInfoDb.spendingTxIdOpt is defined

* Remove unused SpendingInfoDAO.updateTxoState()

* Remove comment

* Make TxoState and explicit parameter for test methods. Most test methods depend on a sample utxo state, so make the caller of the method specify what state they want the txo to be in

* Fix github actions config

* Fix RoutesSpec test
2021-04-18 09:02:49 -05:00
benthecarman
d0629486ab Wallet Rebroadcast Transactions thread (#2711)
* Wallet Rebroadcast Logic

* Use wallet scheduler for address queue rather than it's own thread

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>

* Bump default

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2021-04-07 08:13:43 -05:00
Nadav Kohen
e6899b20b1 Made ECPrivateKey signing synchronous and got src compiling (#2652)
Fixed tests

De-futured tx buidling, finalizing and signing

Responded to review
2021-03-23 17:05:58 -05:00
benthecarman
07514e2348 Remove logging from core (#2810)
* Remove logging from core

* Remove slf4j from testkit core

* Remove logback from coretest
2021-03-21 19:11:55 -05:00
benthecarman
12bff309c2 Add Broadcast TxoState (#2735)
* Add broadcast TxoState

* Update scaladoc
2021-03-18 14:16:53 -05:00
benthecarman
9494eec1b8 Move blockhash to tx table from spending info table (#2744)
* Move blockhash to tx table from spending info table

* Add test, fix spending tx id col name

* Scaladocs, add test

* Add more unit test

* Make id not comparable in process tx test

* Fix tests

* attempt to fix

* Add mempool comment to scaladoc

* Deparallelize process inputs & outputs
2021-03-16 10:05:29 -05:00
Chris Stewart
bf4afd63d1 Begin re-introducing parallelism in the wallet to make everything faster (#2705) 2021-02-24 07:10:46 -06:00
Chris Stewart
1a2ddf6a0d Reduce usage of .findAll() (doesn't scale for large dbs). Now pass in… (#2706)
* Reduce usage of .findAll() (doesn't scale for large dbs). Now pass in the specific things we are searching for

* Add timestamps for checking how long it takes to processBlock()
2021-02-21 16:15:50 -06:00
Chris Stewart
b63333327f Allow implicit execution context to be passed in to RescanHandling.findMatches() & RescanHandling.fetchFiltersInRange() (#2704) 2021-02-21 07:14:29 -06:00
Aris
85087b0f70 Refactoring chain (#2662) 2021-02-13 14:10:40 -06:00
Aris
f8694eb097 Fix/typos (#2633)
* Fix typos

* Collection minor fixes

Removes redundant .headOption on options
Replaces .exists(_ == ???) with .contains(???)
Replaces option == None with option.isEmpty
2021-02-07 15:00:25 -06:00
Chris Stewart
8f958bf5d9 Update scalafmt (#2619) 2021-02-03 16:37:12 -06:00
benthecarman
03ca6f617e Optimize filter sync and fetching filter heights (#2568) 2021-01-30 14:56:47 -06:00
Chris Stewart
756bfc4427 2021 01 27 conectionpool (#2578)
* Give connection pool example config

* Get hikari logging working on postgres

* fix 2.12.x

* Bump metrics core to a supported version

* Add hikari-logging and hikari-logging-interval configuration options to turn off/on hikari and schedule how often logs appear

* Turn off hikari logging in the db-commons/reference.conf and add it to appserver/reference.conf

* default the logging to off in scala code

* Make sure connection pool is enabled in db-commons, add default database configuration to website

* Address code review, fix things that didn't need to be changed
2021-01-28 09:44:03 -06:00
benthecarman
fa27fb3fb1 Create BlockSyncState type (#2567) 2021-01-27 14:15:07 -06:00
benthecarman
29bde74cb8 Fix all DAOs to use safeDatabase (#2556) 2021-01-22 16:13:45 -06:00
benthecarman
e10e278aa4 Add walletinfo rpc (#2546)
* Add walletinfo rpc

* Move rootXpub to key manager obj

* Make configuration keys for getinfo the same as the ones in our configuration files

* Cleanup

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2021-01-22 13:55:21 -06:00
benthecarman
e2ace38a1f Fix rescan to fetch blocks during scan (#2540) 2021-01-22 10:28:53 -06:00
benthecarman
2291bf0eae Create isChange function for wallet (#2535)
* Create isChange function for wallet

* Scaladocs
2021-01-20 12:11:24 -06:00
Chris Stewart
6193af7709 Remove WalletSync.sync() -> WalletSync.syncFullBlocks() (#2522) 2021-01-15 17:05:11 -06:00
Chris Stewart
98ace6f14e 2021 01 02 issue 2457 (#2461)
* WIP

* Get neutrino node with wallet 'receive information about received payments' working again

* Fix compile

* Remove initial sync logic from test case

* Remove sync logic in NeutrinoNodeWithWallet test cases

* Improve logging and rename a few things

* WIP2

* WIP3

* Get NeutrinoNodeWithWallet tests working

* Implement WalletSync, which allows you to sync a wallet from a arbitrary data source

* Get all tests passing again

* Use spv.appConfig in DataMessageHandlerTest rather than caching the config

* Modify cleanup to hopefully get CI passing

* Fix postgres tests by cleaning the table during the destroy phase of the test fixture. This is needed because the same postgres database is shared between tests in the same test suite

* Revert logback-test.xml

* Get sqlite/postgres tests passing pt 2

* syncHeight -> syncDescriptorOpt()

* Add case for genesis block hash in WalletSync

* Fix SpvNodeWithWallet test case to actually test spv functionality

* Remove nodeCallbacks parameters, callbacks should be registered on nodeAppConfig
2021-01-09 09:33:37 -06:00
Chris Stewart
dc73cb26ff Rename wallet.getSyncHeight() -> wallet.getSyncDescriptorOpt(). We don't just use height in the descriptor, the hash is just as valuable for connecting to chains (#2479) 2021-01-07 07:40:30 -06:00
Chris Stewart
4e1a6c69f0 2020 12 18 enable lint options (#2454)
* Enable -Xlint:adapted-args,nullary-unit on our for compile scope

* Add Xlint:infer-any compiler option

* Fix build so test compiler options are actually enabled

* Enable more scalac linting options

* Add '-Xlint:eta-sam'

* Add 'Xlint:inaccessible,Xlint:missing-interpolator'
2021-01-03 08:01:26 -06:00
benthecarman
4a41bacaa0 Use written txDbs in TransactionProcessing (#2449) 2020-12-31 07:39:24 -06:00