Commit Graph

391 Commits

Author SHA1 Message Date
rorp
de43dadf52
Network notifications (#4774)
* Network notifications

* add more tests

* don't drop futures on the floor

* Improve logging, don't drop futures on floor

* scalafmt

* add error messages to the failure notifications

* update unit tests

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2022-10-09 07:46:13 -05:00
Chris Stewart
73200b64bf
re-order RPC command order to have commands first that do not depend on tor (#4811) 2022-09-30 17:06:17 -05:00
Chris Stewart
bcddb015ae
2022 09 12 Filter header and filter websocket events (#4777)
* implement chain callbacks for compact filter headers / compact filters

* Wire it up through the websocket

* Fix type for compactfilterprocessed

* Fix bug to write json rather than write a string
2022-09-15 20:05:54 -05:00
Chris Stewart
7b7847885e
Attempt to reduce redundant work on startup to speed up start up time (#4764)
* Attempt to reduce redundant work on startup to speed up start up time

* Move dlcConf out of dependentTorConfigs
2022-09-11 13:15:53 -05:00
Chris Stewart
7460dcd255
Parallelize running of individual migrations (#4761) 2022-09-10 15:10:14 -05:00
Chris Stewart
7d9d0c577f
Run scalafmt (#4757) 2022-09-09 14:58:29 -05:00
rorp
16893f999e
DLC connection checks and notifications (#4720)
* DLC connection checks and notifications

* asynchronous connection checks
2022-09-09 14:09:38 -05:00
Chris Stewart
7fe9bdbe35
re-add Compile / fork in server.sbt so we can run appServer from sbt console (#4730) 2022-09-08 10:13:09 -05:00
Chris Stewart
6b432ea509
Emit 2000th header received during IBD (#4719) 2022-09-06 14:24:03 -05:00
Chris Stewart
a36e55c892
Fix patterns I missed when updating logback (#4713) 2022-09-06 10:41:32 -05:00
benthecarman
2cfd6f3591
Allow for custom config file name (#4709) 2022-09-05 16:04:25 -05:00
Chris Stewart
2bf1c9d1a5
Try to debug what is happening on CI with zmq test (#4708)
* Try to debug what is happening on CI

* Turn off logging

* Fix compile errors

* try to add sleep to get zmq started

* Cleanup

* Empty commit
2022-09-05 13:23:10 -05:00
Chris Stewart
3ee4fe1138
Upgrade dependencies (#4705)
* Upgrade dependencies

* upgrade micropickle to 2.0.0

* Add .env file with FLYWAY_CLEAN_DISABLED=false to allow cleaning in test cases

* Fix logging pattern bug

* Add action to load .env file

* Add default flyway.conf at root of repository, disable flywayClean

* remove env action
2022-09-05 07:11:23 -05:00
rorp
2448fe13e8
Revert the unique outpoint index DB migration (#4652)
* Revert the unique outpoint index DB migration

* fix tests

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2022-09-04 10:24:30 -05:00
rorp
7322186b56
Fix Tor hidden service config (#4689)
* Fix Tor hidden service config

* default the target list to an empty list

* Scala 2.12 compatibility

* update docs

* Fix typo

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2022-09-01 16:54:39 -05:00
Chris Stewart
f65b483d6f
Allow LnMessage(tlv) & raw tlv to be used in DLCRoutes (#4682)
* Allow LnMessage(tlv) & raw tlv to be used in DLCRoutes from when reading binary files

* Implement actual fix with test case

* Fix try deserializing to tlv in a few more places

* Turn logging off
2022-08-30 06:01:46 -05:00
Chris Stewart
c266ba849f
Set entrypoint scripts so they are executable by anyone, revert daemonUserId to test on umbrel (#4678) 2022-08-29 10:38:50 -05:00
Chris Stewart
ac24bfb230
Revert #4601 / #4624 so that docker works on umbrel again (#4665)
* Revert #4601 / #4624 so that docker works on umbrel again

* Fix
2022-08-25 11:21:29 -05:00
Chris Stewart
288918d705
Partially revert #4601 and re-add daemonUser := bitcoin-s (#4655)
* Partially revert #4601 and re-add daemonUser := bitcoin-s

* scalafmt

* scalafmt again?
2022-08-23 14:36:08 -05:00
Chris Stewart
2001e86ab9
If DISABLE_JLINK set, don't try to chmod jre/bin/java (#4651)
* If DISABLE_JLINK set, don't try to chmod jre/bin/java

* Do it for oracle server as well
2022-08-23 13:16:42 -05:00
Chris Stewart
a0eb0824a2
Fix some bugs around resetting rescan state when a rescan fails (#4642)
* Fix some bugs around resetting rescan state when a rescan fails

* Add test case

* Use .fail() method
2022-08-22 12:55:18 -05:00
Chris Stewart
86463231d6
2022 08 18 Add configuration to only emit websocket events when IBD is done (#4627)
* Implement isinitialblockdownload flag in databaes

* Change to IsInitialBlockDownload to be less confusing

* Fix missing refactors, add ChainApi.isIBD()

* Implement logic for ChainApi.isIBD(), add configuration flag to toggle whether to emit websocket events while IBD is ongoing or not, add isinitialblockdownload field to getinfo endpoint

* Remove confusing ChainApi.isIBD() that didn't accurately account for the case where IBD wasn't set in the database

* Fix bug around emitting the websocket events

* Add some documentation
2022-08-22 10:09:00 -05:00
Chris Stewart
b18da7ac2b
Fix bug where it prevented us from rescanning an empty wallet (#4632)
* Fix bug where it prevented us from rescanning an empty wallet with a seed to try and recover funds

* Fix unit test to not expect isEmpty() wallet calls
2022-08-20 17:03:22 -05:00
Chris Stewart
2dad9f57b8
Fix bug where we weren't specify the bash script for each project to modify file permissions (#4625) 2022-08-18 08:29:10 -05:00
Chris Stewart
b6710e7043
.map() on loadwallet result before starting http server so wallet is fully loaded (#4614) 2022-08-16 09:36:21 -05:00
Chris Stewart
a0851fea58
Run migrations upfront (#4607) 2022-08-15 10:29:38 -05:00
Chris Stewart
d0dadfa423
2022 08 11 issue 4600 and only emit syncing ws event when we are actually syncing (#4604)
* Check if bitcoind is syncing before running scheduled job to poll bitcoind

* Set syncing flag to true if initialblockdownload is true

* Implement logic so we only emit websocket events when state is changed, rather than everytime we poll bitcoind

* Implement logic to only emit websocket event when descriptor flag changes rather than when it is set
2022-08-12 14:34:03 -05:00
GreyMcCarthy
c34b0de886
removed logger parameter from where it's used with callbacks (#4598)
* removed logger parameter from where it's used with callbacks

* removed logger from onTXRecived... and onBlockRecived...

* removed unused imports

* removed logger from walletCallbacks
2022-08-12 12:09:58 -05:00
Chris Stewart
b0831f26a7
Refactor waiting for bitcoind lost blocks into helper method, add context (#4602) 2022-08-12 12:08:27 -05:00
Chris Stewart
95139bdc3d
Improve logging on getBlockchainInfo() (#4590) 2022-08-09 09:18:53 -05:00
rorp
9f89ba9b7a
loadwallet endpoint (#4417)
* `loadwallet` endpoint

* Scala 2.12 compatibility

* remove `getwalletname`

* fix unit tests

* fix DbManagementTest

* fix wallet tests

* fix DbManagementTest

* fix RejectedExecutionException

* fix postgres tests

* fix password override

* update `loadwallet`

* Move WalletHolder into wallet module

* Kill use of null in WalletHolder

* Kill use of null in WalletHolder

* Refactor to use WalletAppConfig.DEFAULT_WALLET_NAME

* update cli and unit tests

* cleanup

* Refactors to get compatible with master

* Fix compile

* Distinguish between loadWalletNeutrino & loadWalletBitcoind

* Fix initialized bug with WalletHolder in loadWallet methods

* Refactor loadWalletBitcoindBackend/loadNeutrinoWallet to have a common helper method

* Introduce DLCWalletLoaderApi, refactor codebase to use it

* Fix dumb initialization bug

* Fix test:compile

* Cleanup

* Get loadwallet mostly working with neutrino wallet

* Fix compile

* scalafmt

* Move loadwallet into app-commons

* Remove {DLCWallet,Wallet}.stop()

* Implement LoadWallet json parsing unit test

* Implement stopping of NodeStreamManager callbacks

* Fix rebase, integrate setRescanState into WalletRoutes

* Fix import

* Fix unit tests

* Implement listwallets in ConsoleCli

* Fix imports

* Cleanup

* scalafmt

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2022-08-05 17:34:14 -05:00
Chris Stewart
ad21a11254
Create BitcoindStreamUtil and refactor to use it (#4578)
* Create BitcoindStreamUtil and refactor to use it

* Fix BitcoindBackendTest bugs

* Fix handling of chainCallbacks
2022-08-05 15:31:00 -05:00
Chris Stewart
c03b158f94
Implement RescanTerminatedEarly exception to terminate the stream, implement ArgumentSource (#4574)
* Implement RescanTerminatedEarly exception to terminate the stream, implement ArgumentSource to understand where arguments passed into bitcoin-s are coming from

* Bump timeout
2022-08-04 14:11:07 -05:00
Chris Stewart
a02e25b0ce
Refactor WalletRoutes to take DLCWalletLoaderApi as a paramete (#4565)
* Refactor WalletRoutes to take DLCWalletLoaderApi as a parameter, use the walletHolder inside of DLCWalletLoaderApi to fulfill requests

Fix rebase

Reduce parallelism in fetchFiltersFlow to 1 to reduce chance of hitting akka's max open requests

* Try fix loader test by catching RejectedExecutionException

* Add println to see if catching excpetion gets hit

* WIP: Try to fix recursive rescan

* Don't map on recursiveRescan so we don't block on it
2022-08-04 09:16:46 -05:00
Chris Stewart
8cb45e8208
Add unit tests for loader rescans (#4570)
* Add unit tests for loader rescans

* Move common methods into super class to be DRY

* Cleanup
2022-08-03 13:16:52 -05:00
Chris Stewart
1d3db879ad
Add rescanState variable to DLCWalletLoaderApi (#4569)
* Add rescanState variable to DLCWalletLoaderApi so we can track rescan state accurately

* Add isRescanStateEmpty

* scalafmt
2022-08-03 06:33:26 -05:00
Chris Stewart
ec0103dab3
2022 08 02 loadwallet startup logic (#4568)
* move handleDuplicateSpendingInfoDb into DLCWalletLoaderApi

* Pull over restartRescanIfNeeded to DLCWalletLoaderApi
2022-08-02 18:22:17 -05:00
Chris Stewart
08699c0ee0
Remove nested Option[Option[AesPassword]] (#4564) 2022-08-02 14:08:26 -05:00
GreyMcCarthy
b7c7cb8fc2
Moving duplicate dataStructures in cli and app-server into app-commons (#4541)
* duplicate data structures in (ConsoleCli.scala) and (ServerJsonModels.scala) moved to (Command.scala)

* removed unused imports

* rebase for merge conflitcts

* change output type for Accept DLC

* more error fixes

* error fixes

* fixed destintaion error

* scala reformatting

* send from outpoints test runs and passes

* removed unnecessary braces, removed incorrect extensions on objects

* scala reformatting

* removed commented out code
2022-08-02 11:55:43 -05:00
Chris Stewart
191df09196
Move pollBitcoind out of startBitcoindBlockPolling (#4559)
* Move pollBitcoind out of startBitcoindBlockPolling

* Rework startBitcoindBlockPolling to not return Future[Cancellable]

* Stop NodeCallbacksStreamManager inside of NodeAppConfig.stop()

* Make BitcoindSyncState which encapsulates syncing with bitcoind and polling of mempool

* Fix bug where processingBitcoindBlocks cannot be top level val as that method may be used by different parts of the codebase

* Rename isPolling -> processingBitcoindBlocks

* Cleanup more resources in appServerTest/test

* Complete polling stream regardless if we synced blocks are not to complete the Future returned by pollBitcoind

* Empty commit to run CI
2022-08-02 06:40:17 -05:00
Chris Stewart
f487c1270e
Implement cleanup of dlcWallet/wallet DbAppConfig threadpools (#4557)
* Implement cleanup of dlcWallet/wallet DbAppConfig threadpools

* Cleanup

* Remove uncessary walletHolder.start()
2022-08-01 06:55:02 -05:00
Chris Stewart
936a65edd4
2022 07 21 Modify helper methods in BitcoindRpcBackendUtil to not materialize streams eagerly. (#4526)
* Refactor syncWalletToBitcoind impl to be completely stream based

* Add getParallelism, refactor usages in BitcoindRpcBackendUtil to use it

* Fix compile

* Replace usages of 'WalletApi with NeutrinoWalletApi' with NeutrinoHDWalletApi

* Revert wallet sync height logic

* Refactor to make things cleaner
2022-07-31 12:16:26 -05:00
Chris Stewart
4b83286922
2022 07 25 wallet api refactor (#4545)
* Move MockChainQueryApi/NodeApi out of BaseWalletTest

* refactor BitcoinSServerMain.start() to return WalletHolder

* Add walletConfig to WalletWithBitcoind

* Move findOutputsBeingSpent into WalletApi

* Add WalletApi.clearAllAddresses(), HDWalletApi.{findAccount, getnewaddress(account)}

* Add HDWalletApi.fundRawTransaction() with an account as parameter

* Add WalletApi.findByScriptPubKey()

* Fix lots of tests in WalletIntegrationTest

* Create WalletApi.processOurTransaction()

* Get things compiling

* Fix tag integration test

* Refactor AnyDLCHDWalletApi -> DLCNeutrinoHDWalletApi

* Fix docs

* Get postgres tests passing locally

* Move initBalance map before calling callback in CallBackUtilTest

* Get compile working, rename to destroyOnlyWalletWithBitcoindCached

* Fix docs

* Fix missing destroyWalletAppConfig

* Fix scalafmt

* Fix bug in dlcWalletTest where wallet db thread pools weren't being shutdown after unit test completes

* Empty commit
2022-07-31 08:04:39 -05:00
Chris Stewart
6c76639ba3
Move processBlock into WalletApi (#4543) 2022-07-27 09:32:22 -05:00
Chris Stewart
d508d65142
Implement ability to cancel bitcoind polling jobs (#4538)
* Implement ability to cancel bitcoind polling jobs

* Fix bug so we don't block application on fully syncing wallet against bitcoind
2022-07-24 19:56:53 -05:00
Chris Stewart
4ef27dd163
Fix bug where dlcNode was not apart of tor dependent configs (#4537) 2022-07-24 16:48:25 -05:00
Chris Stewart
67dd16d8ad
Add missing calls to BitcoinSAppConfig.stop() (#4535) 2022-07-24 15:59:03 -05:00
Chris Stewart
b905afa65e
2022 07 20 wallet rescan stream (#4530)
* Implement rescan with akka streams

Get basic stream working with rescan

Fix bug where we weren't using rescan specific threadpool

Implement killswitch for rescan

* WIP: Expose promise to allow external completion of rescan stream

* Rework RescanStarted.RescanStarted to contain a promise to complete the stream early, and a future that represents the completed streams materialized value

* Comment cleanup

* Fix compile errors, remove killswitch

* Fix 2.12.x compile

* Introduce ActorSystem into wallet, refactor rescans to use that ActorSystem

* Fix import

* Fix bug where we were prepending instead appending to batched Vector

* Propogate RescanState upwards into WalletRoutes

* Refactor fetching of filters to be a Flow
2022-07-24 12:26:21 -05:00
Chris Stewart
d4210fad65
Pull over changes from loadwallet branch (#4527) 2022-07-21 12:12:32 -05:00
Chris Stewart
0a127368f0
2022 07 18 node callback stream manager (#4520)
* Implement akka stream proxy for nodecallbacks

Add killswitch to createBitcoindNodeCallbacksForWallet

Add unit test for killswitch in createBitcoindNodeCallbacksForWallet

Add delays to make sure callbacks are executed

Fix rebase

Move killswitch out of methods into class level val

* Bump timeout

* Attempt to implement NodeCallbacks proxy with akka streams

* Refactor CallbackUtil to use NodeCallbackStreamManager

* Rebase & remove killswitch

* Implement NodeCallbackStreamManager.+ method
2022-07-20 06:28:55 -05:00