Commit Graph

278 Commits

Author SHA1 Message Date
Ben Carman
d54c15d4cb Remove requirement for ZMQ with bitcoind backend (#2137)
* Remove requirement for ZMQ with bitcoind backend

* Add test, move poller to BitcoindRpcBackendUtil

* Add docs & logs, fix niche case

* Use AtomicReference, reset prevCount on failure
2020-10-12 07:59:57 -05:00
Ben Carman
09dfd5eb73 Add BIP45 Multisig Purpose (#2103)
* Add BIP45 Multisig Purpose

* Fix multisig purpose, add tests
2020-10-02 10:43:25 -05:00
Ben Carman
ed5228310b Use SubtractFeeFromOutputsFinalizer when sending full utxos (#2072) 2020-10-01 17:32:24 -05:00
Ben Carman
a694ef16b7 Add ability to fully spend utxos (#2063)
* Let RawTxSigner dummy sign transactions

* Add ability to fully spend utxos

* Fix NeutrinoNodeWithWalletTest

* Fix test, simplify call
2020-09-27 08:16:26 -05:00
Ben Carman
3ec3b5d699 Merge pull request #2029
* Add more processing blocks tests

* Add immature coinbase txo state

* Test balance, add doc
2020-09-20 10:39:31 -05:00
Ben Carman
1e3aee55c3 Merge pull request #2019
* Refactor logging to only use grizzled slf4j

* Fix docs

* test travis config change
2020-09-19 14:09:12 -05:00
Chris Stewart
00480cc391 Use randomized fee rates for wallet tests (#1977)
Co-authored-by: Ben Carman <benthecarman@live.com>
2020-09-09 16:46:33 -05:00
rorp
c62be8b5f6 Update callbacks for LN support (#1938)
Update callbacks for LN support
2020-09-09 11:02:31 -07:00
Ben Carman
5f7356c526 Fix Two KeyManagers in scope for fundRawTransactionInternal (#1986)
* Fix Two KeyManagers in scope for fundRawTransactionInternal

* Only fetch txs once
2020-09-09 09:40:38 -05:00
Ben Carman
edc80bf13e Don't spend immature coinbases, fix tests (#1981) 2020-09-07 15:08:41 -05:00
Ben Carman
c1d0d66398 Calculate correct TxoState when processing a transaction (#1929)
* Calculate correct TxoState when processing a transaction

* Test balance
2020-08-28 15:31:48 -05:00
Ben Carman
8d47b68764 Fix rescans that are larger than the batch size (#1916)
* Fix rescans that are larger than the batch size

* Add test
2020-08-27 14:09:25 -05:00
Chris Stewart
955978f635 Apply string factory to a bunch of easy things (#1891)
* Apply string factory to a bunch of easy things

* Refactor HDPath, ExtKey, and LnTagPrefix to use StringFactory

* Implemen StringFactory on LnHumanReadablePart, ScriptType, ServiceIdentifier

* Implement StringFactory on AesCrypt, NodeUri, AddressTag, PSBT, TxoState

* Fix failing tests, fix website compile

* Apply StringFactory to all ScriptOperations, ChannelState, Script Parsing
2020-08-25 10:48:37 -05:00
Ben Carman
304896e171 Move WalletApi to Core (#1890) 2020-08-25 07:05:20 -05:00
Ben Carman
98164bc5e3 Move WalletApi necessities to core (#1886) 2020-08-24 17:26:58 -05:00
Ben Carman
cfde581d0c Move all remaining wallet db representations (#1885) 2020-08-24 13:13:52 -05:00
Ben Carman
c6ad90139a Move TransactionDb to Core (#1849)
* Move TransactionDb to Core

* Move to api package
2020-08-22 08:04:30 -05:00
Ben Carman
9e06b826e1 Move AddressTagDb to Core (#1850)
* Move AddressTagDb to Core

* Move to wallet api package
2020-08-21 16:31:30 -05:00
Ben Carman
d187834e1a Create KeyManager Api & move SpendingInfoDb to core (#1848)
* Create KeyManager Api & move SpendingInfoDb to core

* Fix docs, remove sealed form KeyManagerApi

* Move SpendingInfoDb to core wallet api
2020-08-21 14:37:12 -05:00
Ben Carman
683c11b2ab Fix selecting Utxos twice in fundRawTransactionInternal (#1866)
* Fix selecting Utxos twice in fundRawTransactionInternal

* Simplify watch an arbitrary SPK test
2020-08-21 10:11:02 -05:00
rorp
2f8dcd1e57 Watch arbitrary SPKs (#1860)
* Watch arbitrary SPKs

* fix unit tests

* fix unit tests

* one more fix

* revert the compiler parameters
2020-08-20 14:33:18 -05:00
rorp
b5a40035ce Refactor wallet DB (#1798)
* Refactor wallet DB

* fix create(), update() and upsert()

* remove accout foreign key

* fix wallet fixtures

* update Postgres schema

* update SQLite schema

* fix AddressHandlingTest

* fix SQLITE_BUSY exception in TransactionProcessing

* fix RescanHandlingTest

* revert NeutrinoNodeTest

* fix SpendingInfoDAOTest

* fix SQLITE_BUSY exceptions

* fix DbManagementTest

* fix "Task rejected" Slick exceptions in Postgres mode

* responded to the PR comments
2020-08-18 13:43:12 -05:00
Ben Carman
562f789504 Create primary key of (address, tag_type) for AddressTag table (#1828) 2020-08-15 12:52:00 -05:00
Ben Carman
48a7f6f86e Remove default false for useLogbackConf (#1816)
* Remove default false for useLogbackConf

* Fix docs

* Set logging level to WARN for tests
2020-08-14 08:53:07 -05:00
Ben Carman
403ed33dbd Require utxos aren't spent when sending from outpoints (#1817) 2020-08-14 08:43:18 -05:00
Ben Carman
62e9e08cf6 Address Tagging/Labeling Support from CLI (#1790)
* Address Tagging/Labeling Support from CLI

* Scaladoc
2020-08-12 10:55:15 -05:00
Ben Carman
4ab65abec4 Fix for Wallet confirmed states (#1782)
* Fix for Wallet confirmed states

* Only process if we do not have the header
2020-08-08 08:12:30 -05:00
Ben Carman
ce2601002e Have wallet shuffle inputs and outputs (#1721) 2020-08-05 16:03:27 -05:00
rorp
628fd80d30 Implement Wallet.listTransactions() (#1744) 2020-08-05 07:21:12 -07:00
Ben Carman
e542a00d19 Callbacks to appconfig (#1740)
* Node Callbacks to appconfig

* Move WalletCallbacks to WalletAppConfig

* Update Docs
2020-08-04 12:27:21 -05:00
Ben Carman
26d5a09532 Move Node type of out Wallet API (#1708)
* Move Node type of out wallet api

* Remove extensions, add scaladocs
2020-07-29 05:57:48 -05:00
Ben Carman
73628e7e1f Add ProcessBlockTest (#1674) 2020-07-27 14:14:24 -05:00
Ben Carman
4abcf3f321 Create HDWalletApi (#1693)
* Create HDWalletApi

* Rename createWallet to createHDWallet, add listDefaultAccountUtxos function

* Change return types to HDWalletApi

* Fix warning
2020-07-27 14:06:48 -05:00
Ben Carman
5e8a049fd2 Unreserve spent utxos (#1676) 2020-07-19 10:40:09 -05:00
Ben Carman
d29a04c480 Let Wallet find utxos by state (#1668) 2020-07-15 13:07:55 -05:00
Ben Carman
38489e146a Random Coin Selection (#1663) 2020-07-14 10:12:05 -05:00
Ben Carman
ecd3449100 Address Tagging Attempt 2 (#1320)
* Address & UTXO tagging

* Fix docs

* Remove useless function, improve docs

* Fix rebase errors

* Rebase fixes

* Fix docs

* Fix small test errors

* Fix Postgres migration

* Fix postgres
2020-07-08 14:38:39 -05:00
Ben Carman
40d0c7337d Temporary fix for update all on CRUDAutoInc Tables (#1596)
* Temporary fix for update all on CRUDAutoInc Tables

* Add comment
2020-06-20 08:46:59 -05:00
Scala Steward
97a4886c6e Update scalafmt-core to 2.6.0 (#1579)
* Update scalafmt-core to 2.6.0

* Upgrade scalafmt, fix all formatting errors

* Scalafmt on 2.12

* fix test

* More fixes

Co-authored-by: Ben Carman <benthecarman@live.com>
2020-06-19 10:38:12 -05:00
Ben Carman
5319b4c927 Wallet callbacks (#1543)
* Wallet Callbacks

* Move noop to Callback

* Add docs

* Add tests for wallet callbacks

* Use BitcoinAddress instead of AddressDb, improve tests
2020-06-17 17:21:19 -05:00
Chris Stewart
ba0f38ccf6 2020 06 14 wallet root accounts (#1556)
* Implement unit test cases for initializing a wallet twice, and failing with an exception when we initialize a wallet with a different key-manager

* Use bip39PasswordOpt cached in the WalletUnitTest suite

* Turn off logging

* Run scalafmt

* Turn log level back to WARN

* Run scalafmt
2020-06-17 15:11:42 -05:00
Chris Stewart
1753f53fd1 Make it so wallet fixtures take a bip39PasswordOpt as a paramter (#1555)
* Make it so wallet fixtures take a bip39PasswordOpt as a paramter

* Fix doc
2020-06-16 10:45:41 -05:00
Nadav Kohen
027bd39474 Fixed all walletTest threads not being closed (#1552) 2020-06-15 10:50:53 -05:00
Chris Stewart
c54d6dcbdc 2020 06 09 addr invoice string factory (#1538)
* Implement StringFactory with Address and LnInvoice

* Address code review

* Run scalafmt
2020-06-11 10:44:12 -05:00
Ben Carman
1dd6025b9d Testkit wallet with bitcoind uses bitcoind as api (#1499)
* Testkit wallet with bitcoind uses bitcoind as api

* Fix docs
2020-06-03 14:05:22 -05:00
rorp
823a392e98 PostgreSQL support (#1315) 2020-06-02 16:37:48 -05:00
Ben Carman
c10d5bc89b Fee Rate Api Support (#1470)
* Fee Rate Api Support

* FeeRateProvider Tests

* Rename folder

* Fix docs
2020-05-29 13:01:20 -05:00
Chris Stewart
c6aae0dbf9 Create a simplified version of the WalletApi.unmarkUTXOsAsReserved() that just takes in a tx and scans outpoints if they are in our wallet, also move the mark/unmark methods out of Wallet.scala and into UtxoHandling.scala (#1463) 2020-05-25 14:45:38 -05:00
Ben Carman
c4c660158e Add ability to unreserve utxos (#1458) 2020-05-22 06:29:11 -05:00
Ben Carman
99db9062ac Add option to correctly use logback config (#1398)
* Logger now uses correct parent logger for all, names appenders correctly

* Set logback to true for tests, rename file

* Fix docs
2020-05-20 15:46:24 -05:00
Ben Carman
51fcb793be Add ability to make OP_RETURN commitments (#1417) 2020-05-18 14:27:05 -05:00
Ben Carman
e44a08bd39 Wallet Send with coin selection algorithim (#1409) 2020-05-18 11:26:39 -05:00
Ben Carman
867e2a9755 Add list unused addresses call (#1408) 2020-05-12 09:29:02 -05:00
Ben Carman
711f5cb99c Add listFundedAddresses call (#1407) 2020-05-12 07:33:22 -05:00
Ben Carman
721b28aefd Add listSpentAddresses call (#1406) 2020-05-12 07:24:52 -05:00
Ben Carman
c571585b3b Wallet send from outpoints (#1405) 2020-05-12 07:24:36 -05:00
Nadav Kohen
a503441619 Crypto Project Refactor (#1380)
* Added crypto project and decoupled BitcoinSLogger from NetworkElement

Decoupled BitcoinSLogger from Factory

Moved NetworkElement into crypto project

Moved Factory and BitcoinSUtil (renamed to BytesUtil) to crypto project

Moved MaskedToString to crypto project

Added BytesUtil to imports and cleaned up CryptoUtil.recoverPoint

Moved the rest of crypto stuff to the crypto project

Moved crypto tests to crypto-test project

* Added documentation for crypto project
2020-04-30 12:34:53 -05:00
Ben Carman
6d7685b76e Add wallet creation time for rescans (#1353)
* Add wallet creation time for rescans

* Fix docs

* Clean up and add test

* Remove account bday

* Fix compile issue and docs

* Add more chain handler tests

* Use Instant over Long, add docs

* Fix docs
2020-04-29 09:49:41 -05:00
Ben Carman
4234561db9 Remove Unlocked vs Locked Wallet distinction (#1379)
* Remove Unlocked vs Locked Wallet distinction

* Add back TODO comment
2020-04-28 13:58:14 -05:00
Chris Stewart
4c3f6f8e67 2020 04 08 multi db dao (#1355)
* Implement Slick's multi database support via our DAOs

WIP2 work on chain project

WIP make JdbcProfieComponent so we have a common JdbcProfile

WIP2

Clean up a few things

WIP got CRUDAutoInc compiling

Get closer to everything compiling in the chain project

Begin moving database code into JdbcProfileComponent

WIP source compiling

Get dbCommonsTest/ test cases passing

Get rest of tests compiling and passing

* Rebase onto master, fix conflicts

* Run scalafmt

* Run scalafmt again

* Cleanup a few compiler warnings by adding type annotations to thigns

* Explicitly cast tables where they are needed in the wallet project where we need TableQuery for foreignkey / joins in slick

* Redo definition of 'table' to use the profile.api.TableQuery in scope

* Rename all *Table.scala files to *Db.scala

* First crack at address ben's code review

* Fix docs compile issue

* Fixed casting (#30)

* Fixed AppConfig casting

* Fixed TableQuery casting in DbManagement subtypes

* Fixed casts in DAOs relating to internal types

* Address nadav's code review, move JdbcProfileComponent into it's own file

* Remove unused listTables() method

Co-authored-by: Nadav Kohen <nadavk25@gmail.com>
2020-04-25 11:28:58 -05:00
Ben Carman
3f14add2d0 Ignore immature coinbase outputs when funding a transaction (#1365)
* Ignore immature coinbase outputs when funding a transaction

* Create Consensus.coinbaseMaturity
2020-04-24 18:04:42 -05:00
Ben Carman
3d26ad8f34 Rescan from account (#1348) 2020-04-24 09:37:12 -05:00
Ben Carman
c696a7306b Require that addresses are the correct network when sending (#1332)
* Require that addresses are the correct network when sending

* Compare network bytes so typed addresses work

* Use isSameNetworkBytes
2020-04-13 13:58:41 -05:00
Chris Stewart
3f71216eb3 Bump number of addresses generated in negative test case for AddressHandling where we expect a illegal state exception to be thrown (#1333) 2020-04-11 19:10:35 -05:00
Ben Carman
aa8a2e14a6 Move Json Reader, Writers, and Serializers to App Commons (#1328)
* Move Json Reader, Writers, and Serializers to App Commons

* Move app-commons outside of app/
2020-04-10 14:33:37 -05:00
Chris Stewart
76a3edd0af 2020 04 02 get new address queue (#1299)
* WIP: Build queue for getnewaddress

* Implement background thread to process requests to getNewAddress so that we fix issue 1009 with it's async issues

* Run scalafmt

* Replace the mutable.ArrayBuffer in AddressHandling with ConcurrentLinkedQueue

* Put FundTransactionhandling.fundRawTransactionInternal's call to getNewChangeAddress() into the for expression so we don't generate an address when the funding of the transaction fails when selecting utxos

* Move thread initialization out of method

* Switch to using ArrayBlockingQueue, which will block thread with .take() so we don't need to loop constantly, add 'addressQueueSize' and 'addressQueueTimeout' configurations

* Update wallet-test/src/test/scala/org/bitcoins/wallet/AddressHandlingTest.scala

Co-Authored-By: rorp <rorp@users.noreply.github.com>

* Add error handling to return the failed future if the queue is full, add a unit test for when the queue is full

* Run scalafmt

* Rebase

* Add scaladoc for throwing an exception

* Run scalafmt again

Co-authored-by: rorp <rorp@users.noreply.github.com>
2020-04-10 14:19:39 -05:00
Ben Carman
891e6da11d GetAddress Wallet API call (#1287)
* GetAddress Wallet API call

* Add ability to get unfunded address

* Generate new address if no unfunded addresses

* Add test

* Add wallet get address documentation
2020-04-09 15:33:11 -05:00
Ben Carman
ea20f394e8 Add wallet pay to many (#1317)
* Add wallet pay to many

* Create sendToOutputs

* Add sendToAddresses Unit Test

* Send to addresses can reserve utxos
2020-04-09 15:00:26 -05:00
Ben Carman
b3250dcfba Update TxoState for transactions after they've been confirmed (#1178)
* Add onBlockHeadersReceived NodeCallback

* Add requiredConfirmations config option

* Update TxoState for transactions after they've been confirmed

* Response to review

* Fix compile issue
2020-04-08 16:51:17 -05:00
Ben Carman
29eb6c2e05 Wallet Transaction Tracking (#1197)
* Incoming and Outgoing Transaction Tables

* Remove script sigs for witness txs

* Create parent tx_table for incoming and outgoing txs

* Response to review

* Use isCloseEnough

* Fix test

* Fix rebase error

* Test that tx is tracking all sats correctly
2020-04-02 06:55:09 -05:00
Ben Carman
ff8df7065c Wallet integration for nested segwit v0 spending (#1272)
* Wallet integration for nested-segwit spending

* Move parameter
2020-03-28 09:40:05 -05:00
Chris Stewart
cd84e3483d 2020 03 06 wallet rescan test (#1218)
* Add wallet rescan tests

* Create FilterSync, which gives us an API inside of the chain project to sync filters with

Add another unit test to filter sync

Add more unit tests for ChainSync and FilterSync

Clean up some docs, remove some extra lines of code

Run scalafmt

Add filter-sync.md

Cleanup some nits

Add more information of how FilterSync.syncFilters() works

Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project

Run scalafmt, hide imports in filter-sync.md so code appears cleaner

Move implicits out of invisible block as it seems to cause errors

Make it so FilterSync processes filters in batches rather than fetching them all at once

Fix compile error

* WIP bitcoind implement ChainQueryApi

* rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets

* Fix rebase problem

* Implement getFiltersBetweenHeight() with ben's solution

* WIP Start implementing NodeApi against bitcoind

* Actually inject the bitcoind backed nodeApi into our fixture

* Get first rescan test working for rescanning the entire blockchain

* Implement test case for rescanning from a specific height

* Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18

* Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses

* Add another log to try to debug CI

* Address code review

* Fix wrong ordering of deletion of tables in clearUtxosAndAddresses()

* reset logging level
2020-03-13 16:11:02 -05:00
Ben Carman
bc87bc64eb TxoState Life Cycle Tests 2020-02-28 17:18:28 -06:00
Ben Carman
cf3dd113b5 Reserved TxoState (#1111)
* Reserved TxoState

* Allow fund transaction handling to mark utxos as Reserved

* Add test, fix overloading default arg issue
2020-02-13 06:51:50 -06:00
Chris Stewart
f8a21201ac Run test:scalafmt, and enforce it on CI (#1023)
* Run test:scalafmt, and enforce it on CI

* Run test:scalafmt
2020-01-23 14:46:57 -06:00
Chris Stewart
b8c59b4c93 2020 1 11 accounts (#1022)
* Create 'fundRawTransaction' and 'fundRawTransactionInternal'

* Move TransactionTestUtil to testkit, begin writing tests for fundrawtransaction

* Add FundTransactionTests

* Move things to 'FundTransactionHandling' trait

* Start segregating things by HDAccount in the wallet, this gives us the ability to query for balances based on account, fund a transaction based on an account, generate a new address based on an account etc. All old api calls are now based on the default account, i.e. getBalance() returns the balance for account 0, fundRawTransaction funds the transaction from account 0 etc.

* Fix compile issue with 2.11.x

* Bump address generation Thread.sleep() to 500ms

* Address ben code review

* Address code review
2020-01-21 14:20:58 -06:00
Chris Stewart
54b2fa814e 2020 1 04 fund raw tx (#1010)
* Create 'fundRawTransaction' and 'fundRawTransactionInternal'

* Move TransactionTestUtil to testkit, begin writing tests for fundrawtransaction

* Add FundTransactionTests

* Move things to 'FundTransactionHandling' trait

* Address code review

* Run scalafmt
2020-01-13 13:45:33 -06:00
Chris Stewart
3bedcb20e0 Add more expressive txo states for the wallet (#1001)
* Add more expressive txo states for the wallet

* Rename 'PendingReceived/PendingSpent' -> 'UnconfirmedReceived/UnconfirmedSent'

* Address code review

* Fix posInt implementation to work with older scala versions

* Rename to PendingConfirmationReceived/PendingConfirmationSpent
2020-01-05 18:42:20 -06:00
Chris Stewart
55f168f920 Implement abililty to use BIP39 password. This means this password ne… (#990)
* Implement abililty to use BIP39 password. This means this password needs to be password through our various projects to be able to correctly generate the key that controls the wallet. This also renames 'CreateKeyManagerApi' -> 'BIP39CreateKeymanagerApi' as the bip39 password is needed when creating the KeyManager

* Add bip39 password to BIP39KeyManager.fromParams(), fix mdocs

* Fix bug in unit test were whe weren't specifying password

* Fix rebase issues
2020-01-03 11:03:45 -06:00
rorp
93c02f02d0 Automated wallet recovery (#985)
* Automated wallet recovery

* responded to the PR comments

* some more changes

* fix docs

* cleanup
2020-01-02 13:18:41 -06:00
Chris Stewart
c3fb7c9a67 This creates a subtype BIP39KeyManager and moves all existing KeyMana… (#988)
* This creates a subtype BIP39KeyManager and moves all existing KeyManager functionality to that subtype. We leave a empty 'KeyManager' trait for now

* Fix website docs
2019-12-27 10:09:04 -06:00
Chris Stewart
24ff908240 Move initialization of wallet entropy into the key manager (#966)
* Move initialization of wallet entropy into the key manager

* Enable key-manager in ci, add coverage minimum

* Add coverage minimum for key manager, add KeyManagerParams, start adding KeyManager test harness

* Remove success cases on ADTs, start using Either

* fix import

* Get test coverage to 90%

* Add documentation for key-manager.md, add side bar for key-manager

* Fix wallet.md

* Make seedPath default to baseDatadir, so we don't have a unique seed for every network. Add more documentation

* Address code review

* Address code review from Ben

* Add missing import

* Add more tests for coverage
2019-12-26 19:12:08 -06:00
rorp
72ead16e5c Move rescan logic from node to wallet (#974)
* Move rescan logic from node to wallet
2019-12-23 08:57:55 -08:00
rorp
2c53a39fd1 Compute confirmations dynamically (#938)
* Compute confirmations dynamically
2019-12-14 12:06:22 -08:00
rorp
4acd1fbba9 Add a reference to ChainQueryApi to the wallet (#936)
* Add reference to ChainQueryApi to wallet
2019-12-10 14:22:33 -08:00
rorp
af70acb03b ChainQueryApi (#926)
* ChainQueryApi

* cleanup
2019-12-09 09:39:35 -06:00
rorp
cbd038b514 Neutrino Wallet: OnCompactFilter handler (#905)
* Neutrino Wallet: OnCompactFilter handler
2019-12-06 13:10:33 -08:00
Chris Stewart
dd4787f2af 2019 11 30 key manager (#904)
* Segregate key manager from wallet

* More consistent naming in build.sbt, run scalafmt

* Add test case making sure we don't overwrite a mnemonic seed

* Add key-manager.md

* Fix compile issues with older scala versions

* Address code review
2019-12-04 07:43:29 -06:00
rorp
74a44d1c4d Initial BIP157 support (#695)
* WIP: Initial BIP157 support

* store block hash and hetgh along with its compact filter header

* download and parse block filters

* getcfilters/cfilter

* cfilter table

* rescan full filter chain

* improved rescan performance

* optimize compact headers download

* addressed the PR comments

* split SVP and Neutrino node implementations

* configurable filter batch sizes

* initial filter sync

* addressed comments

* chage filter table's primary key

* fix Golomb filter deserialization

* batch database inserts

* neutrino node test

* fixed node test

* addressed the PR comments

* serializers tests

* cleanup

* fix compilation errors

* fix unit tests

* increase test coverage

* enable NeutrinoNodeTest

* make scalafmt happy

* don't cache experimental binaries

* inclease test coverage

* fix unit tests

* more granular CI tests

* disable NeutrinoNodeTest

* refactor tests

* addressed comments

* test coveage

* fix formatting

* responded to the comments

* some more changes

* fix the build

* test coverage

* revert testnet3 config parameter

* minor changes

* cleanup
2019-09-25 13:18:51 -05:00
Chris Stewart
67ead0bc7e 2019 09 05 scalafmt update (#735)
* Update scalafmt to 2.0.1

* Scalafmt

In this commit we
1) Reconfigure the .scalafmt.conf file
2) Add a scalafmt binary (same version as specified in conf file)
3) Use said binary on CI to check that everything is formatted correctly

* Ran scalafmt

* Update scalafmt ignore

* Add scalafmt plugin to project/plugins.sbt, run it

* run scalafmt
2019-09-05 20:02:58 -05:00
Torkel Rogstad
e0a5646258 Deprecate generate (#728) 2019-09-01 07:46:38 -05:00
Chris Stewart
c934d8efc2 Address issue #673, also refactor fixture stuff to be in companion ob… (#676)
* Address issue #673, also refactor fixture stuff to be in companion objects rather than traits so they can be re-used without extending the trait itself. This allows for more modularity with our fixtures

* fix bugs in unit tests, address #675 to make MerkleBufferTests faster

* Refactor test timeout in UpdateBloomFitlerTest

* fix callback logic bug on matching unrelated txs/addresses in UpdateBloomFilterTest, re-order broadcasting of things to avoid async bugs hopefully

* Make  a def in NodeUnitTest, this keeps tests in the same suite from using the same config. This allows us to write multiple tests per suite, instead of just one. This also adds implicit parameters to our fixture constructors/destructors to properly create and destroy this config. The long term goal here also should be getting rid of config.initialize() we are calling everywhere as this is a anti-pattern, fixtures should take care of construction and destruction of things

* Broadcast tx earlier in test case for UpdateBloomFilterTest

* Rework NodeWithWalletTest to use the new fixtures we have in the node project, now use SpvNodeFundedWalletBitcoind
2019-08-06 11:49:17 -05:00
Torkel Rogstad
46280c9e59 Add functionality for updating SPV node bloom filter (#585)
* Add functionality for updating SPV node bloom filter

* Add SPV node shutdown to exit hook

* Clean up traits/case classes

* Change fixture in WalletBloomTest

* Fix logging bug in TransactionProcessing

* Add MerkleBuffers

In this commit we add MerkleBuffers, which is an object
that lets us aggreagate merkleblocks with their corresponding
transactions before sending them out. This is global, mutable
state (bad!) but it's a working solution for now;

* Use TestAsyncUtil

* Add MerkleBuffers test

* Send getdata if receiving single header

* Change awaitSync to use block count

* Fix UpdateBloomFilterTest

* Add more logging of chain/headers validation

* Send getdata for all blocks

* Nits: Scaladocs, comments toString
2019-08-02 09:22:20 -05:00
Torkel Rogstad
a76f61f97c Add configurable logging to data directory (#640)
* Add logging to data directory

In this commit we add the ability for the node, chain
and wallet projects (+ the server) to log to the users
data directory instead of whatever directory the
binaries was launched from. This is inherently a bit
more complicated than our previous setup, because
we need to read the user directory before we can create
loggers. As a result of this, some files/methods were
moved around, so the relevant app config could be
found in scope.

We also  introduce several logging categories that can be
tuned individually through user configuration. These logggers
are exposed both as traits that give a field `logger`, or
as methods that return the required logger.

* Add datadir configuration to AppConfig

In this commit we add support for AppConfig to pick up
the data directory configuration file. We also add
a section to the contributing guide on how to tune
logging levels.

* Pass data directories explicitly for configuration
2019-08-01 06:01:56 -05:00
Torkel Rogstad
5194fb1f8b Merge pull request #594 from torkelrogstad/2019-07-10-bloom
Add all wallet outpoints to bloom filter
2019-07-16 17:07:21 +02:00
Torkel Rogstad
45a3e93a3d Add all wallet outpoints to bloom filter
This is a workaround for Bloom filters never being
updated for SegWit. See comment in
LockedWallet#getBloomFilter for more context.
2019-07-16 14:18:29 +02:00
Torkel Rogstad
4bdc7966d0 Add notion of address types in wallet
In this commit we
1) Add the notion of address types in the wallet, and remove
    the emphasis on account types. Users now just request
    a segwit/nested-segwit/legacy address, and we take care
    of querying for the correct account
2) Fix a bug where a wallet could only get addresses for
    its default address type. This was a pretty minor bug,
    where a few values that should have been dynamic was
    hard coded.
2019-07-16 12:36:38 +02:00
Torkel Rogstad
30e6d7030f Somewhat dirty standalone server and CLI binary (#558)
* PoC bitcoin-s-cli

* Add CLI, Server sbt projects, remove Ammonite

In this commit we set up sbt configuration for
CLI, Server (in-work-name) and corresponding
test projects.

We also remove Ammonite shell from sbt, as that
isn't really being used. bloop console offers
the same functionality way more ergonimic.

* Move BitcoinSAppConfig into new server project

Server project depends on node, chain wand wallet
so this is a good time for introducing this class
into main sources. We also introduce
BitcoinSTestAppConfig in testkit, to replace the
functionality in BitcoinSAppConfig related to
tests.

* Type chain in blockchainresult

* MVP server setup for node, chain and wallet

* Extremely dirty CLI for interacting with server

* initial attempt at mimicking Bitcoin Core API

* WalletStorage: add method for checking for seed existance

* Check for seed existance on wallet startup

* Fix bug where MnemonicNotFound was not an error

* Segregate confirmed and unconfirmed balance methods

* Add error handling, improve formatting of CLI output

* Tweak build

Bump Sttp version, downgrade to uPickle 2.11 compat,
skip publish in cli-test and server-test

* Add CLI, server and picklers to root project
2019-07-10 06:33:17 -05:00
Torkel Rogstad
a71aecec52 Process outgoing transactions (#555)
* Split wallet functionality into multiple traits

In this commit we refactor LockedWallet into multiple traits
that provide functionality related to a subset of total wallet
functionality. This has the benefit of making it clear which
methods are helper methods that are only intended to be used
in a very specific setting, and which methods are part of the
internal wallet API that other parts of the wallet can use.

* Rework TransactionOutput and TransactionOutPoint to case classes

* Add extension methods for flattening lists of assertions

* Segregate confirmed and unconfirmed balance methods

* Add test for FutureUtil.sequentially

* Add trace logging of balance fetching

* Process outgoing TXOs

Move TX processing into separate trait, add internal API method

Unify DB representation of TXOs

    Prior to this commit we stored TXO information
    across diferent tables, with joins and tuples
    needed a bunch of places to keep track of
    everything we needed. In this commit we unify
    the tables, leaving us with only one table for
    TXOs.
2019-07-09 06:25:24 -05:00
Torkel Rogstad
6caaf1938a Tune down excessive logging 2019-07-02 11:22:18 +02:00
Torkel Rogstad
c2349c44c2 Rebase cleanup 2019-06-27 15:34:41 +02:00
Torkel Rogstad
6ffd505946 Add sampleSome method to scalacheck.Gen 2019-06-27 15:29:21 +02:00
Torkel Rogstad
c884b9c048 Fix bad comparison in ProcessTransactionTest 2019-06-27 15:28:08 +02:00
Torkel Rogstad
86517c0a6d Replace incoming/outgoing TXs with TXOs
Rework the wallet DB model so that instead
of dealing with incoming and outgoing transactions
we deal with incoming and outgoing transaction
outputs.
2019-06-27 15:28:08 +02:00
Torkel Rogstad
06b8375c81 Add ProcessTransactionTest 2019-06-27 15:28:08 +02:00
Torkel Rogstad
5a04260461 Update wallet integration tests 2019-06-27 15:28:08 +02:00
Torkel Rogstad
5513344373 Add findTx to IncomingTxDAO 2019-06-27 15:28:08 +02:00
Torkel Rogstad
e2278f7606 Get UTXO DB unit tests passing 2019-06-27 15:28:08 +02:00
Torkel Rogstad
88e318d485 Move WalletDAOs to testkit 2019-06-27 15:28:08 +02:00
Torkel Rogstad
b630cfbf5c Generalize DAO fixture for wallet 2019-06-27 15:28:08 +02:00
Torkel Rogstad
bdd0468383 Add tables for incoming and outgoing transactions 2019-06-27 15:28:08 +02:00
Torkel Rogstad
e4e9ec9db9 Add SPK field to AddressTable 2019-06-27 15:23:23 +02:00
Torkel Rogstad
db1987b36a Merge pull request #546 from torkelrogstad/2019-06-21-aes-cfb
Use key based encryption in AesCrypt
2019-06-27 12:16:53 +02:00
Torkel Rogstad
000fe1e2ef Merge pull request #431 from nkohen/coin-selection
Basic Coin Selection
2019-06-26 13:13:27 +02:00
Torkel Rogstad
505d3cbaa1 Fix Scala 2.11 complaints 2019-06-25 11:58:18 +02:00
Torkel Rogstad
334d410220 Fix CoinSelectorTest after rebase 2019-06-24 17:45:31 +02:00
nkohen
269d3c8f21 Implemented simple greedy coin selection
Added tests for current coin selection algorithm options

Responded to code review

Implemented fee estimation calculation in coin selection

Updated tests after rebase

Fixed CoinSelectorTest

Fixed CoinSelectorTest after rebase
2019-06-24 17:41:43 +02:00
Torkel Rogstad
f93b8cadbe Update wallet project to work new AES changes 2019-06-24 17:25:11 +02:00
Torkel Rogstad
4eb5aacae6 Remove ??? from GetAddresses 2019-06-20 14:53:07 +02:00
Torkel Rogstad
0621088f41 Refactor TrezorAddressTest 2019-06-20 14:52:13 +02:00
Torkel Rogstad
5819f4016e Trezor test vectors
In this commit we add:

1) A program for dumping a bunch of address
   information from a running Trezor
2) A test in the wallet module that verifies
   that our addresses line up with the ones
   Trezor generates
2019-06-20 14:29:50 +02:00
Torkel Rogstad
435aef7b0b Pass bloom filter to SPV node (#514)
* Add getAddressInfo to Wallet API

* Add Bloom filter and event callbacks to SPV node

* Move wallet test trait to testkit

* Test for connecting SPV node with wallet

* Tune logging in SPV node

* Clean up BloomFilter

* Proper toString in inventory and inventorymessage

* Actually pass in callbacks to SPV node

* Fix rebase screwup and partially respond to code review

* Default to file based databases in tests

This is due to issues with deadlocks with
in-memory based databases. We need DBs
to stay alive between connections, but
not across tests.

In DB intensive chain validation tests we
use in-memory databases instead. This
seems like a reasonable tradeoff between
simplicity and speed.

* Make SpvNodeCallbacks contain sequence of functions within

* Make Bloom filter non-optional in SpvNode
2019-06-17 06:48:26 -05:00
Torkel Rogstad
8d141ac450 Generate bloom filters from wallet (#501)
* Add findAll() to CRUD

* Add getAddressInfo to Wallet API

* Insert pubkey into bloomfilter also inserts hash

In this commit we also simplify parts of
BloomFilterTest, and move what used to be in
BloomFilterSpec into BloomFilterTest.

* Add bloom filter generation to wallet

* Add bloom false positive rate to wallet config

* Add logging to SQL errors

* Fix a bug where HDPath.next returned the wrong path

* Add FutureUtil.sequentially

* Make bloom filter size calculation more explicit

* Return Vector of pubkeys instead of Seq

* Verbose handling of address fetching in test
2019-06-12 09:29:20 -05:00
Torkel Rogstad
5dbfefc7ed Respond to code review on legacy address generation 2019-06-06 15:10:20 +02:00
Torkel Rogstad
42c161d3e3 Fix bug when overriding configurations
This commit fixes (and adds a test for) a bug
where overriding a value in our configuration
reset the rest of the configuration to the
default values.
2019-06-05 18:37:38 +02:00
Torkel Rogstad
6fa5ae85d8 Add legacy addresses to wallet
In this commit we add the ability to generate
legacy addresses in the wallet. We also make
the default accound kind (legacy, segwit,
nested segwit) configurable.
2019-06-05 18:36:57 +02:00
Torkel Rogstad
3c4157a698 Multi module configuration (#494)
* Replace AppConfig with more specific types

* Rework database configuration

In this commit we:

1) Introduce new database settings for all submodules,
and at the same time remove the singular datbase setting
for the entire project
2) Introduce a new method .initialize on all configs
that is responsible for creating needed directories,
databases and files
3) Introduce a new class BitcoinSAppConfig that wraps
configs for all three other subproject config. We
also provide implicit conversions that enable this
super-config to be passed in wherever a specialized
config is required.
4) Add more tests for our configuration setup.

* Add Ammonite to Docs deps
2019-06-05 10:48:15 -05:00
Chris Stewart
a3dd824e57 Node (#490)
* WIP: 2018 12 22 node project (#280)

* Add files from old spv node project

src compiling

test files compiling

ran scalafmt

Fix serializer tests

Get non networking test cases to work

WIP: Debug peermessagehandler

Update CRUD, remove all of the Actor craziness. Add DbManagement trait and unit test db

WIP: Rewroking PeerMessageHandler, create Peer, DataMessageHandler, PeerHandler

Reworking Client to handle all tcp messages and message alignment for bitcoin p2p messages

* Wip: Node refactor

* Create node test project, move all node tests into that project and move all generators for the node project into testkit

* Rework ClientTest to use testkit, start minimizing akka usage, implement connect(),isConnected(), disconnect(), isDisconnected() in PeerMessageReceiver

* Create Peer, PeerHandler, PeerMessageSender and PeerMessageReceiver

* update readme about status of node project (#359)

* Add flyway plugin to manage database schemas (#361)

* Add flyway plugin to manage database schemas

* Switch database driver to sqlite3 to be more portable, rework configs for sqlite3

* Set up sqlite database directories and files if they are not already created

* Add torkel's review

* Add chain, wallet, db-commons projects (#367)

* Add chain, wallet, db-commons projects

* Rework db creation logic if they db does not exist

* Add config logging to try to debug travis ci

* Pass explicit class loader for db config

* Remove duplicate call to dbConfig

* Make DbConfig.dbConfig a lazy val

* Remove noisy log

* Add scaladoc to DbConfig

* Switch dbConfig readme paragraphs

* Fix compile issues introduced during rebase onto master with rpc changes (#394)

* WIP: 2019 03 12 tip validation (#378)

* Implement blockchain handling data structures

Add TipValidation happy path

Add more test cases for TipValidation.checkNewTip for badPrevBlockHash and badPOW

Add overflow check, fix endianness bug for checking proof of work

Add pow transition check, refactor difficultyChangeInterval into chain params, add more tests

fix a few nits

Fix compile error, clean up unused import

Remove redundant files from node project

* Implement GetNextWorkRequrired/CalculateNextWorkRequired, move BlockHeaderDAOTest cases into chain project

* Add full POW change check in TipValidation, address code review nits

* Configure logging in chainTest, turn logging OFF in other test projects

* Address code review pt2

* Add coverage minimum for chain project (#398)

* Add coverage minimum for chain project

* Add first Blockchain.connectTip() unit test, switch to a in memory sqlite database for unit tests, starting using fixtures for BlockHeaderDAO in unit tests

* Add tests for ChainHandler.processNewHeader(), ChainHandler.getHeader(), Blockchain.connectTip(). Refactor redundant configurations being passed around excessivly

* Address code review, fix a flaky test in ClientTest.scala

* Test Fixtures (#403)

* Working test fixtures

* Removed ChainTestFixture trait in main code

* Composing Fixtures (#413)

* Downloaded over 9000 mainnet BlockHeaders into a json file

* Added new fixture with populated blockHeaderDAO

* Split writing to db into batches

* Rebased

* Simplified fixtures with makeFixture abstraction

* Added util functions for composing builders

* Add integration test between bitcoind <-> zmq <-> bitcoin-s-chain project. Test that we can relay a header from bitcoind over zmq into the bitcoin-s chain project. Redo ZmqConfig to use InetSocketAddress

* Address code review

* wip

* A compiling withBitcoindZmqChainHandler fixture

* Tests passing!

* Made blockHeaderDAO private

* Got 9000 new block headers from 562375 to 571375

* Added offset to populated blockHeaderDAO fixture

* Added scaladocs to fixture things

* Initial wallet: import UTXO and spend it (#391)

* Updates ExtKeyVersion with fromChainParams method

* Add equals to Address

* Update BIP44 classes

* Add ScriptType

* Initial work on wallet support

* Add foreign keys pragma for SQLite

* Add UTXO models and DAO

* Add addres P2WPKH generation and WIP for addUTXO

* Add logging config for wallet

* Add change address generation, proper-ish addUtxo and sendToAddress

* Address code review on #391

* Add empty AES passphrase invariant

* Add poor mans test fixtures

* Add listUtxos, listAddresses and getBalance to wallet API

* Use fixtures from chain project

* Fix CI test failures

* Fix broken up package path

* Updates bloop config for new projects (#424)

* Multi fixture file (#419)

* Created FixtureTag and ChainFixture

Used ChainFixture in BitcoinPowTest

Added implicit conversions for nice syntactic sugar

* Added documentation for multi-fixture

* Made defaultTag a val

* add a logback-test.xml to the wallet project (#433)

* Introduce AppConfig that combines ChainParams and DbConfig (#432)

* 2019 04 23 app config per project db config per project (#434)

* Add DB conf file resolution that works across projects

* Create applicatoin configurations for specific projects, rework DbConfig structures for individual projects. Force network to be mixed into DbConfig rather than DbConfig to be mixed into the network

* Add ammonite to db-commons, remove noisy logs

* remove mixin for DbConfig that required a NetworkDb. Now networkDb is just a field on 'DbConfig', this simplifies things downstream type wise when interacting with the projects AppConfig. This commit also removes a parameter from AppConfig, now only a DbConfig needs to be passed in, and we can derive the network and chain params from the DbConfig. The only exemption is UnitTestDbConfig as it is sometimes handy to specify a different network (i.e. mainnet) when testing

* Turn DbConfig objects to case objects, wrap those case objects in their parent type companion object

* remove cast in Wallet.scala

* Add EnhancedEither class for 2.11 compat (#437)

Add implicit conversion from Either to
2.11-compatible Either-wrapper. Also
remove trailing comma in WalletTestUtil
that breaks 2.11 build.

* Fix CI tests hanging (#438)

* Execute wallet tests sequentially to avoid SQLite deadlocks

* Refactor logback config to reduce duplication

* Use in-memory SQLite DB for unit tests

* Debug prints for DatabaseConfig.forConfig

* Fork JVMs in test to ensure proper in-memory DBs

* Pass in Akka config to Eclair tests, avoid cluttering Akka log output

* Don't fork JVM on node tests'

* Move things out of ChainUnitTest (#448)

* Move things out of ChainUnitTest

* Remove printlns

* 2019 04 29 client test (#449)

* Bump timeout on connect to node test

* Change from isConnected -> isInitialized to avoid the error trying to disconnect before we are fully initialized

* Wrote tests for POW difficulty change calculation and header processing (#429)

Fixed BitcoinPowTest

Rebased onto AppConfig code

Rewrote ChainHandler integration test

Made chain handler test synchronous

Fixed a couple test bugs

Implmented a more efficient getAncestorByHeight

Fixed ChainHandler integration test by using the correct starting conditions

Responded to code review

Responded to more code review

Deleted redundant Pow test

Made BlockHeaderDAO.getAncestorAtHeight use a List for its loop to improve performance

* WIP: Create ChainSync, BitcoindChainHandlerViaRpc, add simple ChainSyncTes… (#450)

* Create ChainSync, BitcoindChainHandlerViaRpc, add simple ChainSyncTest to sync one block from a external bitcoind instance via rpc

* Add check for having the best block hash in our chain state already

* Fix prev block hash to be empty hash if genesis block header

* BlockchainBuilder (#439)

* First commit for implementing a BlockchainBuilder

* use Builder rather than ReusableBuilder to be compatible with scala 2.11.x

* Decouple Blockchain & BlockHeaderDAO

* Rebase onto node, incorporate changes in #429

* Add more comments

* Reverse order of headers in builder

* rebase onot node branch, refactor apis

* DB: Add utility method for listing tables in a DB (#447)

* Node rebase (#458)

* Implement BIP32 path diffing

* Rebase node onto newest HD changes in master

* Fix 2.11 compile errors

* 2019 05 01 wallet ammonite scripts pt2 (#452)

* wip -- not finding testkit in doc worksheet

Wip -- classdef not found for create-wallet.sc

zmq bug

Clean up some logs

nest zmq start in bitcoindF

update jeromq to 0.5.2-SNAPSHOT to get rid of annoying log to stdout

Rebase onto node branch with new configs

Successfully running ammonite script create-wallet.sc

2019 05 01 wallet ammonite scripts pt2 (#25)

* Refactor Ammonite dep

* Add basic error handling in AmmoniteBridge

* Add very basic README for doc project

Fix compile issues after rebasing onto master

Add code to sync our wallet code with bitcoind after creating a tx

* refactor ZMQSubscriber to _hopefully_ avoid hanging when we call context.term(). We do this by closing the socket before calling context.term() and using socket.setLinger()

* Update doc/src/main/scala/org/bitcoins/doc/wallet/create-wallet.sc

Co-Authored-By: Christewart <stewart.chris1234@gmail.com>

* 2019 05 05 sync chain (#460)

* Add code to sync our wallet code with bitcoind after creating a tx

Add script to illustrate how the chain persisted and how to sync against a running bitcoind instance on regtest

* fix bug relating to subtraction operator not being communative in Pow.getNextWorkRequired(). This kept us from being able to switch proof of work intervals correctly

* rename script from persist-chain.sc -> sync-chain.sc

* fix 2.11.x compile issues

* Refactor chain, node, wallet config (#463)

* Refactor chain, node, wallet config

Get rid of NetworkDb, DbConfig
Add proper structure to conf system,
moving everything under the bitcoin-s
root key.

* Remove Scalacheck from node project

* Add doc on configuration

* Add override feature to AppConfig

* Address code review in #463

* Throw if default data dir is used in tests, add Scaladoc to AppConfig

* Add explanations for withOverrides, link to configuration.md from AppConfig

* Fix compile error

* Moves chain fixtures to testkit project (#475)

reset node files

* Store encrypted mnemonic to disk (#462)

* Add WalletStorage object

* Add encrypted mnemonic storage, locked wallet

Add lock and unlock operations to wallet.
Separate between locked and unlock wallet.

* Handle non-existant seed file

* Respond to code review from Chris

* Use val instead of import

* Add doc on how mnemonics are encrypted/stored

* 2019 05 15 spv sync headers (#479)

* Implement SpvNode skeleton, create NodeUnitTest and move it to the testkit

* Implement test case to sync a header via spv into bitcoin-s

* Fix compiler errors

* Make node project Main runnable (#26)

* Add logging configuration to node project

* Make default config workable in non-test environments

* Add more logging of config in BH DAO and AppConfig

* Make Peer id optional

* Make node Main.scala runnable

* Implement Main.scala to sync with a locally running bitcoind instance. You can now run with 'bloop run node' and sync the node if you adjust the parameters inside of Main.scala. This also reworks the structure of 'AppConfig'. It turns the *AppConfig into a case class intead of case objects. This allows us to pass custom configs into those case classes

* Address code review from torkel

* Reintroduce withOverrides (#29)

* Turn off chain validation logs

* Make datadir a parameter to bitcoind config rather than having it implicitly written to the bitcoin.conf file. This was a difference that was occurring in the node branch which had a parameter for the datadir and master which was implicitly writing it to bitcoin.conf

* Add ability to overrwrite conf file except in the case of overwriting the DEFAULT_DATADIR & DEFAULT_CONF

* remove extra Bitcoind.stopServers in WalletIntegrationTest
2019-06-04 09:53:00 -05:00