* Output logger configuration on startup now
* Remove core-test/logback-test.xml and core/common-logback.xml, consolidate and move them into testkit/src/main/resources/logback.xml
* Add comment for logging configuration
* Cap the amount of threads an actor system can spin up to 2 in tests
* Add limits for internal dispatcher and default blocking io dispatcher
* Fix config
* Increase thread count to 2, remove unneeded lines
* Update scalafmt-core to 2.6.0
* Upgrade scalafmt, fix all formatting errors
* Scalafmt on 2.12
* fix test
* More fixes
Co-authored-by: Ben Carman <benthecarman@live.com>
* Implement unit test cases for initializing a wallet twice, and failing with an exception when we initialize a wallet with a different key-manager
* Use bip39PasswordOpt cached in the WalletUnitTest suite
* Turn off logging
* Run scalafmt
* Turn log level back to WARN
* Run scalafmt
* Pulled down crypto things from dlc branch including: Schnorr data structures, Schnorr Bouncy Castle implementation, FieldElement, and tests for these things
* Call fromValidHex in CryptoUtil
* Implement best block hash correctly
* Handle no common history
* Fix getBlockProof to be BigInt instead of UInt32
* Fix warnings
* Make get best filter use chainwork
* Remove unused function, test isMissingChainWork
* Fix comparisons for chain work in sql
* Fix migrations from rebase
* Fix CI error
* Fix postgresql
* Made the standard non-interactive tx finalizer simple using composition of its (reusable) pieces
* Responded to code review and made minor improvements
* Fixed docs
* Refactored Transaction
Created RawTxBuilder
Created RawTxFinalizer as layer on top of RawTxBuilder
Introduced RawTxSigner and replaced TxBuilder!
Deleted TxBuilder!
* fixed things after rebase
* Made RawTxBuilder compatible with older versions of scala
* Began responding to review
* Made Finalizer take a Builder rather than the other way around
* Added composition for finalizers
* Ran scalafmt
* Updated txbuilder example documentation
* Moved tests from old TxBuilderTest files to relevant new test files
* Added scaladocs
* Reduce number of property based tests on BIP32PathTest and add a Future to NetworkPayloadTest's property based tests so it hopefully runs faster on CI
* Run scalafmt
* Make a few more property based tests async
* Make forAllParallel, refactor some test cases to use it
* Make BlockMessageTest async
* Bump timeout
Added OutputReference and introduced public key computation on SPKs
Made InputInfo into an ADT
Made UTXOSpendingInfo use InputInfo
Replaced UTXOSpendingInfo with NewSpendingInfo and got non-test things compiling
Made aliases for (NewSpendingInfo/NewSpendingInfoFull/NewSpendingInfoSingle)[InputInfo]
Got rid of source code mention of UTXOSpendingInfo
sbt compile passes
tests compile
Fixed all coreTest tests
Renamed to UTXOInfo
Some cleanup
Moved redeem script and script witness accessors to companion object
Responded to code review
Added OutputReference and scaladocs
Moved p2pkhPreImageOpt downstream and generalized to hashPreImages
Fixed adding-spks.md
Fixed psbts.md and txbuilder.md
Renamed UTXOInfo
Apply renaming to docs
* Make Bech32Address.toString actually print the string representation not the object one
* Remove Bech32Address.toString all together and use the super class definition
* Add test case
* Run scalafmt
* Added crypto project and decoupled BitcoinSLogger from NetworkElement
Decoupled BitcoinSLogger from Factory
Moved NetworkElement into crypto project
Moved Factory and BitcoinSUtil (renamed to BytesUtil) to crypto project
Moved MaskedToString to crypto project
Added BytesUtil to imports and cleaned up CryptoUtil.recoverPoint
Moved the rest of crypto stuff to the crypto project
Moved crypto tests to crypto-test project
* Added documentation for crypto project
* Add wallet creation time for rescans
* Fix docs
* Clean up and add test
* Remove account bday
* Fix compile issue and docs
* Add more chain handler tests
* Use Instant over Long, add docs
* Fix docs
* Re-enable NeutrinoNodeWithWalletTest with it disabled for mac
* Change to only Linux
* Only disable for CI runs
* Move IsCI to BitcoinSUtil
* Fix compile issue for earlier versions
* Fix syncing compact filters to not skip last filter
* Fix test to be compatible with new start state
* Use correct isCI
* Fix compile issue
* set CI env
* Implement Slick's multi database support via our DAOs
WIP2 work on chain project
WIP make JdbcProfieComponent so we have a common JdbcProfile
WIP2
Clean up a few things
WIP got CRUDAutoInc compiling
Get closer to everything compiling in the chain project
Begin moving database code into JdbcProfileComponent
WIP source compiling
Get dbCommonsTest/ test cases passing
Get rest of tests compiling and passing
* Rebase onto master, fix conflicts
* Run scalafmt
* Run scalafmt again
* Cleanup a few compiler warnings by adding type annotations to thigns
* Explicitly cast tables where they are needed in the wallet project where we need TableQuery for foreignkey / joins in slick
* Redo definition of 'table' to use the profile.api.TableQuery in scope
* Rename all *Table.scala files to *Db.scala
* First crack at address ben's code review
* Fix docs compile issue
* Fixed casting (#30)
* Fixed AppConfig casting
* Fixed TableQuery casting in DbManagement subtypes
* Fixed casts in DAOs relating to internal types
* Address nadav's code review, move JdbcProfileComponent into it's own file
* Remove unused listTables() method
Co-authored-by: Nadav Kohen <nadavk25@gmail.com>
* WIP: Build queue for getnewaddress
* Implement background thread to process requests to getNewAddress so that we fix issue 1009 with it's async issues
* Run scalafmt
* Replace the mutable.ArrayBuffer in AddressHandling with ConcurrentLinkedQueue
* Put FundTransactionhandling.fundRawTransactionInternal's call to getNewChangeAddress() into the for expression so we don't generate an address when the funding of the transaction fails when selecting utxos
* Move thread initialization out of method
* Switch to using ArrayBlockingQueue, which will block thread with .take() so we don't need to loop constantly, add 'addressQueueSize' and 'addressQueueTimeout' configurations
* Update wallet-test/src/test/scala/org/bitcoins/wallet/AddressHandlingTest.scala
Co-Authored-By: rorp <rorp@users.noreply.github.com>
* Add error handling to return the failed future if the queue is full, add a unit test for when the queue is full
* Run scalafmt
* Rebase
* Add scaladoc for throwing an exception
* Run scalafmt again
Co-authored-by: rorp <rorp@users.noreply.github.com>
* Incoming and Outgoing Transaction Tables
* Remove script sigs for witness txs
* Create parent tx_table for incoming and outgoing txs
* Response to review
* Use isCloseEnough
* Fix test
* Fix rebase error
* Test that tx is tracking all sats correctly
* Add wallet rescan tests
* Create FilterSync, which gives us an API inside of the chain project to sync filters with
Add another unit test to filter sync
Add more unit tests for ChainSync and FilterSync
Clean up some docs, remove some extra lines of code
Run scalafmt
Add filter-sync.md
Cleanup some nits
Add more information of how FilterSync.syncFilters() works
Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project
Run scalafmt, hide imports in filter-sync.md so code appears cleaner
Move implicits out of invisible block as it seems to cause errors
Make it so FilterSync processes filters in batches rather than fetching them all at once
Fix compile error
* WIP bitcoind implement ChainQueryApi
* rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets
* Fix rebase problem
* Implement getFiltersBetweenHeight() with ben's solution
* WIP Start implementing NodeApi against bitcoind
* Actually inject the bitcoind backed nodeApi into our fixture
* Get first rescan test working for rescanning the entire blockchain
* Implement test case for rescanning from a specific height
* Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18
* Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses
* Add another log to try to debug CI
* Address code review
* Fix wrong ordering of deletion of tables in clearUtxosAndAddresses()
* reset logging level
* Create FilterSync, which gives us an API inside of the chain project to sync filters with
Add another unit test to filter sync
Add more unit tests for ChainSync and FilterSync
Clean up some docs, remove some extra lines of code
Run scalafmt
Add filter-sync.md
Cleanup some nits
Add more information of how FilterSync.syncFilters() works
Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project
Run scalafmt, hide imports in filter-sync.md so code appears cleaner
Move implicits out of invisible block as it seems to cause errors
Make it so FilterSync processes filters in batches rather than fetching them all at once
Fix compile error
* Add comment about trust model
* Run scalafmt
* Create migration to drop confirmations column from txo_spending_info table in the wallet database, also fix a long standing issue in BitcoinSFixture where failure cases in the testing suite were not being handled properly
* Fix scala 2.11.12 compile issue
* PSBT Serialization/Deserialization (#917)
* PSBT Serialization and Deserialization
* Remove debug code
* Create PSBTParseResult, Add more psbt test cases, and various small nits
* PSBT version checking
* Remove PSBTHelper, give PSBTRecord and PSBTMap a fromBytes
* Fix compile issue, preserve ADT
* PSBT Combiner
* PSBT Updater
* Fix PSBT version number to be read as little endian
* Make function names more clear, add scaladoc, require not finalized
* Combiner scaladocs
* Test case from BIP
* Make compatible with older versions
* PSBT: Fix for EmptyWitness changes (#968)
* PSBTInputMap to UTXOSigningInfo
* Remove vars
* Add missing param
* Fix byteVectorOrdering for when x == y
* getUTXOSpendingInfo Tests
* Fix formatting errors
* Give conditional path default value
* formatting fix
* Fixed InputPSBTRecord.FinalizedScriptSig using asmBytes (#1004)
* Fixed PSBTTest after rebasing onto master
* PSBT Constructors and Extraction (#997)
* Added PSBT constructors
* Revived option to use signers in toUTXOSpendingInfo
* Added transaction extraction
* Responded to code review
* Added extraction test from BIP
* Added validation (option) to PSBT extraction
* Added property based test for PSBT.extractTransactionAndValidate
* Marked PSBT as new code for PropertyCheckConfiguration
* PSBT Finalizer (#1002)
* Added finalizing functionality to PSBTs
* Cleaned things up
* Added finalizer test, fails
* Check that PSBT is not already finalized when finalizing PSBT
* Added P2PKWithTimeout cases
* Formatting fix
* PSBT Generators (#1019)
* Introduced non-finalized constructor
* Added property based test comparing finalized and un-finalized construction, it currently fails due to EmptyScriptPubKey not being supported in various ways. Also fixed a ton of bugs!
* Fixed easy bugs relating to EmptyScriptPubKey, one is left
* Fixed the last bugs (multisig order and 0-of-n stuff) and now tests pass!
* Separated out direct finalized PSBT construction from non-finalized PSBT construction
* Translated construction work in tests into PSBT generators
* Added serialization symmetry test
* Made PSBT fee generation dependent on other generators to ensure low enough fees for signing
* Responded to code review
* Added scaladocs
* Fixed arbitraryPSBT to work with older scala versions
* Fixed PSBT compilation breakages from rebase
* Fixed test breakages from rebase
* Validate the PSBT unknown is not a known field
* Increase code coverage on PSBT tests
* Address review
* formatting
* Add error messages, rename function
* Psbt Signer (#1025)
* PSBT Signer
* Create addSignature function for PSBTs
* Use PartialSignature instead of Tuple
* Create extra util functions
* scaladoc and simplify case
* Formatting fix for psbt scaladocs
* Clean up code, add error messages, and scaladoc
* add type hints
* Change InputPSBTMap.fromUTXOSpendingInfo to use BitcoinSingleSigner
* optimize import
* Define separator byte for psbt map
* getRecords simplification
* remove braces, add error message
* PSBT stuff assigned to nkohen (#1035)
* Responded to psbt review assigned to me
* Moved separatorByte to a place where all can use it
* PSBT independent record calls (#1037)
* PSBT independent record calls
* move logic to super class
Co-authored-by: Nadav Kohen <nadavk25@gmail.com>
* Reworked PSBT finalization to return a Try[PSBT] rather than an Option[PSBT] (#1036)
* PSBT signer property based tests (#1038)
* PSBT property based tests
* Address review
* PSBT Combiner property based tests (#1039)
* PSBT Combiner property based tests
* Create pruneGlobal util function
* Switch to arbitrary gen
* Use BaseTransaction instead of Transaction
* Fix transaction witness to work from base transactions
* PSBTs response to Chris's review (#1046)
* PSBT chris's review
* Rename val, give master fingerprint a val
* Formatting fix
* Responded to code review from chris assigned to nkohen
* Split PSBT.scala into many files
* PSBT Docs (#1048)
* PSBT docs
* Fix compiler error, remove nesting
* PSBT output updater tests
* Validate psbt outputs
* Clean up code
* Remove P2WPKHWitnessV0 addition
* format
* PSBT Updater property based tests
* Move code to match block
* EmptyScriptWitness check
* Address nadav review
* Remove unnecessary code, make utxo adder better
* Move require and add error message
* Move require
* PSBT unit tests by Ben
* Change to EmptyScriptPubKey
* Added unit tests to increase code coverage
* responded to review
* Move PSBT unit test to separate file
* Make unknown generator only generate distinct records
* Use groupBy key instead of distinct
* use only first element
* Combine distinctness by key
Co-authored-by: Ben Carman <benthecarman@live.com>
* Create 'fundRawTransaction' and 'fundRawTransactionInternal'
* Move TransactionTestUtil to testkit, begin writing tests for fundrawtransaction
* Add FundTransactionTests
* Move things to 'FundTransactionHandling' trait
* Start segregating things by HDAccount in the wallet, this gives us the ability to query for balances based on account, fund a transaction based on an account, generate a new address based on an account etc. All old api calls are now based on the default account, i.e. getBalance() returns the balance for account 0, fundRawTransaction funds the transaction from account 0 etc.
* Fix compile issue with 2.11.x
* Bump address generation Thread.sleep() to 500ms
* Address ben code review
* Address code review
* Introduced UTXOSpendingInfoSingle
* Implemented constructors for UTXOSpendingInfoSingle
* Introduced BitcoinSignerSingle
* Implemented SingleSigner
* Added property-based test for SingleSigner, it fails
* Fixed SignerTest :)
* Responded to code review
* Added scaladocs
* Fixed p2sh(p2wsh) signing
* Responded to code review and separated UTXOSpendingInfoSingle from UTXOSpendingInfoFull
* Fixed outputGen after rebase
* Fixed txbuilder.md to show this branch's changes
* Create 'fundRawTransaction' and 'fundRawTransactionInternal'
* Move TransactionTestUtil to testkit, begin writing tests for fundrawtransaction
* Add FundTransactionTests
* Move things to 'FundTransactionHandling' trait
* Address code review
* Run scalafmt
* Add more expressive txo states for the wallet
* Rename 'PendingReceived/PendingSpent' -> 'UnconfirmedReceived/UnconfirmedSent'
* Address code review
* Fix posInt implementation to work with older scala versions
* Rename to PendingConfirmationReceived/PendingConfirmationSpent
* Implement abililty to use BIP39 password. This means this password needs to be password through our various projects to be able to correctly generate the key that controls the wallet. This also renames 'CreateKeyManagerApi' -> 'BIP39CreateKeymanagerApi' as the bip39 password is needed when creating the KeyManager
* Add bip39 password to BIP39KeyManager.fromParams(), fix mdocs
* Fix bug in unit test were whe weren't specifying password
* Fix rebase issues
* This creates a subtype BIP39KeyManager and moves all existing KeyManager functionality to that subtype. We leave a empty 'KeyManager' trait for now
* Fix website docs
* Move initialization of wallet entropy into the key manager
* Enable key-manager in ci, add coverage minimum
* Add coverage minimum for key manager, add KeyManagerParams, start adding KeyManager test harness
* Remove success cases on ADTs, start using Either
* fix import
* Get test coverage to 90%
* Add documentation for key-manager.md, add side bar for key-manager
* Fix wallet.md
* Make seedPath default to baseDatadir, so we don't have a unique seed for every network. Add more documentation
* Address code review
* Address code review from Ben
* Add missing import
* Add more tests for coverage
* bitcoind v19 new RPC calls and tests (#863)
* bitcoind v19 new RPC calls and tests
* Code review changes
* Review part 2
* Rename variable to be more descriptive
* Explanitory comment
* Ignore broken test cases
* Add missing signing functions
* Add test to check avoid_reuse flag is on (#870)
* Add test to check avoid_reuse flag is on
* Add test to make sure flags weren't set
* bitcoind v19 Update mempool RPCs and tests (#868)
* Update mempool RPC calls to bitcoind v19 compatibility
* Typo fix
* Add parameter name to calls
* Fix remaining rpc calls
* Formatting
* scaladoc for param
* Change param to correct type
* Clarify on scaladoc
* Add missing fees parmater to mempool rpcs (#875)
* Add weight field to mempool entries after v19 (#876)
* Move DescriptorRpc to be able to be used by future versions of bitcoind (#878)
* Add window_final_block_height to GetChainTxStatsResult (#880)
* Add passphrase argument to createwallet for later versions (#883)
* Add passphrase argument to createwallet for later versions
* Scaladoc + empty passphrase requirement
* Error message
* Add new services names parameter to P2P rpcs (#874)
* Add new services names parameter to P2P rpcs
* Add ServiceIdentifier Reads
* Add fallback case
* Address review
* Change to Try
* Move PsbtRpc to be able to be used by future versions of bitcoind (#877)
* Move PsbtRpc to be able to be used by future versions of bitcoind
* Add test
* Address comment
* Enable bloom filters for v19
* Enable bip 61 for tests
* Change to official binaries
* Force v18 for Spv Tests
* Remove unused config line
* Segregate key manager from wallet
* More consistent naming in build.sbt, run scalafmt
* Add test case making sure we don't overwrite a mnemonic seed
* Add key-manager.md
* Fix compile issues with older scala versions
* Address code review
* Added MultiSignatureWithTimeoutScriptPubKey
* Renamed If and NotIf to NonStandardIf and NonStandardNotIf and removed multiple representations
* Added MultiSignatureScriptPubKey to CreditingTxGenerators
* Responded to code review
* Fixed UTXOSpendingInfoTest
* Added ConditionalScriptPubKeys to CreditingTxGen so that we are actually testing Conditional signing now
* Responded to code review
* Renamed scriptPubKeyTooBig => redeemScriptTooBig
* Fix monitorInvoice unit test
* more logs
* even more logging
* fix build error
* custom eclair jar
* custom eclair
* more logging
* remove println's
* more logging
* more logging
* fix 'payment expiry is too close to the current block height'
* pobably the final fix
* cleanup
* fix download URL
* responded to PR comments
* Introduced ConditionalScriptPubKey
Some more progress
Added Conditional signing infrastracture, tests do not pass and there are no Conditional tests
Fixed tests!
* Redid calcLockTime and calcSequenceForInputs in TxBuilder (added Conditional support)
* Added ScriptGenerators for Conditional scripts, only one test to go!
* Fixed CLTV(Empty) tests
* Responded to code review
* responded to code review
* ScriptSignature only types Policy-compliant script signatures
* Fixed md doc
* Replace scriptPubKeyToSatisfy with spendingInfoToSatisfy and made UTXOSpendingInfo ADT nicer
Tightened up types on Signer with a type parameter
Factored out common functionality in Signer
Responded to code review
* More review
* Refactored Signer interface to use TxSigComponent and UTXOSpendingInfo
Fixed all the broken things
Replaced TxSigComponent with unsigned Transaction in Signer interface to remove illegal states
* Added p2sh throw case in Signer
* Create base traits for async tests called BaseAsyncTest, and then create two subtypes, BitcoinSAsyncTest and BitcoinSAsyncFixtureTest to make it easier to manage resources related to test cases. Now it is easier to shutdown things like an actor system in a structured manner
* Remove two bitcoind tests that don't make sense, 'duration' in the parameter is _really_ an interval (bad naming by usTesting an interval does not make sense inside of TestRpcUtilTest
* Turn log off for ChainAppConfigTest we expect to fail
* Bump timeout for BitcoinSUnitTest, use builtint sbt scalafmtCheck
* Remove hardcoded scalafmt binary
* reduce generator size to attempt to fix#768
* Remove AKKA_CONFIG that was preventing sane logging for akka, move db-commons reference.conf to testkit so we can control testkit applications, and then rename app/server/.../application.conf to reference.conf like good libraries should do
* Update contributing.md
* update a few more links inside of contributing.md
* Run scalafmt
* Update scalafmt to 2.0.1
* Scalafmt
In this commit we
1) Reconfigure the .scalafmt.conf file
2) Add a scalafmt binary (same version as specified in conf file)
3) Use said binary on CI to check that everything is formatted correctly
* Ran scalafmt
* Update scalafmt ignore
* Add scalafmt plugin to project/plugins.sbt, run it
* run scalafmt
* This is for colin
* Fix remaining errors on V18 RPC calls
* remove unused code final formatting fix
* implemented fixed listwalletdir rpc test
* responding to code review and replacing flatmaps
* Remove BlockchainBuilder, speed up ChainHandler test that processes mainnet headers, add test for ChainAppConfig
* Add test for BlockHeaderDAO.getAncestorAtHeight()
* Consolidate ChainAppConfigTest, add ScalaTestUtil to help with working with scalatest, speed up ChainHandlerTest a bit more
* Build and CI improvements
In this commit we:
1) Parallelize the Travis CI config, by splitting
each project into its own Travis task
2) Download bitcoind binaries through sbt
* Use binaries downloaded by sbt task
* Make BitcoindRpcTestUtil work on Travis without bitcoind on PATH
* Add new downloadEclair task to sbt
* use sbt downloaded binaries in tests
* Fix Eclair and Bitcoind tests
* Bump Scala versions
Support Scala 2.12.9
and 2.13.0.
To make this easier, we delete the `scripts` project. Everything
that was in here was covered by content on the website. We also
delete the `doc` folder, as that was a remnant from when `scripts`
was called `doc`.
* Crib uPickle akka-http support while we wait for publish
* Fix compiler warnings
* Add note on test logging to contribution guide
* Reduce duplication in Blockchain implementation
* Use Scala 2.12 for website
* Introduce compat package object for collections converters
* Fix Either compiler warnings
* Add sync-chain and create-wallet docs from deleted scripts
* Fix rebase goofup
* Split app loggers into their corresponding modules
This avoids a bunch of places with awkward implicit resolution.
We also make sure that all logging in node, chain and wallet
happens with the app loggers, and not BitcoinSLogger
* Tune test logging level to WARN
* Start refactoring BroadcastTransactionTest to use new SpvNodeFundedWalletBitcoind fixture
* Double timeout to 10.seconds on spv node disconnect
* Attempt to bump disconnect even more
* Add more logging around starting and stoping spv node to try and debug failures with fixtures on test suites
* move starting of spv node into one place in the fixtures -- createSpvNode() --, also look at ordering of destruction to make sure that the spv node is stopped/destroyed BEFORE bitcoind is stopped/destroyed. If the inverse order happens, our spv node won't receive a response from bitcoind and will hang waiting for the Tcp connection to close gracefully
* Try switching from to to get CI to pass
* Add more logging and throws to try and debug failures on ci
* Add logging in the case we send a close command and peer isn't defined
* Match on Tcp.Connect on default receive in P2PClient
* Make SpvNode.disconnect() check to see if we are connected before attempting to disconnect
* Add more logging in fixtures, change behavior of SpvNode.disconnect() to not throw if we are already disconnected
* Switch to preferred way of terminating actor system in tests
* Make PeerMessageSender.sendMsg() return a , make sure a message is not sent to P2PClient until we are fully intialized
* Switch p2p logging to DEBUG
* Complete disconnect promise on peer message recv in the case where we didn't have a peer to begin with
* Make SpvNodeTest more robust, fix bug for when we send 'sendheaders' message. We shouldn't do this until our peer has sent us a verack
* Only send 'sendheaders' message to our peer after we receive a 'sendheaders' message from our peer. Hopefully this solves async issues where we were sending 'sendheaders' before our handshake was fully completed with our peer which means we wouldn't get headers sent to us
* Cleanup pt1
* Start the process of refactoring our ChainHandler to be able to avoid database calls on TipValidation
WIP: Begin explicity passing state back and forth in return types of PeerMessageReceiver, P2PClient, , DataMessageHandler. This commit also implements the ability to keep our blockchain completely in memory. Previously when we were updating the tip of the chain, we had to make a database read to figure out what the best tips are. This is suboptimal for performance because a database read needs to be done for every block header we see, now we just keep the chain in memory
Fix bug in DataMessageHandler that pre-emptively sent a getheadersmsg to our peer. Make 'chainApiF' internal to our spvNode (not a parameter). This forces the chainApi to be created from disk everytime a new SpvNode is spun up. This keeps us in sync with the blockchain at disk at the cost of disk access and less modularity of SpvNode
Address torkel code review
Fix rebase issues
Address code review
Address nadav code review
* Rebase onto master, fix api changes
* Address issue #673, also refactor fixture stuff to be in companion objects rather than traits so they can be re-used without extending the trait itself. This allows for more modularity with our fixtures
* fix bugs in unit tests, address #675 to make MerkleBufferTests faster
* Refactor test timeout in UpdateBloomFitlerTest
* fix callback logic bug on matching unrelated txs/addresses in UpdateBloomFilterTest, re-order broadcasting of things to avoid async bugs hopefully
* Make a def in NodeUnitTest, this keeps tests in the same suite from using the same config. This allows us to write multiple tests per suite, instead of just one. This also adds implicit parameters to our fixture constructors/destructors to properly create and destroy this config. The long term goal here also should be getting rid of config.initialize() we are calling everywhere as this is a anti-pattern, fixtures should take care of construction and destruction of things
* Broadcast tx earlier in test case for UpdateBloomFilterTest
* Rework NodeWithWalletTest to use the new fixtures we have in the node project, now use SpvNodeFundedWalletBitcoind
* Add functionality for updating SPV node bloom filter
* Add SPV node shutdown to exit hook
* Clean up traits/case classes
* Change fixture in WalletBloomTest
* Fix logging bug in TransactionProcessing
* Add MerkleBuffers
In this commit we add MerkleBuffers, which is an object
that lets us aggreagate merkleblocks with their corresponding
transactions before sending them out. This is global, mutable
state (bad!) but it's a working solution for now;
* Use TestAsyncUtil
* Add MerkleBuffers test
* Send getdata if receiving single header
* Change awaitSync to use block count
* Fix UpdateBloomFilterTest
* Add more logging of chain/headers validation
* Send getdata for all blocks
* Nits: Scaladocs, comments toString
* Add logging to data directory
In this commit we add the ability for the node, chain
and wallet projects (+ the server) to log to the users
data directory instead of whatever directory the
binaries was launched from. This is inherently a bit
more complicated than our previous setup, because
we need to read the user directory before we can create
loggers. As a result of this, some files/methods were
moved around, so the relevant app config could be
found in scope.
We also introduce several logging categories that can be
tuned individually through user configuration. These logggers
are exposed both as traits that give a field `logger`, or
as methods that return the required logger.
* Add datadir configuration to AppConfig
In this commit we add support for AppConfig to pick up
the data directory configuration file. We also add
a section to the contributing guide on how to tune
logging levels.
* Pass data directories explicitly for configuration
* This PR is attempting to remove the publishing errors when publish
locally many of which are incorrectly linked things in scaladocs which
there are some differences in formatting. Hopefully this removes alot
if not all of the "link not found for ..." warning messages that show
up.
* Add functionality for broadcasting TXs to node
In this commit we add functionality and tests
for broadcasting a TX from our node. To accomplish
this we introduce a table over broadcastable TXs
that's added to when the externally facing method
broadcastTransaction(tx) withing SpvNode is called.
We send out a inv message for the TX we just added,
and upon receiving a getdata message we search in
the previously mentioned table for entries where
the hashes match up.
* Broadcast TX from server to SPV node
* Perform assertions on the balance of bitcoind after sending a TX
* Remove typeclass from broadcast TX
* Refactor withFundedWalletAndBitcoind
* Match on BitcoindExecption instead of throwable
* Clean up broadcast functionality after code review
* Cleanup
In this commit we cleanup a few Scaladocs and change some
trait/impl pairs into just a case class.
* Node cleanup
In this commit we
1) Rename Client to P2PClient. Client is a very generic name.
2) Clean up some Scaladocs
3) Remove some unecessary objects/traits/classes and replace them
with case classes
4) Add trace logging of bytes received in P2PClient
* PoC bitcoin-s-cli
* Add CLI, Server sbt projects, remove Ammonite
In this commit we set up sbt configuration for
CLI, Server (in-work-name) and corresponding
test projects.
We also remove Ammonite shell from sbt, as that
isn't really being used. bloop console offers
the same functionality way more ergonimic.
* Move BitcoinSAppConfig into new server project
Server project depends on node, chain wand wallet
so this is a good time for introducing this class
into main sources. We also introduce
BitcoinSTestAppConfig in testkit, to replace the
functionality in BitcoinSAppConfig related to
tests.
* Type chain in blockchainresult
* MVP server setup for node, chain and wallet
* Extremely dirty CLI for interacting with server
* initial attempt at mimicking Bitcoin Core API
* WalletStorage: add method for checking for seed existance
* Check for seed existance on wallet startup
* Fix bug where MnemonicNotFound was not an error
* Segregate confirmed and unconfirmed balance methods
* Add error handling, improve formatting of CLI output
* Tweak build
Bump Sttp version, downgrade to uPickle 2.11 compat,
skip publish in cli-test and server-test
* Add CLI, server and picklers to root project
* Split wallet functionality into multiple traits
In this commit we refactor LockedWallet into multiple traits
that provide functionality related to a subset of total wallet
functionality. This has the benefit of making it clear which
methods are helper methods that are only intended to be used
in a very specific setting, and which methods are part of the
internal wallet API that other parts of the wallet can use.
* Rework TransactionOutput and TransactionOutPoint to case classes
* Add extension methods for flattening lists of assertions
* Segregate confirmed and unconfirmed balance methods
* Add test for FutureUtil.sequentially
* Add trace logging of balance fetching
* Process outgoing TXOs
Move TX processing into separate trait, add internal API method
Unify DB representation of TXOs
Prior to this commit we stored TXO information
across diferent tables, with joins and tuples
needed a bunch of places to keep track of
everything we needed. In this commit we unify
the tables, leaving us with only one table for
TXOs.
In this commit we replace addUtxo with
processTransaction in the wallet API. addUtxo
is kept around, but only as an internal
implementation detail.
The new flow looks like this:
1) The wallet gets notified about a new TX
through a processTransaction call
2) It then searches through our already
processed incoming TXs:
if found, updates the confirmation
status
else, inserts it into the DB
3) We then call addUtxo, if the previous
step resulted in a new TX in our
DB
In this commit we update our AesCrypt
functionality to work with AES keys rather
than passphrases that are extended to keys.
This makes it easier to integrate with other
languages, as the key stretching introduces
a bunch of additional parameters to the
actual AES encryption.
We also include tests that verify that we're
able to encrypt/decrypt using Bitcoin-S,
Crypto-JS (JavaScript), pycrypto (Python)
and OpenSSL CLI.
* Attempt to sync with sendheaders
Fix sync with headers by generating one block in regtest to get out of IBD
Fix comment
Add more comments
Add missing actor system import
more clean up
* refactor test case to avoid sleeping the thread
* Log messages more granularely in node
* Add pong message
* Rework P2P service identifier
The old implementation didn't catch the notion
that service identifiers was a bit field where
multiple services could be set at the same time.
* Make Peer take InetSocketAddress
Previously it took NetworkIpAddress.
This doesn't make any sense, as we
need to know the services bitfield
to construct a NetworkIpAddress.
* Clean up logs and toString methods
* Clean up node README and main method
* Make all networks final case objects
* Skip publishing of wallet, node and chain
* Clean up logging of node and chain modules
* Add logging of max height in SpvNodeMain
* Clean up Scaladoc and toStrings
* Add logging configuration as symlinks in node
We'll have to figure out a more stable solution to
configure logging for the SPV node, but for the time
being this is an acceptable solution.
* Fix 2.11 compiler error
* Implemented GCS (WIP)
Added some of Chris' tests for golombEncode and golombDecode
Fixed GCS.golombDecodeSet
Separated GCS object from GolombFilter class
Added hanging test for set encoding
Fixed property test for encoding and decoding arbitrary sets
Added test for GCS.buildGCS and hashing function consistency
Added method to construct a Block Filter from a Block
Added scaladocs in GCS.scala
Added TempUtxoProvider to make BlockFilter testable
Wrote a test for GolombFilter.matchesHash
Added test for arbitrary data matching in bip 158 filter
Added optional right-padding on GCS BitVector, BlockFilter.fromHex, and got started on BlockFilterTest
Cleaned up Bip158TestCase.fromJsArray
Fixed parsing of test vectors for bip 158
Wrote test for BlockFilter (doesn't pass)
Responded to some code review
Included CompactSizeUInt in filter
Created BitcoinScriptUtil.getDataTokens for output serialization
Wrote test for BitcoinScriptUtil.getDataTokens
Added test for BitcoinScriptUtil on the genesis block
Fixed name
:(
* Fixed SipHash and block serialization!
* Fixed bugs relating to empty scripts, all tests passgit push --force-with-lease nadav 2019-05-28-bip158 !
* Cleaned up BitcoinScriptUtil.getDataTokens and its test
* Responded to code review
* Factored out common code from BlockFilter apply methods
* Some cleanup
* Add getAddressInfo to Wallet API
* Add Bloom filter and event callbacks to SPV node
* Move wallet test trait to testkit
* Test for connecting SPV node with wallet
* Tune logging in SPV node
* Clean up BloomFilter
* Proper toString in inventory and inventorymessage
* Actually pass in callbacks to SPV node
* Fix rebase screwup and partially respond to code review
* Default to file based databases in tests
This is due to issues with deadlocks with
in-memory based databases. We need DBs
to stay alive between connections, but
not across tests.
In DB intensive chain validation tests we
use in-memory databases instead. This
seems like a reasonable tradeoff between
simplicity and speed.
* Make SpvNodeCallbacks contain sequence of functions within
* Make Bloom filter non-optional in SpvNode
This is due to issues with deadlocks with
in-memory based databases. We need DBs
to stay alive between connections, but
not across tests.
In DB intensive chain validation tests we
use in-memory databases instead. This
seems like a reasonable tradeoff between
simplicity and speed.
* Rename P2P tests and move generators into correct package
* Add more tests for P2P messages
* Add note on coverage to contributing guide
* Replace network in P2P header with typed network
* Add more P2P tests
* Add more tests for ECPriv and ECPub
* Reshuffle package location of P2P messages
Previously the companion objects for P2P messages
and the actual messages were in different pacakges.
This made them more awkward to use than necessary.
We also clean up some Scaladocs in this commit.
* Move all P2P messages to core
* Replace AppConfig with more specific types
* Rework database configuration
In this commit we:
1) Introduce new database settings for all submodules,
and at the same time remove the singular datbase setting
for the entire project
2) Introduce a new method .initialize on all configs
that is responsible for creating needed directories,
databases and files
3) Introduce a new class BitcoinSAppConfig that wraps
configs for all three other subproject config. We
also provide implicit conversions that enable this
super-config to be passed in wherever a specialized
config is required.
4) Add more tests for our configuration setup.
* Add Ammonite to Docs deps
Every now and then the Scalatest framework
has crashed when reporting errors, with
failure to serialize this. There should
be no downside to having the trait here,
and we avoid the nasty Scalatest failure.
* WIP: 2018 12 22 node project (#280)
* Add files from old spv node project
src compiling
test files compiling
ran scalafmt
Fix serializer tests
Get non networking test cases to work
WIP: Debug peermessagehandler
Update CRUD, remove all of the Actor craziness. Add DbManagement trait and unit test db
WIP: Rewroking PeerMessageHandler, create Peer, DataMessageHandler, PeerHandler
Reworking Client to handle all tcp messages and message alignment for bitcoin p2p messages
* Wip: Node refactor
* Create node test project, move all node tests into that project and move all generators for the node project into testkit
* Rework ClientTest to use testkit, start minimizing akka usage, implement connect(),isConnected(), disconnect(), isDisconnected() in PeerMessageReceiver
* Create Peer, PeerHandler, PeerMessageSender and PeerMessageReceiver
* update readme about status of node project (#359)
* Add flyway plugin to manage database schemas (#361)
* Add flyway plugin to manage database schemas
* Switch database driver to sqlite3 to be more portable, rework configs for sqlite3
* Set up sqlite database directories and files if they are not already created
* Add torkel's review
* Add chain, wallet, db-commons projects (#367)
* Add chain, wallet, db-commons projects
* Rework db creation logic if they db does not exist
* Add config logging to try to debug travis ci
* Pass explicit class loader for db config
* Remove duplicate call to dbConfig
* Make DbConfig.dbConfig a lazy val
* Remove noisy log
* Add scaladoc to DbConfig
* Switch dbConfig readme paragraphs
* Fix compile issues introduced during rebase onto master with rpc changes (#394)
* WIP: 2019 03 12 tip validation (#378)
* Implement blockchain handling data structures
Add TipValidation happy path
Add more test cases for TipValidation.checkNewTip for badPrevBlockHash and badPOW
Add overflow check, fix endianness bug for checking proof of work
Add pow transition check, refactor difficultyChangeInterval into chain params, add more tests
fix a few nits
Fix compile error, clean up unused import
Remove redundant files from node project
* Implement GetNextWorkRequrired/CalculateNextWorkRequired, move BlockHeaderDAOTest cases into chain project
* Add full POW change check in TipValidation, address code review nits
* Configure logging in chainTest, turn logging OFF in other test projects
* Address code review pt2
* Add coverage minimum for chain project (#398)
* Add coverage minimum for chain project
* Add first Blockchain.connectTip() unit test, switch to a in memory sqlite database for unit tests, starting using fixtures for BlockHeaderDAO in unit tests
* Add tests for ChainHandler.processNewHeader(), ChainHandler.getHeader(), Blockchain.connectTip(). Refactor redundant configurations being passed around excessivly
* Address code review, fix a flaky test in ClientTest.scala
* Test Fixtures (#403)
* Working test fixtures
* Removed ChainTestFixture trait in main code
* Composing Fixtures (#413)
* Downloaded over 9000 mainnet BlockHeaders into a json file
* Added new fixture with populated blockHeaderDAO
* Split writing to db into batches
* Rebased
* Simplified fixtures with makeFixture abstraction
* Added util functions for composing builders
* Add integration test between bitcoind <-> zmq <-> bitcoin-s-chain project. Test that we can relay a header from bitcoind over zmq into the bitcoin-s chain project. Redo ZmqConfig to use InetSocketAddress
* Address code review
* wip
* A compiling withBitcoindZmqChainHandler fixture
* Tests passing!
* Made blockHeaderDAO private
* Got 9000 new block headers from 562375 to 571375
* Added offset to populated blockHeaderDAO fixture
* Added scaladocs to fixture things
* Initial wallet: import UTXO and spend it (#391)
* Updates ExtKeyVersion with fromChainParams method
* Add equals to Address
* Update BIP44 classes
* Add ScriptType
* Initial work on wallet support
* Add foreign keys pragma for SQLite
* Add UTXO models and DAO
* Add addres P2WPKH generation and WIP for addUTXO
* Add logging config for wallet
* Add change address generation, proper-ish addUtxo and sendToAddress
* Address code review on #391
* Add empty AES passphrase invariant
* Add poor mans test fixtures
* Add listUtxos, listAddresses and getBalance to wallet API
* Use fixtures from chain project
* Fix CI test failures
* Fix broken up package path
* Updates bloop config for new projects (#424)
* Multi fixture file (#419)
* Created FixtureTag and ChainFixture
Used ChainFixture in BitcoinPowTest
Added implicit conversions for nice syntactic sugar
* Added documentation for multi-fixture
* Made defaultTag a val
* add a logback-test.xml to the wallet project (#433)
* Introduce AppConfig that combines ChainParams and DbConfig (#432)
* 2019 04 23 app config per project db config per project (#434)
* Add DB conf file resolution that works across projects
* Create applicatoin configurations for specific projects, rework DbConfig structures for individual projects. Force network to be mixed into DbConfig rather than DbConfig to be mixed into the network
* Add ammonite to db-commons, remove noisy logs
* remove mixin for DbConfig that required a NetworkDb. Now networkDb is just a field on 'DbConfig', this simplifies things downstream type wise when interacting with the projects AppConfig. This commit also removes a parameter from AppConfig, now only a DbConfig needs to be passed in, and we can derive the network and chain params from the DbConfig. The only exemption is UnitTestDbConfig as it is sometimes handy to specify a different network (i.e. mainnet) when testing
* Turn DbConfig objects to case objects, wrap those case objects in their parent type companion object
* remove cast in Wallet.scala
* Add EnhancedEither class for 2.11 compat (#437)
Add implicit conversion from Either to
2.11-compatible Either-wrapper. Also
remove trailing comma in WalletTestUtil
that breaks 2.11 build.
* Fix CI tests hanging (#438)
* Execute wallet tests sequentially to avoid SQLite deadlocks
* Refactor logback config to reduce duplication
* Use in-memory SQLite DB for unit tests
* Debug prints for DatabaseConfig.forConfig
* Fork JVMs in test to ensure proper in-memory DBs
* Pass in Akka config to Eclair tests, avoid cluttering Akka log output
* Don't fork JVM on node tests'
* Move things out of ChainUnitTest (#448)
* Move things out of ChainUnitTest
* Remove printlns
* 2019 04 29 client test (#449)
* Bump timeout on connect to node test
* Change from isConnected -> isInitialized to avoid the error trying to disconnect before we are fully initialized
* Wrote tests for POW difficulty change calculation and header processing (#429)
Fixed BitcoinPowTest
Rebased onto AppConfig code
Rewrote ChainHandler integration test
Made chain handler test synchronous
Fixed a couple test bugs
Implmented a more efficient getAncestorByHeight
Fixed ChainHandler integration test by using the correct starting conditions
Responded to code review
Responded to more code review
Deleted redundant Pow test
Made BlockHeaderDAO.getAncestorAtHeight use a List for its loop to improve performance
* WIP: Create ChainSync, BitcoindChainHandlerViaRpc, add simple ChainSyncTes… (#450)
* Create ChainSync, BitcoindChainHandlerViaRpc, add simple ChainSyncTest to sync one block from a external bitcoind instance via rpc
* Add check for having the best block hash in our chain state already
* Fix prev block hash to be empty hash if genesis block header
* BlockchainBuilder (#439)
* First commit for implementing a BlockchainBuilder
* use Builder rather than ReusableBuilder to be compatible with scala 2.11.x
* Decouple Blockchain & BlockHeaderDAO
* Rebase onto node, incorporate changes in #429
* Add more comments
* Reverse order of headers in builder
* rebase onot node branch, refactor apis
* DB: Add utility method for listing tables in a DB (#447)
* Node rebase (#458)
* Implement BIP32 path diffing
* Rebase node onto newest HD changes in master
* Fix 2.11 compile errors
* 2019 05 01 wallet ammonite scripts pt2 (#452)
* wip -- not finding testkit in doc worksheet
Wip -- classdef not found for create-wallet.sc
zmq bug
Clean up some logs
nest zmq start in bitcoindF
update jeromq to 0.5.2-SNAPSHOT to get rid of annoying log to stdout
Rebase onto node branch with new configs
Successfully running ammonite script create-wallet.sc
2019 05 01 wallet ammonite scripts pt2 (#25)
* Refactor Ammonite dep
* Add basic error handling in AmmoniteBridge
* Add very basic README for doc project
Fix compile issues after rebasing onto master
Add code to sync our wallet code with bitcoind after creating a tx
* refactor ZMQSubscriber to _hopefully_ avoid hanging when we call context.term(). We do this by closing the socket before calling context.term() and using socket.setLinger()
* Update doc/src/main/scala/org/bitcoins/doc/wallet/create-wallet.sc
Co-Authored-By: Christewart <stewart.chris1234@gmail.com>
* 2019 05 05 sync chain (#460)
* Add code to sync our wallet code with bitcoind after creating a tx
Add script to illustrate how the chain persisted and how to sync against a running bitcoind instance on regtest
* fix bug relating to subtraction operator not being communative in Pow.getNextWorkRequired(). This kept us from being able to switch proof of work intervals correctly
* rename script from persist-chain.sc -> sync-chain.sc
* fix 2.11.x compile issues
* Refactor chain, node, wallet config (#463)
* Refactor chain, node, wallet config
Get rid of NetworkDb, DbConfig
Add proper structure to conf system,
moving everything under the bitcoin-s
root key.
* Remove Scalacheck from node project
* Add doc on configuration
* Add override feature to AppConfig
* Address code review in #463
* Throw if default data dir is used in tests, add Scaladoc to AppConfig
* Add explanations for withOverrides, link to configuration.md from AppConfig
* Fix compile error
* Moves chain fixtures to testkit project (#475)
reset node files
* Store encrypted mnemonic to disk (#462)
* Add WalletStorage object
* Add encrypted mnemonic storage, locked wallet
Add lock and unlock operations to wallet.
Separate between locked and unlock wallet.
* Handle non-existant seed file
* Respond to code review from Chris
* Use val instead of import
* Add doc on how mnemonics are encrypted/stored
* 2019 05 15 spv sync headers (#479)
* Implement SpvNode skeleton, create NodeUnitTest and move it to the testkit
* Implement test case to sync a header via spv into bitcoin-s
* Fix compiler errors
* Make node project Main runnable (#26)
* Add logging configuration to node project
* Make default config workable in non-test environments
* Add more logging of config in BH DAO and AppConfig
* Make Peer id optional
* Make node Main.scala runnable
* Implement Main.scala to sync with a locally running bitcoind instance. You can now run with 'bloop run node' and sync the node if you adjust the parameters inside of Main.scala. This also reworks the structure of 'AppConfig'. It turns the *AppConfig into a case class intead of case objects. This allows us to pass custom configs into those case classes
* Address code review from torkel
* Reintroduce withOverrides (#29)
* Turn off chain validation logs
* Make datadir a parameter to bitcoind config rather than having it implicitly written to the bitcoin.conf file. This was a difference that was occurring in the node branch which had a parameter for the datadir and master which was implicitly writing it to bitcoin.conf
* Add ability to overrwrite conf file except in the case of overwriting the DEFAULT_DATADIR & DEFAULT_CONF
* remove extra Bitcoind.stopServers in WalletIntegrationTest
* Add proper parsing of bitcoin.conf, remove datadir
Somewhat recently a new config format was introduced
in Bitcoin Core. Options can now be specified under
section headers ([regtest], [main], [test]), as
well as with network prefixes (regtest.rpcport,
main.prune) and the traditional format. In this commit
we introduce a new type BitcoindConfig that parses
this format. We also make the necessary changes in
the bitcoind/Eclair RPC clients to accomodate this new
type.
We also remove the datadir field from
BitcoindAuthCredentials. This is not strictly necessary
to connect to a bitcoind, so this is a bad separation
of concerns. The datadir field is instead moved into
the BitcoindInstance.
Finally we go over some places in tests and test
utils and assert that we are operating in the
user temporary directory. This is a safety measure
so that other developers/users won't do the same
mistake I did while working on this and accidentally
blow up their $HOME/.bitcoin directory.
* Add BitcoindRpcClient.fromDatadir
* Address code review from Chris
* Add BIP44 and BIP32 path support
Make paths from strings, indexes/coins/chains,
BIP32 children types.
Derive xprivs and xpubs from paths.
* Address code review on BIP44 from Chris
* Rename children -> path, bip32child -> bip32node
* update README with bip44 name changes
* Reduce number of requests we send to eclair in testkits, add commandName param to parseResult in EclairRpcClient
* Factor our EclairRpcClient error message
* Make jvm opts more sane, refactor error message code again
* Add error message on EclairRpcTestUtil.sendPayments, scalafmt
* Fix bug of sign issue when serializing r,s ln invoice signatures, we weren't using the proper sign in some rare cases. We always want the BigInteger to be non-negative for r,s
* Fixing initialization problem with EmptyDigitalSignature
* revert default Deps.scala bitcoinsV to published version
* Address code review nits
* remove negative ln currency unit test case
* Add BasicArithmetic to MilliSatoshis
* Add BasicArithmetic
* Add LnCurrencyUnit, CurrencyUnit, Number to BasicArithmetic
* Add tests for BasicArithmetic
* Make logger in unit test class protected and annotated
* Use BitcoinSUnitTest class
* Address code review
WIP: Refactoring EclairTestUtil to use Futures
Finished refactor EclairTestUtil to use futures
Address code review
fix async bug in chan opening
Change snapshot version to something that should compile
