Commit Graph

1410 Commits

Author SHA1 Message Date
Torkel Rogstad
0621088f41 Refactor TrezorAddressTest 2019-06-20 14:52:13 +02:00
Torkel Rogstad
5819f4016e Trezor test vectors
In this commit we add:

1) A program for dumping a bunch of address
   information from a running Trezor
2) A test in the wallet module that verifies
   that our addresses line up with the ones
   Trezor generates
2019-06-20 14:29:50 +02:00
Torkel Rogstad
3fe9a466b1 Add createNewAccount to wallet 2019-06-20 14:26:49 +02:00
Torkel Rogstad
0fb5af2899 Clean up versions.js 2019-06-20 13:55:29 +02:00
Torkel Rogstad
b0bb5adf22 Add website version for 0.1.0 2019-06-20 13:48:12 +02:00
Torkel Rogstad
0ffee0097e Add versions.js to website 2019-06-20 13:48:12 +02:00
Chris Stewart
a716d374ca Attempt to sync with sendheaders (#537)
* Attempt to sync with sendheaders

Fix sync with headers by generating one block in regtest to get out of IBD

Fix comment

Add more comments

Add missing actor system import

more clean up

* refactor test case to avoid sleeping the thread
2019-06-20 05:58:25 -05:00
Nadav Kohen
67fb821b00 Implemented BIP 157 Block Filter Headers (#532) 2019-06-17 14:52:10 -05:00
Torkel Rogstad
a87858247c Service identifier and node cleanup (#522)
* Log messages more granularely in node

* Add pong message

* Rework P2P service identifier

The old implementation didn't catch the notion
that service identifiers was a bit field where
multiple services could be set at the same time.

* Make Peer take InetSocketAddress

Previously it took NetworkIpAddress.
This doesn't make any sense, as we
need to know the services bitfield
to construct a NetworkIpAddress.

* Clean up logs and toString methods

* Clean up node README and main method

* Make all networks final case objects

* Skip publishing of wallet, node and chain

* Clean up logging of node and chain modules

* Add logging of max height in SpvNodeMain

* Clean up Scaladoc and toStrings

* Add logging configuration as symlinks in node

We'll have to figure out a more stable solution to
configure logging for the SPV node, but for the time
being this is an acceptable solution.

* Fix 2.11 compiler error
2019-06-17 14:27:51 -05:00
Nadav Kohen
c89104bfda BIP 158 Golomb-Rice Coded Sets and block filters (#481)
* Implemented GCS (WIP)

Added some of Chris' tests for golombEncode and golombDecode

Fixed GCS.golombDecodeSet

Separated GCS object from GolombFilter class

Added hanging test for set encoding

Fixed property test for encoding and decoding arbitrary sets

Added test for GCS.buildGCS and hashing function consistency

Added method to construct a Block Filter from a Block

Added scaladocs in GCS.scala

Added TempUtxoProvider to make BlockFilter testable

Wrote a test for GolombFilter.matchesHash

Added test for arbitrary data matching in bip 158 filter

Added optional right-padding on GCS BitVector, BlockFilter.fromHex, and got started on BlockFilterTest

Cleaned up Bip158TestCase.fromJsArray

Fixed parsing of test vectors for bip 158

Wrote test for BlockFilter (doesn't pass)

Responded to some code review

Included CompactSizeUInt in filter

Created BitcoinScriptUtil.getDataTokens for output serialization

Wrote test for BitcoinScriptUtil.getDataTokens

Added test for BitcoinScriptUtil on the genesis block

Fixed name

:(

* Fixed SipHash and block serialization!

* Fixed bugs relating to empty scripts, all tests passgit push --force-with-lease nadav 2019-05-28-bip158 !

* Cleaned up BitcoinScriptUtil.getDataTokens and its test

* Responded to code review

* Factored out common code from BlockFilter apply methods

* Some cleanup
2019-06-17 11:17:20 -05:00
Torkel Rogstad
435aef7b0b Pass bloom filter to SPV node (#514)
* Add getAddressInfo to Wallet API

* Add Bloom filter and event callbacks to SPV node

* Move wallet test trait to testkit

* Test for connecting SPV node with wallet

* Tune logging in SPV node

* Clean up BloomFilter

* Proper toString in inventory and inventorymessage

* Actually pass in callbacks to SPV node

* Fix rebase screwup and partially respond to code review

* Default to file based databases in tests

This is due to issues with deadlocks with
in-memory based databases. We need DBs
to stay alive between connections, but
not across tests.

In DB intensive chain validation tests we
use in-memory databases instead. This
seems like a reasonable tradeoff between
simplicity and speed.

* Make SpvNodeCallbacks contain sequence of functions within

* Make Bloom filter non-optional in SpvNode
2019-06-17 06:48:26 -05:00
Torkel Rogstad
d30a03a50e Revamp README (#513)
* Revampt README

Users are now deferred to Bitcoin-S.org for instructions
on how to use the library. Added a logo that'll hopefully
appear in social media sharing posts.

* Remove IRC, add Maven Central badge to README
2019-06-17 05:53:58 -05:00
Torkel Rogstad
f525f5b53e Merge pull request #519 from rorp/kill_all_bitcoind_instances_after_running
[Tests] Kill all bitcoind instances after running
2019-06-17 09:38:13 +02:00
rorp
aa7f9012af Fixed ClientTest 2019-06-14 10:28:44 -07:00
Torkel Rogstad
fa84c11f52 Default to file based databases in tests (#517)
This is due to issues with deadlocks with
in-memory based databases. We need DBs
to stay alive between connections, but
not across tests.

In DB intensive chain validation tests we
use in-memory databases instead. This
seems like a reasonable tradeoff between
simplicity and speed.
2019-06-14 11:10:36 -05:00
Torkel Rogstad
8fb3d4542b Fix issue where bloop didn't pick up test resources' (#518) 2019-06-14 10:25:58 -05:00
rorp
3a1c9284dc [Tests] Kill all bitcoind instances after running 2019-06-13 19:22:56 -07:00
Torkel Rogstad
6f003e09fb Merge pull request #508 from Christewart/2019-06-10-bump-deps
Bump various dependencies versions for our libraries
2019-06-13 15:06:38 +02:00
Torkel Rogstad
8d141ac450 Generate bloom filters from wallet (#501)
* Add findAll() to CRUD

* Add getAddressInfo to Wallet API

* Insert pubkey into bloomfilter also inserts hash

In this commit we also simplify parts of
BloomFilterTest, and move what used to be in
BloomFilterSpec into BloomFilterTest.

* Add bloom filter generation to wallet

* Add bloom false positive rate to wallet config

* Add logging to SQL errors

* Fix a bug where HDPath.next returned the wrong path

* Add FutureUtil.sequentially

* Make bloom filter size calculation more explicit

* Return Vector of pubkeys instead of Seq

* Verbose handling of address fetching in test
2019-06-12 09:29:20 -05:00
Torkel Rogstad
35b1d726fb Merge pull request #511 from torkelrogstad/2019-06-11-website-guides
Website guides on HD key and address generation
2019-06-11 18:06:13 +02:00
Torkel Rogstad
34a06258ed Cleanup TransactionTest 2019-06-11 18:01:50 +02:00
Torkel Rogstad
6e5ca0eeeb Add sections on generating addresses and expand HD section in website 2019-06-11 18:01:44 +02:00
Torkel Rogstad
6de0ba268e Bump P2P + crypto test coverage (#506)
* Rename P2P tests and move generators into correct package

* Add more tests for P2P messages

* Add note on coverage to contributing guide

* Replace network in P2P header with typed network

* Add more P2P tests

* Add more tests for ECPriv and ECPub
2019-06-11 10:01:46 -05:00
Chris Stewart
e2ec692779 bump play-json version to 2.7.2 2019-06-11 08:47:34 -05:00
Chris Stewart
adfeff504f Bump various dependencies versions for our libraries 2019-06-11 06:39:28 -05:00
Torkel Rogstad
2723ef6f5d Move Logback config into test resources (#509)
I though I had solved our logging problem:
    We want projects to pick up logging config
    in core, without it being published.

The previous attempt was filtering out XML files
from unmanagedResources. That caused the XML
files to not get published, and when running
tests through Bloop everything worked fine.
However, when running them through sbt the
config did not get picked up. I suspect this
is because of a bug in how Bloop exports sbt
builds.

In this attempt we instead move the XML files
to test resources in core, and then make the
test projects depend on test sources in core,
with the awkward syntax 'core % "test->test"'.
2019-06-11 06:38:42 -05:00
Torkel Rogstad
36766089f6 Merge pull request #510 from torkelrogstad/2019-06-11-blog
Docs: removes blog from website
2019-06-11 13:17:33 +02:00
Torkel Rogstad
6b5c5e581c Docs: removes blog from website 2019-06-11 12:59:58 +02:00
Torkel Rogstad
03be955717 Refactor logback configuration to reduce repetition (#499) 2019-06-07 07:29:40 -05:00
Torkel Rogstad
af47809300 Add Scaladoc title and version number (#502)
* Add version and name to Scaladoc

* Compile website on Travis as part of CI
2019-06-07 07:23:32 -05:00
Torkel Rogstad
1ff0dee934 Reshuffle package location of P2P messages (#495)
* Reshuffle package location of P2P messages

Previously the companion objects for P2P messages
and the actual messages were in different pacakges.
This made them more awkward to use than necessary.

We also clean up some Scaladocs in this commit.

* Move all P2P messages to core
2019-06-06 09:13:32 -05:00
Torkel Rogstad
bd10f1c35e Merge pull request #497 from torkelrogstad/2019-06-05-legacy-addresses
Legacy addresses
2019-06-06 16:08:49 +02:00
Torkel Rogstad
5dbfefc7ed Respond to code review on legacy address generation 2019-06-06 15:10:20 +02:00
Torkel Rogstad
346a8a2bcb Merge pull request #498 from torkelrogstad/2019-06-06-bitcoind-debug
Add debug prints to bitcoindCall when not on mainnet
2019-06-06 13:25:27 +02:00
Torkel Rogstad
15da0d3b4c Add debug prints to bitcoindCall when not on mainnet 2019-06-06 13:25:09 +02:00
Torkel Rogstad
8a338d542f Create all root level accounts on wallet creation 2019-06-05 20:01:59 +02:00
Torkel Rogstad
42c161d3e3 Fix bug when overriding configurations
This commit fixes (and adds a test for) a bug
where overriding a value in our configuration
reset the rest of the configuration to the
default values.
2019-06-05 18:37:38 +02:00
Torkel Rogstad
6fa5ae85d8 Add legacy addresses to wallet
In this commit we add the ability to generate
legacy addresses in the wallet. We also make
the default accound kind (legacy, segwit,
nested segwit) configurable.
2019-06-05 18:36:57 +02:00
Torkel Rogstad
3c4157a698 Multi module configuration (#494)
* Replace AppConfig with more specific types

* Rework database configuration

In this commit we:

1) Introduce new database settings for all submodules,
and at the same time remove the singular datbase setting
for the entire project
2) Introduce a new method .initialize on all configs
that is responsible for creating needed directories,
databases and files
3) Introduce a new class BitcoinSAppConfig that wraps
configs for all three other subproject config. We
also provide implicit conversions that enable this
super-config to be passed in wherever a specialized
config is required.
4) Add more tests for our configuration setup.

* Add Ammonite to Docs deps
2019-06-05 10:48:15 -05:00
Torkel Rogstad
49ffbea801 Make TestAsyncUtil serializable (#493)
Every now and then the Scalatest framework
has crashed when reporting errors, with
failure to serialize this. There should
be no downside to having the trait here,
and we avoid the nasty Scalatest failure.
2019-06-05 07:18:17 -05:00
Chris Stewart
a3dd824e57 Node (#490)
* WIP: 2018 12 22 node project (#280)

* Add files from old spv node project

src compiling

test files compiling

ran scalafmt

Fix serializer tests

Get non networking test cases to work

WIP: Debug peermessagehandler

Update CRUD, remove all of the Actor craziness. Add DbManagement trait and unit test db

WIP: Rewroking PeerMessageHandler, create Peer, DataMessageHandler, PeerHandler

Reworking Client to handle all tcp messages and message alignment for bitcoin p2p messages

* Wip: Node refactor

* Create node test project, move all node tests into that project and move all generators for the node project into testkit

* Rework ClientTest to use testkit, start minimizing akka usage, implement connect(),isConnected(), disconnect(), isDisconnected() in PeerMessageReceiver

* Create Peer, PeerHandler, PeerMessageSender and PeerMessageReceiver

* update readme about status of node project (#359)

* Add flyway plugin to manage database schemas (#361)

* Add flyway plugin to manage database schemas

* Switch database driver to sqlite3 to be more portable, rework configs for sqlite3

* Set up sqlite database directories and files if they are not already created

* Add torkel's review

* Add chain, wallet, db-commons projects (#367)

* Add chain, wallet, db-commons projects

* Rework db creation logic if they db does not exist

* Add config logging to try to debug travis ci

* Pass explicit class loader for db config

* Remove duplicate call to dbConfig

* Make DbConfig.dbConfig a lazy val

* Remove noisy log

* Add scaladoc to DbConfig

* Switch dbConfig readme paragraphs

* Fix compile issues introduced during rebase onto master with rpc changes (#394)

* WIP: 2019 03 12 tip validation (#378)

* Implement blockchain handling data structures

Add TipValidation happy path

Add more test cases for TipValidation.checkNewTip for badPrevBlockHash and badPOW

Add overflow check, fix endianness bug for checking proof of work

Add pow transition check, refactor difficultyChangeInterval into chain params, add more tests

fix a few nits

Fix compile error, clean up unused import

Remove redundant files from node project

* Implement GetNextWorkRequrired/CalculateNextWorkRequired, move BlockHeaderDAOTest cases into chain project

* Add full POW change check in TipValidation, address code review nits

* Configure logging in chainTest, turn logging OFF in other test projects

* Address code review pt2

* Add coverage minimum for chain project (#398)

* Add coverage minimum for chain project

* Add first Blockchain.connectTip() unit test, switch to a in memory sqlite database for unit tests, starting using fixtures for BlockHeaderDAO in unit tests

* Add tests for ChainHandler.processNewHeader(), ChainHandler.getHeader(), Blockchain.connectTip(). Refactor redundant configurations being passed around excessivly

* Address code review, fix a flaky test in ClientTest.scala

* Test Fixtures (#403)

* Working test fixtures

* Removed ChainTestFixture trait in main code

* Composing Fixtures (#413)

* Downloaded over 9000 mainnet BlockHeaders into a json file

* Added new fixture with populated blockHeaderDAO

* Split writing to db into batches

* Rebased

* Simplified fixtures with makeFixture abstraction

* Added util functions for composing builders

* Add integration test between bitcoind <-> zmq <-> bitcoin-s-chain project. Test that we can relay a header from bitcoind over zmq into the bitcoin-s chain project. Redo ZmqConfig to use InetSocketAddress

* Address code review

* wip

* A compiling withBitcoindZmqChainHandler fixture

* Tests passing!

* Made blockHeaderDAO private

* Got 9000 new block headers from 562375 to 571375

* Added offset to populated blockHeaderDAO fixture

* Added scaladocs to fixture things

* Initial wallet: import UTXO and spend it (#391)

* Updates ExtKeyVersion with fromChainParams method

* Add equals to Address

* Update BIP44 classes

* Add ScriptType

* Initial work on wallet support

* Add foreign keys pragma for SQLite

* Add UTXO models and DAO

* Add addres P2WPKH generation and WIP for addUTXO

* Add logging config for wallet

* Add change address generation, proper-ish addUtxo and sendToAddress

* Address code review on #391

* Add empty AES passphrase invariant

* Add poor mans test fixtures

* Add listUtxos, listAddresses and getBalance to wallet API

* Use fixtures from chain project

* Fix CI test failures

* Fix broken up package path

* Updates bloop config for new projects (#424)

* Multi fixture file (#419)

* Created FixtureTag and ChainFixture

Used ChainFixture in BitcoinPowTest

Added implicit conversions for nice syntactic sugar

* Added documentation for multi-fixture

* Made defaultTag a val

* add a logback-test.xml to the wallet project (#433)

* Introduce AppConfig that combines ChainParams and DbConfig (#432)

* 2019 04 23 app config per project db config per project (#434)

* Add DB conf file resolution that works across projects

* Create applicatoin configurations for specific projects, rework DbConfig structures for individual projects. Force network to be mixed into DbConfig rather than DbConfig to be mixed into the network

* Add ammonite to db-commons, remove noisy logs

* remove mixin for DbConfig that required a NetworkDb. Now networkDb is just a field on 'DbConfig', this simplifies things downstream type wise when interacting with the projects AppConfig. This commit also removes a parameter from AppConfig, now only a DbConfig needs to be passed in, and we can derive the network and chain params from the DbConfig. The only exemption is UnitTestDbConfig as it is sometimes handy to specify a different network (i.e. mainnet) when testing

* Turn DbConfig objects to case objects, wrap those case objects in their parent type companion object

* remove cast in Wallet.scala

* Add EnhancedEither class for 2.11 compat (#437)

Add implicit conversion from Either to
2.11-compatible Either-wrapper. Also
remove trailing comma in WalletTestUtil
that breaks 2.11 build.

* Fix CI tests hanging (#438)

* Execute wallet tests sequentially to avoid SQLite deadlocks

* Refactor logback config to reduce duplication

* Use in-memory SQLite DB for unit tests

* Debug prints for DatabaseConfig.forConfig

* Fork JVMs in test to ensure proper in-memory DBs

* Pass in Akka config to Eclair tests, avoid cluttering Akka log output

* Don't fork JVM on node tests'

* Move things out of ChainUnitTest (#448)

* Move things out of ChainUnitTest

* Remove printlns

* 2019 04 29 client test (#449)

* Bump timeout on connect to node test

* Change from isConnected -> isInitialized to avoid the error trying to disconnect before we are fully initialized

* Wrote tests for POW difficulty change calculation and header processing (#429)

Fixed BitcoinPowTest

Rebased onto AppConfig code

Rewrote ChainHandler integration test

Made chain handler test synchronous

Fixed a couple test bugs

Implmented a more efficient getAncestorByHeight

Fixed ChainHandler integration test by using the correct starting conditions

Responded to code review

Responded to more code review

Deleted redundant Pow test

Made BlockHeaderDAO.getAncestorAtHeight use a List for its loop to improve performance

* WIP: Create ChainSync, BitcoindChainHandlerViaRpc, add simple ChainSyncTes… (#450)

* Create ChainSync, BitcoindChainHandlerViaRpc, add simple ChainSyncTest to sync one block from a external bitcoind instance via rpc

* Add check for having the best block hash in our chain state already

* Fix prev block hash to be empty hash if genesis block header

* BlockchainBuilder (#439)

* First commit for implementing a BlockchainBuilder

* use Builder rather than ReusableBuilder to be compatible with scala 2.11.x

* Decouple Blockchain & BlockHeaderDAO

* Rebase onto node, incorporate changes in #429

* Add more comments

* Reverse order of headers in builder

* rebase onot node branch, refactor apis

* DB: Add utility method for listing tables in a DB (#447)

* Node rebase (#458)

* Implement BIP32 path diffing

* Rebase node onto newest HD changes in master

* Fix 2.11 compile errors

* 2019 05 01 wallet ammonite scripts pt2 (#452)

* wip -- not finding testkit in doc worksheet

Wip -- classdef not found for create-wallet.sc

zmq bug

Clean up some logs

nest zmq start in bitcoindF

update jeromq to 0.5.2-SNAPSHOT to get rid of annoying log to stdout

Rebase onto node branch with new configs

Successfully running ammonite script create-wallet.sc

2019 05 01 wallet ammonite scripts pt2 (#25)

* Refactor Ammonite dep

* Add basic error handling in AmmoniteBridge

* Add very basic README for doc project

Fix compile issues after rebasing onto master

Add code to sync our wallet code with bitcoind after creating a tx

* refactor ZMQSubscriber to _hopefully_ avoid hanging when we call context.term(). We do this by closing the socket before calling context.term() and using socket.setLinger()

* Update doc/src/main/scala/org/bitcoins/doc/wallet/create-wallet.sc

Co-Authored-By: Christewart <stewart.chris1234@gmail.com>

* 2019 05 05 sync chain (#460)

* Add code to sync our wallet code with bitcoind after creating a tx

Add script to illustrate how the chain persisted and how to sync against a running bitcoind instance on regtest

* fix bug relating to subtraction operator not being communative in Pow.getNextWorkRequired(). This kept us from being able to switch proof of work intervals correctly

* rename script from persist-chain.sc -> sync-chain.sc

* fix 2.11.x compile issues

* Refactor chain, node, wallet config (#463)

* Refactor chain, node, wallet config

Get rid of NetworkDb, DbConfig
Add proper structure to conf system,
moving everything under the bitcoin-s
root key.

* Remove Scalacheck from node project

* Add doc on configuration

* Add override feature to AppConfig

* Address code review in #463

* Throw if default data dir is used in tests, add Scaladoc to AppConfig

* Add explanations for withOverrides, link to configuration.md from AppConfig

* Fix compile error

* Moves chain fixtures to testkit project (#475)

reset node files

* Store encrypted mnemonic to disk (#462)

* Add WalletStorage object

* Add encrypted mnemonic storage, locked wallet

Add lock and unlock operations to wallet.
Separate between locked and unlock wallet.

* Handle non-existant seed file

* Respond to code review from Chris

* Use val instead of import

* Add doc on how mnemonics are encrypted/stored

* 2019 05 15 spv sync headers (#479)

* Implement SpvNode skeleton, create NodeUnitTest and move it to the testkit

* Implement test case to sync a header via spv into bitcoin-s

* Fix compiler errors

* Make node project Main runnable (#26)

* Add logging configuration to node project

* Make default config workable in non-test environments

* Add more logging of config in BH DAO and AppConfig

* Make Peer id optional

* Make node Main.scala runnable

* Implement Main.scala to sync with a locally running bitcoind instance. You can now run with 'bloop run node' and sync the node if you adjust the parameters inside of Main.scala. This also reworks the structure of 'AppConfig'. It turns the *AppConfig into a case class intead of case objects. This allows us to pass custom configs into those case classes

* Address code review from torkel

* Reintroduce withOverrides (#29)

* Turn off chain validation logs

* Make datadir a parameter to bitcoind config rather than having it implicitly written to the bitcoin.conf file. This was a difference that was occurring in the node branch which had a parameter for the datadir and master which was implicitly writing it to bitcoin.conf

* Add ability to overrwrite conf file except in the case of overwriting the DEFAULT_DATADIR & DEFAULT_CONF

* remove extra Bitcoind.stopServers in WalletIntegrationTest
2019-06-04 09:53:00 -05:00
Torkel Rogstad
8179ac6090 Docs: Merge pull request #489 from torkelrogstad/2019-06-03-website-ci
Docs: conditional Travis based on commit message
2019-06-03 11:29:23 +02:00
Torkel Rogstad
2815243e58 Docs: conditional Travis based on commit message 2019-06-03 11:18:19 +02:00
Torkel Rogstad
f387aaf7b1 Merge pull request #487 from torkelrogstad/2019-05-31-publish-website
Publish website
2019-06-03 10:29:19 +02:00
Torkel Rogstad
a19dc94bef Doc: Add stable and unstable version, try bitcoin-s script 2019-05-31 17:14:15 +02:00
Torkel Rogstad
b14a053bb6 Fix documentation for connecting to bitcoind 2019-05-31 17:14:08 +02:00
Chris Stewart
5256327a56 Downgrade sbt-api-mappings plugin to try and get it to work with mac osx (#486)
* Downgrade sbt-api-mappings plugin to try and get it to work with mac osx

* Change version to 2.1.0
2019-05-30 13:38:36 -05:00
Chris Stewart
8bede3fddc Give sbt-api-mappings plugin a concrete version, fix compiler warnings with unused imports (#485) 2019-05-30 09:47:21 -05:00
Chris Stewart
08b4ee1afe update readmes (#467) 2019-05-30 09:02:03 -05:00
Torkel Rogstad
d3e6e4a65f Add Torkels PGP key to security.md (#483) 2019-05-29 05:26:28 -05:00