* Create base traits for async tests called BaseAsyncTest, and then create two subtypes, BitcoinSAsyncTest and BitcoinSAsyncFixtureTest to make it easier to manage resources related to test cases. Now it is easier to shutdown things like an actor system in a structured manner
* Remove two bitcoind tests that don't make sense, 'duration' in the parameter is _really_ an interval (bad naming by usTesting an interval does not make sense inside of TestRpcUtilTest
* Turn log off for ChainAppConfigTest we expect to fail
* Bump timeout for BitcoinSUnitTest, use builtint sbt scalafmtCheck
* Remove hardcoded scalafmt binary
* reduce generator size to attempt to fix#768
* Remove AKKA_CONFIG that was preventing sane logging for akka, move db-commons reference.conf to testkit so we can control testkit applications, and then rename app/server/.../application.conf to reference.conf like good libraries should do
* Update contributing.md
* update a few more links inside of contributing.md
* Run scalafmt
* Update scalafmt to 2.0.1
* Scalafmt
In this commit we
1) Reconfigure the .scalafmt.conf file
2) Add a scalafmt binary (same version as specified in conf file)
3) Use said binary on CI to check that everything is formatted correctly
* Ran scalafmt
* Update scalafmt ignore
* Add scalafmt plugin to project/plugins.sbt, run it
* run scalafmt
* This is for colin
* Fix remaining errors on V18 RPC calls
* remove unused code final formatting fix
* implemented fixed listwalletdir rpc test
* responding to code review and replacing flatmaps
* Remove BlockchainBuilder, speed up ChainHandler test that processes mainnet headers, add test for ChainAppConfig
* Add test for BlockHeaderDAO.getAncestorAtHeight()
* Consolidate ChainAppConfigTest, add ScalaTestUtil to help with working with scalatest, speed up ChainHandlerTest a bit more
* Build and CI improvements
In this commit we:
1) Parallelize the Travis CI config, by splitting
each project into its own Travis task
2) Download bitcoind binaries through sbt
* Use binaries downloaded by sbt task
* Make BitcoindRpcTestUtil work on Travis without bitcoind on PATH
* Add new downloadEclair task to sbt
* use sbt downloaded binaries in tests
* Fix Eclair and Bitcoind tests
* Bump Scala versions
Support Scala 2.12.9
and 2.13.0.
To make this easier, we delete the `scripts` project. Everything
that was in here was covered by content on the website. We also
delete the `doc` folder, as that was a remnant from when `scripts`
was called `doc`.
* Crib uPickle akka-http support while we wait for publish
* Fix compiler warnings
* Add note on test logging to contribution guide
* Reduce duplication in Blockchain implementation
* Use Scala 2.12 for website
* Introduce compat package object for collections converters
* Fix Either compiler warnings
* Add sync-chain and create-wallet docs from deleted scripts
* Fix rebase goofup
* Split app loggers into their corresponding modules
This avoids a bunch of places with awkward implicit resolution.
We also make sure that all logging in node, chain and wallet
happens with the app loggers, and not BitcoinSLogger
* Tune test logging level to WARN
* Start refactoring BroadcastTransactionTest to use new SpvNodeFundedWalletBitcoind fixture
* Double timeout to 10.seconds on spv node disconnect
* Attempt to bump disconnect even more
* Add more logging around starting and stoping spv node to try and debug failures with fixtures on test suites
* move starting of spv node into one place in the fixtures -- createSpvNode() --, also look at ordering of destruction to make sure that the spv node is stopped/destroyed BEFORE bitcoind is stopped/destroyed. If the inverse order happens, our spv node won't receive a response from bitcoind and will hang waiting for the Tcp connection to close gracefully
* Try switching from to to get CI to pass
* Add more logging and throws to try and debug failures on ci
* Add logging in the case we send a close command and peer isn't defined
* Match on Tcp.Connect on default receive in P2PClient
* Make SpvNode.disconnect() check to see if we are connected before attempting to disconnect
* Add more logging in fixtures, change behavior of SpvNode.disconnect() to not throw if we are already disconnected
* Switch to preferred way of terminating actor system in tests
* Make PeerMessageSender.sendMsg() return a , make sure a message is not sent to P2PClient until we are fully intialized
* Switch p2p logging to DEBUG
* Complete disconnect promise on peer message recv in the case where we didn't have a peer to begin with
* Make SpvNodeTest more robust, fix bug for when we send 'sendheaders' message. We shouldn't do this until our peer has sent us a verack
* Only send 'sendheaders' message to our peer after we receive a 'sendheaders' message from our peer. Hopefully this solves async issues where we were sending 'sendheaders' before our handshake was fully completed with our peer which means we wouldn't get headers sent to us
* Cleanup pt1
* Start the process of refactoring our ChainHandler to be able to avoid database calls on TipValidation
WIP: Begin explicity passing state back and forth in return types of PeerMessageReceiver, P2PClient, , DataMessageHandler. This commit also implements the ability to keep our blockchain completely in memory. Previously when we were updating the tip of the chain, we had to make a database read to figure out what the best tips are. This is suboptimal for performance because a database read needs to be done for every block header we see, now we just keep the chain in memory
Fix bug in DataMessageHandler that pre-emptively sent a getheadersmsg to our peer. Make 'chainApiF' internal to our spvNode (not a parameter). This forces the chainApi to be created from disk everytime a new SpvNode is spun up. This keeps us in sync with the blockchain at disk at the cost of disk access and less modularity of SpvNode
Address torkel code review
Fix rebase issues
Address code review
Address nadav code review
* Rebase onto master, fix api changes
* Address issue #673, also refactor fixture stuff to be in companion objects rather than traits so they can be re-used without extending the trait itself. This allows for more modularity with our fixtures
* fix bugs in unit tests, address #675 to make MerkleBufferTests faster
* Refactor test timeout in UpdateBloomFitlerTest
* fix callback logic bug on matching unrelated txs/addresses in UpdateBloomFilterTest, re-order broadcasting of things to avoid async bugs hopefully
* Make a def in NodeUnitTest, this keeps tests in the same suite from using the same config. This allows us to write multiple tests per suite, instead of just one. This also adds implicit parameters to our fixture constructors/destructors to properly create and destroy this config. The long term goal here also should be getting rid of config.initialize() we are calling everywhere as this is a anti-pattern, fixtures should take care of construction and destruction of things
* Broadcast tx earlier in test case for UpdateBloomFilterTest
* Rework NodeWithWalletTest to use the new fixtures we have in the node project, now use SpvNodeFundedWalletBitcoind
* Add functionality for updating SPV node bloom filter
* Add SPV node shutdown to exit hook
* Clean up traits/case classes
* Change fixture in WalletBloomTest
* Fix logging bug in TransactionProcessing
* Add MerkleBuffers
In this commit we add MerkleBuffers, which is an object
that lets us aggreagate merkleblocks with their corresponding
transactions before sending them out. This is global, mutable
state (bad!) but it's a working solution for now;
* Use TestAsyncUtil
* Add MerkleBuffers test
* Send getdata if receiving single header
* Change awaitSync to use block count
* Fix UpdateBloomFilterTest
* Add more logging of chain/headers validation
* Send getdata for all blocks
* Nits: Scaladocs, comments toString
* Add logging to data directory
In this commit we add the ability for the node, chain
and wallet projects (+ the server) to log to the users
data directory instead of whatever directory the
binaries was launched from. This is inherently a bit
more complicated than our previous setup, because
we need to read the user directory before we can create
loggers. As a result of this, some files/methods were
moved around, so the relevant app config could be
found in scope.
We also introduce several logging categories that can be
tuned individually through user configuration. These logggers
are exposed both as traits that give a field `logger`, or
as methods that return the required logger.
* Add datadir configuration to AppConfig
In this commit we add support for AppConfig to pick up
the data directory configuration file. We also add
a section to the contributing guide on how to tune
logging levels.
* Pass data directories explicitly for configuration
* This PR is attempting to remove the publishing errors when publish
locally many of which are incorrectly linked things in scaladocs which
there are some differences in formatting. Hopefully this removes alot
if not all of the "link not found for ..." warning messages that show
up.
* Add functionality for broadcasting TXs to node
In this commit we add functionality and tests
for broadcasting a TX from our node. To accomplish
this we introduce a table over broadcastable TXs
that's added to when the externally facing method
broadcastTransaction(tx) withing SpvNode is called.
We send out a inv message for the TX we just added,
and upon receiving a getdata message we search in
the previously mentioned table for entries where
the hashes match up.
* Broadcast TX from server to SPV node
* Perform assertions on the balance of bitcoind after sending a TX
* Remove typeclass from broadcast TX
* Refactor withFundedWalletAndBitcoind
* Match on BitcoindExecption instead of throwable
* Clean up broadcast functionality after code review
* Cleanup
In this commit we cleanup a few Scaladocs and change some
trait/impl pairs into just a case class.
* Node cleanup
In this commit we
1) Rename Client to P2PClient. Client is a very generic name.
2) Clean up some Scaladocs
3) Remove some unecessary objects/traits/classes and replace them
with case classes
4) Add trace logging of bytes received in P2PClient
* PoC bitcoin-s-cli
* Add CLI, Server sbt projects, remove Ammonite
In this commit we set up sbt configuration for
CLI, Server (in-work-name) and corresponding
test projects.
We also remove Ammonite shell from sbt, as that
isn't really being used. bloop console offers
the same functionality way more ergonimic.
* Move BitcoinSAppConfig into new server project
Server project depends on node, chain wand wallet
so this is a good time for introducing this class
into main sources. We also introduce
BitcoinSTestAppConfig in testkit, to replace the
functionality in BitcoinSAppConfig related to
tests.
* Type chain in blockchainresult
* MVP server setup for node, chain and wallet
* Extremely dirty CLI for interacting with server
* initial attempt at mimicking Bitcoin Core API
* WalletStorage: add method for checking for seed existance
* Check for seed existance on wallet startup
* Fix bug where MnemonicNotFound was not an error
* Segregate confirmed and unconfirmed balance methods
* Add error handling, improve formatting of CLI output
* Tweak build
Bump Sttp version, downgrade to uPickle 2.11 compat,
skip publish in cli-test and server-test
* Add CLI, server and picklers to root project
* Split wallet functionality into multiple traits
In this commit we refactor LockedWallet into multiple traits
that provide functionality related to a subset of total wallet
functionality. This has the benefit of making it clear which
methods are helper methods that are only intended to be used
in a very specific setting, and which methods are part of the
internal wallet API that other parts of the wallet can use.
* Rework TransactionOutput and TransactionOutPoint to case classes
* Add extension methods for flattening lists of assertions
* Segregate confirmed and unconfirmed balance methods
* Add test for FutureUtil.sequentially
* Add trace logging of balance fetching
* Process outgoing TXOs
Move TX processing into separate trait, add internal API method
Unify DB representation of TXOs
Prior to this commit we stored TXO information
across diferent tables, with joins and tuples
needed a bunch of places to keep track of
everything we needed. In this commit we unify
the tables, leaving us with only one table for
TXOs.
In this commit we replace addUtxo with
processTransaction in the wallet API. addUtxo
is kept around, but only as an internal
implementation detail.
The new flow looks like this:
1) The wallet gets notified about a new TX
through a processTransaction call
2) It then searches through our already
processed incoming TXs:
if found, updates the confirmation
status
else, inserts it into the DB
3) We then call addUtxo, if the previous
step resulted in a new TX in our
DB
In this commit we update our AesCrypt
functionality to work with AES keys rather
than passphrases that are extended to keys.
This makes it easier to integrate with other
languages, as the key stretching introduces
a bunch of additional parameters to the
actual AES encryption.
We also include tests that verify that we're
able to encrypt/decrypt using Bitcoin-S,
Crypto-JS (JavaScript), pycrypto (Python)
and OpenSSL CLI.
* Attempt to sync with sendheaders
Fix sync with headers by generating one block in regtest to get out of IBD
Fix comment
Add more comments
Add missing actor system import
more clean up
* refactor test case to avoid sleeping the thread
* Log messages more granularely in node
* Add pong message
* Rework P2P service identifier
The old implementation didn't catch the notion
that service identifiers was a bit field where
multiple services could be set at the same time.
* Make Peer take InetSocketAddress
Previously it took NetworkIpAddress.
This doesn't make any sense, as we
need to know the services bitfield
to construct a NetworkIpAddress.
* Clean up logs and toString methods
* Clean up node README and main method
* Make all networks final case objects
* Skip publishing of wallet, node and chain
* Clean up logging of node and chain modules
* Add logging of max height in SpvNodeMain
* Clean up Scaladoc and toStrings
* Add logging configuration as symlinks in node
We'll have to figure out a more stable solution to
configure logging for the SPV node, but for the time
being this is an acceptable solution.
* Fix 2.11 compiler error
* Implemented GCS (WIP)
Added some of Chris' tests for golombEncode and golombDecode
Fixed GCS.golombDecodeSet
Separated GCS object from GolombFilter class
Added hanging test for set encoding
Fixed property test for encoding and decoding arbitrary sets
Added test for GCS.buildGCS and hashing function consistency
Added method to construct a Block Filter from a Block
Added scaladocs in GCS.scala
Added TempUtxoProvider to make BlockFilter testable
Wrote a test for GolombFilter.matchesHash
Added test for arbitrary data matching in bip 158 filter
Added optional right-padding on GCS BitVector, BlockFilter.fromHex, and got started on BlockFilterTest
Cleaned up Bip158TestCase.fromJsArray
Fixed parsing of test vectors for bip 158
Wrote test for BlockFilter (doesn't pass)
Responded to some code review
Included CompactSizeUInt in filter
Created BitcoinScriptUtil.getDataTokens for output serialization
Wrote test for BitcoinScriptUtil.getDataTokens
Added test for BitcoinScriptUtil on the genesis block
Fixed name
:(
* Fixed SipHash and block serialization!
* Fixed bugs relating to empty scripts, all tests passgit push --force-with-lease nadav 2019-05-28-bip158 !
* Cleaned up BitcoinScriptUtil.getDataTokens and its test
* Responded to code review
* Factored out common code from BlockFilter apply methods
* Some cleanup
