Commit graph

54 commits

Author SHA1 Message Date
Nadav Kohen
4d9692f61e
Crypto Project Refactor (#1380)
* Added crypto project and decoupled BitcoinSLogger from NetworkElement

Decoupled BitcoinSLogger from Factory

Moved NetworkElement into crypto project

Moved Factory and BitcoinSUtil (renamed to BytesUtil) to crypto project

Moved MaskedToString to crypto project

Added BytesUtil to imports and cleaned up CryptoUtil.recoverPoint

Moved the rest of crypto stuff to the crypto project

Moved crypto tests to crypto-test project

* Added documentation for crypto project
2020-04-30 12:34:53 -05:00
Ben Carman
38fe580c06
Neutrino Logging (#1382) 2020-04-30 09:05:26 -05:00
Ben Carman
556f713589
Add wallet creation time for rescans (#1353)
* Add wallet creation time for rescans

* Fix docs

* Clean up and add test

* Remove account bday

* Fix compile issue and docs

* Add more chain handler tests

* Use Instant over Long, add docs

* Fix docs
2020-04-29 09:49:41 -05:00
Ben Carman
2194196e76
Re-enable NeutrinoNodeWithWalletTest for Linux (#1366)
* Re-enable NeutrinoNodeWithWalletTest with it disabled for mac

* Change to only Linux

* Only disable for CI runs

* Move IsCI to BitcoinSUtil

* Fix compile issue for earlier versions

* Fix syncing compact filters to not skip last filter

* Fix test to be compatible with new start state

* Use correct isCI

* Fix compile issue

* set CI env
2020-04-27 18:20:47 -05:00
Ben Carman
20c6e43df2
Batch processing compact filters (#1363)
* Batch processing compact filters

* Use one vector, rename param

* Fix compile issue on older versions

* Process sequentially
2020-04-26 09:34:41 -05:00
Chris Stewart
64b13846d9
2020 04 08 multi db dao (#1355)
* Implement Slick's multi database support via our DAOs

WIP2 work on chain project

WIP make JdbcProfieComponent so we have a common JdbcProfile

WIP2

Clean up a few things

WIP got CRUDAutoInc compiling

Get closer to everything compiling in the chain project

Begin moving database code into JdbcProfileComponent

WIP source compiling

Get dbCommonsTest/ test cases passing

Get rest of tests compiling and passing

* Rebase onto master, fix conflicts

* Run scalafmt

* Run scalafmt again

* Cleanup a few compiler warnings by adding type annotations to thigns

* Explicitly cast tables where they are needed in the wallet project where we need TableQuery for foreignkey / joins in slick

* Redo definition of 'table' to use the profile.api.TableQuery in scope

* Rename all *Table.scala files to *Db.scala

* First crack at address ben's code review

* Fix docs compile issue

* Fixed casting (#30)

* Fixed AppConfig casting

* Fixed TableQuery casting in DbManagement subtypes

* Fixed casts in DAOs relating to internal types

* Address nadav's code review, move JdbcProfileComponent into it's own file

* Remove unused listTables() method

Co-authored-by: Nadav Kohen <nadavk25@gmail.com>
2020-04-25 11:28:58 -05:00
Chris Stewart
ce33c57ed9
Bump scala version to 2.13.2 (#1360) 2020-04-23 07:38:20 -05:00
Ben Carman
be37fda2e3
Add sendrawtransaction CLI and Api commands 2020-04-21 13:14:02 -05:00
Chris Stewart
831e89c72b
Use FutureUtil.foldLeftAsync() to process messages we parsed on the p2p network. This moves the blocking with Await.result() from inside of each message we process, to after the entire batch of messages is processed (#1326) 2020-04-20 15:34:14 -05:00
Ben Carman
d6b4ac07a3
Update TxoState for transactions after they've been confirmed (#1178)
* Add onBlockHeadersReceived NodeCallback

* Add requiredConfirmations config option

* Update TxoState for transactions after they've been confirmed

* Response to review

* Fix compile issue
2020-04-08 16:51:17 -05:00
Ben Carman
717434dcaa
Change NodeCallbacks to Future[Unit] s (#1206)
* Change NodeCallbacks to Future[Unit] s

* Fix compile issue

* Change try to future

* Change Future.sucessful(()) to FutureUtil.unit

* Callback failure handling, callback docs

* Create functions for executing callbacks, make merkleblock a future

* Convert MerkleBuffersTest to BitcoinSAsyncTest

* Fix compiler errors

* Reorder UpdateBloomFilter tests so that txFromWalletP is a success for both calls and the other test does not wait on the promise to complete
2020-04-03 08:57:41 -05:00
Chris Stewart
4559edde07
Actually add all files for 0.3.0 on the website so they show up (#1256) 2020-03-19 07:32:02 -05:00
Nadav Kohen
d215680e74
Got rid of warning messages! (#1133)
* Got rid of warning messages!

* Reverted collection conversion changes because scala 2.11 is dumb
2020-02-19 08:02:33 -06:00
Nadav Kohen
a6e21fe43b
Multiple Logger Fix (#1086)
* An attempt to fix logging to file using MDC

* Responded to review
2020-02-11 11:35:28 -06:00
rorp
7fbc6423a5
Fix sync issues (#1090) 2020-02-04 10:16:16 -06:00
Ben Carman
c4ade3ba3c Txo state flyway (#1052)
* Add flyway migrations

* Make different project's migrations independent of each other

* Rework all AppConfig.initialize() to use migrations rather than what we were doing before

* TXO State migration

* Move to new file, drop old column

* Add block hash column

Co-authored-by: Chris Stewart <stewart.chris1234@gmail.com>
2020-01-22 15:34:36 -06:00
rorp
d776e1c952 Automated wallet recovery (#985)
* Automated wallet recovery

* responded to the PR comments

* some more changes

* fix docs

* cleanup
2020-01-02 13:18:41 -06:00
rorp
a31066d17e
Move rescan logic from node to wallet (#974)
* Move rescan logic from node to wallet
2019-12-23 08:57:55 -08:00
rorp
b92fc1cb8f
Compute confirmations dynamically (#938)
* Compute confirmations dynamically
2019-12-14 12:06:22 -08:00
rorp
7cfd33b753 ChainQueryApi (#926)
* ChainQueryApi

* cleanup
2019-12-09 09:39:35 -06:00
rorp
a752c5301f
Neutrino Wallet: OnCompactFilter handler (#905)
* Neutrino Wallet: OnCompactFilter handler
2019-12-06 13:10:33 -08:00
Nadav Kohen
15ec7b89b6 Nicer Satoshis.apply (#899)
* Replaced Satoshis(Int64(_)) with Satoshis(_)

* Fixed SignerTest
2019-11-26 14:43:14 -06:00
rorp
43ec85af39 Rescan RPC (#854)
* Rescan RPC

* cleanup

* fix optional args

* cleanup

* unit tests

* fix build error

* fix unit tests

* scalaMock

* cleanup

* more unit tests

* some more validation checks
2019-11-21 08:03:32 -06:00
rorp
abaa0581c0 Rescan and fetch blocks (#835)
* Rescan and fetch blocks

* unit test

* fix log level

* addessed the PR comments

* fix test timeout

* improve unit tests
2019-10-30 09:51:03 -05:00
Chris Stewart
0b75160455
Run scalafmt, i forgot to run on #783 (#788) 2019-10-07 08:16:28 -05:00
Chris Stewart
09ea1fb3cd
Use the fact that network headers specify the number of bytes in the … (#783)
* Use the fact that network headers specify the number of bytes in the payload rather than just parsing from bytes.size, this should allow us to be more precise when parsing NetworkPayloads rather than _hoping_ that bytes.size does not land on pseudo-valid NetworkPayload. This hopefully resolves #782

* Add safety check around HeadersMessage.toString()

* Add invariant to NetworkMessage saying payloadSize in header must be the actual payload size
2019-10-06 09:48:53 -05:00
rorp
5cc0b30544 Optimize org.bitcoins.chain.blockchain.BaseBlockChain (#781)
* Optimize org.bitcoins.chain.blockchain.BaseBlockChain

* remove toStream calls

* cleanup

* fix IndexOutOfBoundsException
2019-10-06 09:47:02 -05:00
Chris Stewart
6476e3405d
Add a log at INFO level to indicate we are making progress while syncing (#780)
* Add a log at INFO level to indicate we are making progress while syncing

* Adjust sync log message to INFO in p2p code

* Fix off by one error

* Run scalafmt
2019-10-03 14:57:02 -05:00
rorp
a42d297eeb Remove bitcoind dependency from node (#770)
* Remove bitcoind dependency from node

* addded some comments
2019-10-01 14:11:47 -05:00
Chris Stewart
e66bf4c2e0
Remove isInitialized() check in sendMsg, which was causing a deadlock (#763)
* Remove isInitialized() check in sendMsg, which was causing a deadlock

* Enable akka logging to help debug NeutrinoNodeTest

* Revert to bypassing the isInitialized() check in sendMsg()

* Run scalafmt again
2019-10-01 06:25:03 -05:00
Chris Stewart
81dcdbeb57
2019 09 27 logging refactor pt2 (#765)
* Remove AKKA_CONFIG that was preventing sane logging for akka, move db-commons reference.conf to testkit so we can control testkit applications, and then rename app/server/.../application.conf to reference.conf like good libraries should do

* Update contributing.md

* update a few more links inside of contributing.md

* Run scalafmt
2019-09-28 09:50:42 -05:00
Chris Stewart
5d28e9143d
Add akka logging documentation to our contributing.md (#764) 2019-09-27 10:56:16 -05:00
Chris Stewart
3f0506f7c0
Run scalafmt (#762) 2019-09-26 07:31:30 -05:00
rorp
933f0fcfd0 Initial BIP157 support (#695)
* WIP: Initial BIP157 support

* store block hash and hetgh along with its compact filter header

* download and parse block filters

* getcfilters/cfilter

* cfilter table

* rescan full filter chain

* improved rescan performance

* optimize compact headers download

* addressed the PR comments

* split SVP and Neutrino node implementations

* configurable filter batch sizes

* initial filter sync

* addressed comments

* chage filter table's primary key

* fix Golomb filter deserialization

* batch database inserts

* neutrino node test

* fixed node test

* addressed the PR comments

* serializers tests

* cleanup

* fix compilation errors

* fix unit tests

* increase test coverage

* enable NeutrinoNodeTest

* make scalafmt happy

* don't cache experimental binaries

* inclease test coverage

* fix unit tests

* more granular CI tests

* disable NeutrinoNodeTest

* refactor tests

* addressed comments

* test coveage

* fix formatting

* responded to the comments

* some more changes

* fix the build

* test coverage

* revert testnet3 config parameter

* minor changes

* cleanup
2019-09-25 13:18:51 -05:00
Chris Stewart
71a136b03d
Fix bug where we were sending messages before we were fully initialized (#755)
* Fix bug where we were sending messages before we were fully initialized

* Move the pattern match out of isInitialized() future in sendMsg()
2019-09-24 09:43:27 -05:00
Torkel Rogstad
a303818c1e Bump Scala versions (#697)
* Bump Scala versions

Support Scala 2.12.9
and 2.13.0.

To make this easier, we delete the `scripts` project. Everything
that was in here was covered by content on the website. We also
delete the `doc` folder, as that was a remnant from when `scripts`
was called `doc`.

* Crib uPickle akka-http support while we wait for publish

* Fix compiler warnings

* Add note on test logging to contribution guide

* Reduce duplication in Blockchain implementation

* Use Scala 2.12 for website

* Introduce compat package object for collections converters

* Fix Either compiler warnings

* Add sync-chain and create-wallet docs from deleted scripts

* Fix rebase goofup
2019-08-23 13:53:00 -05:00
Torkel Rogstad
8ba9626500 Tweak module logging (#698)
* Split app loggers into their corresponding modules

This avoids a bunch of places with awkward implicit resolution.

We also make sure that all logging in node, chain and wallet
happens with the app loggers, and not BitcoinSLogger

* Tune test logging level to WARN
2019-08-20 08:11:10 -05:00
Chris Stewart
4afdccb0a9
2019 08 05 broadcast tx test (#680)
* Start refactoring BroadcastTransactionTest to use new SpvNodeFundedWalletBitcoind fixture

* Double timeout to 10.seconds on spv node disconnect

* Attempt to bump disconnect even more

* Add more logging around starting and stoping spv node to try and debug failures with fixtures on test suites

* move starting of spv node into one place in the fixtures -- createSpvNode() --, also look at ordering of destruction to make sure that the spv node is stopped/destroyed BEFORE bitcoind is stopped/destroyed. If the inverse order happens, our spv node won't receive a response from bitcoind and will hang waiting for the Tcp connection to close gracefully

* Try switching from  to  to get CI to pass

* Add more logging and throws to try and debug failures on ci

* Add logging in the case we send a close command and peer isn't defined

* Match on Tcp.Connect on default receive in P2PClient

* Make SpvNode.disconnect() check to see if we are connected before attempting to disconnect

* Add more logging in fixtures, change behavior of SpvNode.disconnect() to not throw if we are already disconnected

* Switch to preferred way of terminating actor system in tests

* Make PeerMessageSender.sendMsg() return a , make sure a message is not sent to P2PClient until we are fully intialized

* Switch p2p logging to DEBUG

* Complete disconnect promise on peer message recv in the case where we didn't have a peer to begin with

* Make SpvNodeTest more robust, fix bug for when we send 'sendheaders' message. We shouldn't do this until our peer has sent us a verack

* Only send 'sendheaders' message to our peer after we receive a 'sendheaders' message from our peer. Hopefully this solves async issues where we were sending 'sendheaders' before our handshake was fully completed with our peer which means we wouldn't get headers sent to us

* Cleanup pt1
2019-08-14 07:18:36 -05:00
Chris Stewart
020a935618
Start the process of refactoring our ChainHandler to be able to avoid… (#655)
* Start the process of refactoring our ChainHandler to be able to avoid database calls on TipValidation

WIP: Begin explicity passing state back and forth in return types of PeerMessageReceiver, P2PClient, , DataMessageHandler. This commit also implements the ability to keep our blockchain completely in memory. Previously when we were updating the tip of the chain, we had to make a database read to figure out what the best tips are. This is suboptimal for performance because a database read needs to be done for every block header we see, now we just keep the chain in memory

Fix bug in DataMessageHandler that pre-emptively sent a getheadersmsg to our peer. Make 'chainApiF' internal to our spvNode (not a parameter). This forces the chainApi to be created from disk everytime a new SpvNode is spun up. This keeps us in sync with the blockchain at disk at the cost of disk access and less modularity of SpvNode

Address torkel code review

Fix rebase issues

Address code review

Address nadav code review

* Rebase onto master, fix api changes
2019-08-06 13:31:54 -05:00
Torkel Rogstad
b706e8ea45 Add functionality for updating SPV node bloom filter (#585)
* Add functionality for updating SPV node bloom filter

* Add SPV node shutdown to exit hook

* Clean up traits/case classes

* Change fixture in WalletBloomTest

* Fix logging bug in TransactionProcessing

* Add MerkleBuffers

In this commit we add MerkleBuffers, which is an object
that lets us aggreagate merkleblocks with their corresponding
transactions before sending them out. This is global, mutable
state (bad!) but it's a working solution for now;

* Use TestAsyncUtil

* Add MerkleBuffers test

* Send getdata if receiving single header

* Change awaitSync to use block count

* Fix UpdateBloomFilterTest

* Add more logging of chain/headers validation

* Send getdata for all blocks

* Nits: Scaladocs, comments toString
2019-08-02 09:22:20 -05:00
Torkel Rogstad
31642aff44 Add configurable logging to data directory (#640)
* Add logging to data directory

In this commit we add the ability for the node, chain
and wallet projects (+ the server) to log to the users
data directory instead of whatever directory the
binaries was launched from. This is inherently a bit
more complicated than our previous setup, because
we need to read the user directory before we can create
loggers. As a result of this, some files/methods were
moved around, so the relevant app config could be
found in scope.

We also  introduce several logging categories that can be
tuned individually through user configuration. These logggers
are exposed both as traits that give a field `logger`, or
as methods that return the required logger.

* Add datadir configuration to AppConfig

In this commit we add support for AppConfig to pick up
the data directory configuration file. We also add
a section to the contributing guide on how to tune
logging levels.

* Pass data directories explicitly for configuration
2019-08-01 06:01:56 -05:00
cwaldron97
b6a5ffea4d Cleaning up Publish Logs (#634)
* This PR is attempting to remove the publishing errors when publish
locally many of which are incorrectly linked things in scaladocs which
there are some differences in formatting. Hopefully this removes alot
if not all of the "link not found for ..." warning messages that show
up.
2019-07-31 12:28:22 +02:00
Chris Stewart
aef741adf7
Delete files from the last iteration of our spv node (#656) 2019-07-22 08:39:12 -05:00
Torkel Rogstad
72018bf831 Fix merkle block parsing error 2019-07-19 10:49:34 +02:00
Torkel Rogstad
8125bca7c2 Add functionality for broadcasting TXs to node (#577)
* Add functionality for broadcasting TXs to node

In this commit we add functionality and tests
for broadcasting a TX from our node. To accomplish
this we introduce a table over broadcastable TXs
that's added to when the externally facing method
broadcastTransaction(tx) withing SpvNode is called.
We send out a inv message for the TX we just added,
and upon receiving a getdata message we search in
the previously mentioned table for entries where
the hashes match up.

* Broadcast TX from server to SPV node

* Perform assertions on the balance of bitcoind after sending a TX

* Remove typeclass from broadcast TX

* Refactor withFundedWalletAndBitcoind

* Match on BitcoindExecption instead of throwable

* Clean up broadcast functionality after code review
2019-07-17 07:32:05 -05:00
Torkel Rogstad
ae134f9742 Node cleanup (#591)
* Cleanup

In this commit we cleanup a few Scaladocs and change some
trait/impl pairs into just a case class.

* Node cleanup

In this commit we
1) Rename Client to P2PClient. Client is a very generic name.
2) Clean up some Scaladocs
3) Remove some unecessary objects/traits/classes and replace them
    with case classes
4) Add trace logging of bytes received in P2PClient
2019-07-13 09:26:00 -05:00
Chris Stewart
91633375a3
Attempt to sync with sendheaders (#537)
* Attempt to sync with sendheaders

Fix sync with headers by generating one block in regtest to get out of IBD

Fix comment

Add more comments

Add missing actor system import

more clean up

* refactor test case to avoid sleeping the thread
2019-06-20 05:58:25 -05:00
Torkel Rogstad
02560ebb5f Service identifier and node cleanup (#522)
* Log messages more granularely in node

* Add pong message

* Rework P2P service identifier

The old implementation didn't catch the notion
that service identifiers was a bit field where
multiple services could be set at the same time.

* Make Peer take InetSocketAddress

Previously it took NetworkIpAddress.
This doesn't make any sense, as we
need to know the services bitfield
to construct a NetworkIpAddress.

* Clean up logs and toString methods

* Clean up node README and main method

* Make all networks final case objects

* Skip publishing of wallet, node and chain

* Clean up logging of node and chain modules

* Add logging of max height in SpvNodeMain

* Clean up Scaladoc and toStrings

* Add logging configuration as symlinks in node

We'll have to figure out a more stable solution to
configure logging for the SPV node, but for the time
being this is an acceptable solution.

* Fix 2.11 compiler error
2019-06-17 14:27:51 -05:00
Torkel Rogstad
90b9b6aa9a Pass bloom filter to SPV node (#514)
* Add getAddressInfo to Wallet API

* Add Bloom filter and event callbacks to SPV node

* Move wallet test trait to testkit

* Test for connecting SPV node with wallet

* Tune logging in SPV node

* Clean up BloomFilter

* Proper toString in inventory and inventorymessage

* Actually pass in callbacks to SPV node

* Fix rebase screwup and partially respond to code review

* Default to file based databases in tests

This is due to issues with deadlocks with
in-memory based databases. We need DBs
to stay alive between connections, but
not across tests.

In DB intensive chain validation tests we
use in-memory databases instead. This
seems like a reasonable tradeoff between
simplicity and speed.

* Make SpvNodeCallbacks contain sequence of functions within

* Make Bloom filter non-optional in SpvNode
2019-06-17 06:48:26 -05:00
Torkel Rogstad
0fd70dc8e1 Refactor logback configuration to reduce repetition (#499) 2019-06-07 07:29:40 -05:00