bitcoin-s/docs/wallet/wallet-rescan.md

120 lines
4 KiB
Markdown
Raw Normal View History

2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
---
title: Wallet Rescans
id: wallet-rescan
---
With [BIP157](https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki) you can cache block filters locally to use
later for rescans in the case you need to restore your wallets. Our [chain](../chain/chain.md) project gives us
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
an API with the ability to query for filters.
### Rescan from CLI
To execute a rescan from the cli because you are restoring a wallet or it has gotten out of sync is fairly simple.
If you have an empty wallet it can be done by simply calling rescan
```bash
./bitcoin-s-cli rescan
```
If your wallet is not empty then you will need to call it with the force command
```bash
./bitcoin-s-cli rescan --force
```
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
You can also specify start and stop heights
```bash
./bitcoin-s-cli rescan --start <start height> --stop <stop height>
```
By default, if you do not set the start height, the rescan will begin at your wallet's creation time.
If you wish to ignore this and start from genesis use the `ignorecreationtime` flag
```bash
./bitcoin-s-cli rescan --ignorecreationtime
```
### Code Example
You can rescan your wallet with filters with [`WalletApi.rescanNeutrinoWallet()`](https://github.com/bitcoin-s/bitcoin-s/blob/master/core/src/main/scala/org/bitcoins/core/api/wallet/NeutrinoWalletApi.scala#L77)
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
To run this example you need to make sure you have access to a bitcoind binary.
You can download this with bitcoin-s by doing `sbt downloadBitcoind`
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
```scala mdoc:invisible
import org.bitcoins.testkit.BitcoinSTestAppConfig
import org.bitcoins.testkit.fixtures._
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
import org.bitcoins.testkit.wallet._
import org.bitcoins.server.BitcoinSAppConfig
import org.bitcoins.wallet.config.WalletAppConfig
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
import akka.actor.ActorSystem
import scala.concurrent._
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
import scala.concurrent.duration.DurationInt
```
```scala mdoc:compile-only
//we need an actor system and app config to power this
implicit val system: ActorSystem = ActorSystem(s"wallet-rescan-example")
implicit val ec: ExecutionContext = system.dispatcher
implicit val appConfig: BitcoinSAppConfig = BitcoinSTestAppConfig.getNeutrinoTestConfig()
implicit val walletAppConfig: WalletAppConfig = appConfig.walletConf
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
val bip39PasswordOpt = None
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
//ok now let's spin up a bitcoind and a bitcoin-s wallet with funds in it
val walletWithBitcoindF = for {
bitcoind <- BitcoinSFixture.createBitcoindWithFunds()
walletWithBitcoind <- BitcoinSWalletTest.createWalletWithBitcoindCallbacks(bitcoind, bip39PasswordOpt)
} yield walletWithBitcoind
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
val walletF = walletWithBitcoindF.map(_.wallet)
val bitcoindF = walletWithBitcoindF.map(_.bitcoind)
//let's see what our initial wallet balance is
val initBalanceF = for {
w <- walletF
balance <- w.getBalance()
} yield {
println(s"Initial wallet balance=${balance}")
balance
}
//ok great! We have money in the wallet to start,
//now let's delete our internal tables that hold our utxos
//and addresses so that we end up with a 0 balance
val clearedWalletF = for {
w <- walletF
_ <- initBalanceF
clearedWallet <- w.clearAllUtxos()
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
zeroBalance <- clearedWallet.getBalance()
} yield {
println(s"Balance after clearing utxos: ${zeroBalance}")
clearedWallet
}
//we need to pick how many addresses we want to generate off of our keychain
//when doing a rescan, this means we are generating 100 addrsses
//and then looking for matches. If we find a match, we generate _another_
//100 fresh addresses and search those. We keep doing this until we find
//100 addresses that do not contain a match.
val addrBatchSize = 100
//ok now that we have a cleared wallet, we need to rescan and find our fudns again!
val rescannedBalanceF = for {
w <- clearedWalletF
2020-04-24 09:37:12 -05:00
_ <- w.fullRescanNeutrinoWallet(addrBatchSize)
2020 03 06 wallet rescan test (#1218) * Add wallet rescan tests * Create FilterSync, which gives us an API inside of the chain project to sync filters with Add another unit test to filter sync Add more unit tests for ChainSync and FilterSync Clean up some docs, remove some extra lines of code Run scalafmt Add filter-sync.md Cleanup some nits Add more information of how FilterSync.syncFilters() works Add 'FilterWithHeaderHash' type so that we can actually validate/verify block headers that are being fed into the chain project Run scalafmt, hide imports in filter-sync.md so code appears cleaner Move implicits out of invisible block as it seems to cause errors Make it so FilterSync processes filters in batches rather than fetching them all at once Fix compile error * WIP bitcoind implement ChainQueryApi * rework fixtures to be able to support injecting ChainQueryApi implemented by bitcoind into our fixture infrastructure for creating wallets * Fix rebase problem * Implement getFiltersBetweenHeight() with ben's solution * WIP Start implementing NodeApi against bitcoind * Actually inject the bitcoind backed nodeApi into our fixture * Get first rescan test working for rescanning the entire blockchain * Implement test case for rescanning from a specific height * Fix NeutrinoNodeWalletTest test case that uses a experimental version of bitcoind, for now i believe our experimental binary is on v18 * Add wallet-rescan.md, add helper method to WalletApi that allows you to clear out the wallet's utxos/addresses * Add another log to try to debug CI * Address code review * Fix wrong ordering of deletion of tables in clearUtxosAndAddresses() * reset logging level
2020-03-13 16:11:02 -05:00
balanceAfterRescan <- w.getBalance()
} yield {
println(s"Wallet balance after rescan: ${balanceAfterRescan}")
()
}
//cleanup
val cleanupF = for {
_ <- rescannedBalanceF
walletWithBitcoind <- walletWithBitcoindF
_ <- BitcoinSWalletTest.destroyWalletWithBitcoind(walletWithBitcoind)
} yield ()
Await.result(cleanupF, 60.seconds)
```