This change is part of the effort to add pruning support to btcd.
cfIndex is a useful index even if the node has been pruned so it's
allowed to be enabled together with pruning. However, if the user had
disabled cfindex and enabled pruning, it's not possible to generate
them. In this case, we tell the user that it's impossible unless the
user deletes and start anew.
Additionally, if the user had enabled cfindex and also enabled pruning
from the start, don't let the user turn the cfindex off without dropping
it explicitly. This is to make sure that the user isn't left at an
inconsistent state where the cfindex isn't able to catch up to the tip
because the blocks have already been pruned.
This change is part of the effort to add pruning support to btcd.
It's not possible to generate the addr or tx indexes from scratch if the
block storage had been pruned previously as it's missing the block data.
When the user asks to create these indexes, tell them it's not possible
and the only way it's possible is if they delete and start anew.
This change is part of the effort to add pruning support to btcd.
The addr and tx indexes are not useful when the node is pruned as the
actual block data that the indexes point to are gone. If the user had
previously enabled them, then explicitly require an action from the user
to remove the indexes before letting the user enable pruning.
This change is part of the effort to add pruning support to btcd.
Allowing the user to not pass in the --prune flag after pruning results
in inaccurate reporting of the prune status for getblockchaininfo and
for signaling NODE_NETWORK_LIMITED to peers. Anything that relies on
cfg.Prune to be accurate is at risk of being incorrect.
To solve the current problems and to prevent potential future problems,
just force the user to keep the prune flag on like bitcoind. In terms
of UX, there isn't that much of a loss since if the user wants to keep
more blocks than they previously did, they can just increase the size
passed to --prune.
Enables Go memory profiling. If the cpuprofile shows a lot of time
spent on gc, it's useful to then do a memory profile to see where the
memory alloctions happen.
Unlike the --profile flag, this allows for easy generation of a memory
profile for the entire duration of which btcd has been running for in
various readble graphs.
Restrict the available set of system calls to the daemon to the basic
network and filesystem operations on OpenBSD. Further reduce potential
harm by limiting file system access to the btcd data dir and the rpc
files.
This propagates the interrupt channel through to blockchain and the
indexers so that it is possible to interrupt long-running operations
such as catching up indexes.
The btclog package has been changed to defining its own logging
interface (rather than seelog's) and provides a default implementation
for callers to use.
There are two primary advantages to the new logger implementation.
First, all log messages are created before the call returns. Compared
to seelog, this prevents data races when mutable variables are logged.
Second, the new logger does not implement any kind of artifical rate
limiting (what seelog refers to as "adaptive logging"). Log messages
are outputted as soon as possible and the application will appear to
perform much better when watching standard output.
Because log rotation is not a feature of the btclog logging
implementation, it is handled by the main package by importing a file
rotation package that provides an io.Reader interface for creating
output to a rotating file output. The rotator has been configured
with the same defaults that btcd previously used in the seelog config
(10MB file limits with maximum of 3 rolls) but now compresses newly
created roll files. Due to the high compressibility of log text, the
compressed files typically reduce to around 15-30% of the original
10MB file.
This rewrites the shutdown logic to simplify the shutdown signalling.
All cleanup is now run from deferred functions in the main function and
channels are used to signal shutdown either from OS signals or from
other subsystems such as the RPC server and windows service controller.
The RPC server has been modified to use a new channel for signalling
shutdown that is exposed via the RequestedProcessShutdown function
instead of directly calling Stop on the server as it previously did.
Finally, it adds a few checks for early termination during the main
start sequence so the process can be stopped without starting all the
subsystems if desired.
This is a backport of the equivalent logic from Decred with a few slight
modifications. Credits go to @jrick.
This reduces the target ratio of freshly allocated data to live data to
10% in order to limit excessive overallocations by the garbage collector
during data bursts such as processing complex blocks or rapidly
receiving a lot of large transactions.
This introduces a new indexing infrastructure for supporting optional
indexes using the new database and blockchain infrastructure along with
two concrete indexer implementations which provide both a
transaction-by-hash and a transaction-by-address index.
The new infrastructure is mostly separated into a package named indexers
which is housed under the blockchain package. In order to support this,
a new interface named IndexManager has been introduced in the blockchain
package which provides methods to be notified when the chain has been
initialized and when blocks are connected and disconnected from the main
chain. A concrete implementation of an index manager is provided by the
new indexers package.
The new indexers package also provides a new interface named Indexer
which allows the index manager to manage concrete index implementations
which conform to the interface.
The following is high level overview of the main index infrastructure
changes:
- Define a new IndexManager interface in the blockchain package and
modify the package to make use of the interface when specified
- Create a new indexers package
- Provides an Index interface which allows concrete indexes to plugin
to an index manager
- Provides a concrete IndexManager implementation
- Handles the lifecycle of all indexes it manages
- Tracks the index tips
- Handles catching up disabled indexes that have been reenabled
- Handles reorgs while the index was disabled
- Invokes the appropriate methods for all managed indexes to allow
them to index and deindex the blocks and transactions
- Implement a transaction-by-hash index
- Makes use of internal block IDs to save a significant amount of
space and indexing costs over the old transaction index format
- Implement a transaction-by-address index
- Makes use of a leveling scheme in order to provide a good tradeoff
between space required and indexing costs
- Supports enabling and disabling indexes at will
- Support the ability to drop indexes if they are no longer desired
The following is an overview of the btcd changes:
- Add a new index logging subsystem
- Add new options --txindex and --addrindex in order to enable the
optional indexes
- NOTE: The transaction index will automatically be enabled when the
address index is enabled because it depends on it
- Add new options --droptxindex and --dropaddrindex to allow the indexes
to be removed
- NOTE: The address index will also be removed when the transaction
index is dropped because it depends on it
- Update getrawtransactions RPC to make use of the transaction index
- Reimplement the searchrawtransaction RPC that makes use of the address
index
- Update sample-btcd.conf to include sample usage for the new optional
index flags
This commit is the first stage of several that are planned to convert
the blockchain package into a concurrent safe package that will
ultimately allow support for multi-peer download and concurrent chain
processing. The goal is to update btcd proper after each step so it can
take advantage of the enhancements as they are developed.
In addition to the aforementioned benefit, this staged approach has been
chosen since it is absolutely critical to maintain consensus.
Separating the changes into several stages makes it easier for reviewers
to logically follow what is happening and therefore helps prevent
consensus bugs. Naturally there are significant automated tests to help
prevent consensus issues as well.
The main focus of this stage is to convert the blockchain package to use
the new database interface and implement the chain-related functionality
which it no longer handles. It also aims to improve efficiency in
various areas by making use of the new database and chain capabilities.
The following is an overview of the chain changes:
- Update to use the new database interface
- Add chain-related functionality that the old database used to handle
- Main chain structure and state
- Transaction spend tracking
- Implement a new pruned unspent transaction output (utxo) set
- Provides efficient direct access to the unspent transaction outputs
- Uses a domain specific compression algorithm that understands the
standard transaction scripts in order to significantly compress them
- Removes reliance on the transaction index and paves the way toward
eventually enabling block pruning
- Modify the New function to accept a Config struct instead of
inidividual parameters
- Replace the old TxStore type with a new UtxoViewpoint type that makes
use of the new pruned utxo set
- Convert code to treat the new UtxoViewpoint as a rolling view that is
used between connects and disconnects to improve efficiency
- Make best chain state always set when the chain instance is created
- Remove now unnecessary logic for dealing with unset best state
- Make all exported functions concurrent safe
- Currently using a single chain state lock as it provides a straight
forward and easy to review path forward however this can be improved
with more fine grained locking
- Optimize various cases where full blocks were being loaded when only
the header is needed to help reduce the I/O load
- Add the ability for callers to get a snapshot of the current best
chain stats in a concurrent safe fashion
- Does not block callers while new blocks are being processed
- Make error messages that reference transaction outputs consistently
use <transaction hash>:<output index>
- Introduce a new AssertError type an convert internal consistency
checks to use it
- Update tests and examples to reflect the changes
- Add a full suite of tests to ensure correct functionality of the new
code
The following is an overview of the btcd changes:
- Update to use the new database and chain interfaces
- Temporarily remove all code related to the transaction index
- Temporarily remove all code related to the address index
- Convert all code that uses transaction stores to use the new utxo
view
- Rework several calls that required the block manager for safe
concurrency to use the chain package directly now that it is
concurrent safe
- Change all calls to obtain the best hash to use the new best state
snapshot capability from the chain package
- Remove workaround for limits on fetching height ranges since the new
database interface no longer imposes them
- Correct the gettxout RPC handler to return the best chain hash as
opposed the hash the txout was found in
- Optimize various RPC handlers:
- Change several of the RPC handlers to use the new chain snapshot
capability to avoid needlessly loading data
- Update several handlers to use new functionality to avoid accessing
the block manager so they are able to return the data without
blocking when the server is busy processing blocks
- Update non-verbose getblock to avoid deserialization and
serialization overhead
- Update getblockheader to request the block height directly from
chain and only load the header
- Update getdifficulty to use the new cached data from chain
- Update getmininginfo to use the new cached data from chain
- Update non-verbose getrawtransaction to avoid deserialization and
serialization overhead
- Update gettxout to use the new utxo store versus loading
full transactions using the transaction index
The following is an overview of the utility changes:
- Update addblock to use the new database and chain interfaces
- Update findcheckpoint to use the new database and chain interfaces
- Remove the dropafter utility which is no longer supported
NOTE: The transaction index and address index will be reimplemented in
another commit.
* Address index is built up concurrently with the `--addrindex` flag.
* Entire index can be deleted with `--dropaddrindex`.
* New RPC call: `searchrawtransaction`
* Returns all transacitons related to a particular address
* Includes mempool transactions
* Requires `--addrindex` to be activated and fully caught up.
* New `blockLogger` struct has been added to factor our common logging
code
* Wiki and docs updated with new features.
This commits removes a number of golint warnings. There is a class of
warnings which I can't fix due to unsufficient knowledge of the domain
at this point. These are listed here:
addrmanager.go:907:1: comment on exported method AddrManager.Attempt
should be of the form "Attempt ..."
addrmanager.go:1048:1: exported function RFC1918 should have comment or
be unexported
addrmanager.go:1058:1: exported function RFC3849 should have comment or
be unexported
addrmanager.go:1065:1: exported function RFC3927 should have comment or
be unexported
addrmanager.go:1073:1: exported function RFC3964 should have comment or
be unexported
addrmanager.go:1081:1: exported function RFC4193 should have comment or
be unexported
addrmanager.go:1089:1: exported function RFC4380 should have comment or
be unexported
addrmanager.go:1097:1: exported function RFC4843 should have comment or
be unexported
addrmanager.go:1105:1: exported function RFC4862 should have comment or
be unexported
addrmanager.go:1113:1: exported function RFC6052 should have comment or
be unexported
addrmanager.go:1121:1: exported function RFC6145 should have comment or
be unexported
addrmanager.go:1128:1: exported function Tor should have comment or be
unexported
addrmanager.go:1143:1: exported function Local should have comment or be
unexported
addrmanager.go:1228:2: exported const InterfacePrio should have comment
(or a comment on this block) or be unexported
discovery.go:26:2: exported var ErrTorInvalidAddressResponse should have
comment or be unexported
limits/limits_unix.go:19:1: exported function SetLimits should have
comment or be unexported
limits/limits_windows.go:7:1: exported function SetLimits should have
comment or be unexported
util/dropafter/dropafter.go:22:6: exported type ShaHash should have
comment or be unexported
util/dropafter/dropafter.go:38:2: exported const ArgSha should have
comment (or a comment on this block) or be unexported
util/dropafter/dropafter.go:128:5: exported var ErrBadShaPrefix should
have comment or be unexported
util/dropafter/dropafter.go:129:5: exported var ErrBadShaLen should have
comment or be unexported
util/dropafter/dropafter.go:130:5: exported var ErrBadShaChar should
have comment or be unexported
util/showblock/showblock.go:24:6: exported type ShaHash should have
comment or be unexported
util/showblock/showblock.go:46:2: exported const ArgSha should have
comment (or a comment on this block) or be unexported
util/showblock/showblock.go:163:1: exported function DumpBlock should
have comment or be unexported
util/showblock/showblock.go:211:5: exported var ErrBadShaPrefix should
have comment or be unexported
util/showblock/showblock.go:212:5: exported var ErrBadShaLen should have
comment or be unexported
util/showblock/showblock.go:213:5: exported var ErrBadShaChar should
have comment or be unexported
This change modifies the params struct to embed a *btcnet.Params,
removing the old parameter fields that are handled by the btcnet
package.
Hardcoded network checks have also been removed in favor of modifying
behavior based on the current active net's parameters.
Not all library packages, notable btcutil and btcchain, have been
updated to use btcnet yet, but with this change, each package can be
updated one at a time since the active net's btcnet.Params are
available at each callsite.
ok @davecgh
This commit adds a new option, --logdir, which works in the same fashion
as the --datadir option. Consequently, the logging directory is name
"namespaced" by the network as well. This resolves the issue where two
btcd instances running (one for mainnet and one for testnet) would
overwrite each other's log files by default.
It also provides the user with a method to change the logging location to
non-default locations if they prefer. For example, it enables multiple
btcd instances on the same network to specify unique logging directories
(even though running multiple btcd instances on the same network is not
the most sane configuration).
Closes#95.
This changes the protocol between btcd and btcwallet to follow
JSON-RPC specifications sending notifications as requests with an
empty ID.
The notification request context handling has been greatly cleaned up
now that IDs no longer need to be saved when sending notifications.
since we don't wait for peers, this largely just waits for the server procs
themselves to die. Unless the entire server is wedged (which is what kill -9 is
for) this should always shut down fairly swiftly.
This should mean we sync addrmanager and disestablish upnp correctly on
interrupt.
Discussed with davec.
This commit modifies btcd to run cleanly as a Windows service. btcd is
intended to be a long running process that stays synchronized with the
bitcoin block chain and provides chain services to multiple users. It
follows that a service is the best option on Windows for this
functionality.
A few key points are:
- Supports graceful shutdown via the service stop/shutdown commands
- Integrates cleanly with the Windows event log
- Adds a new /s flag that can be used to install/remove/start/stop the
service
One outstanding issue is that the application data directory is currently
user specific which means, by default, if you start btcd as a user, the
same data won't be used as when it's running as a service. This needs to
be resovled. The most likely approach will be to put all data into the
common appdata directory Windows provides, but it will require some
additional work to deal with permissions properly as user processes can't
write there by default.
Closes#42.
Also, make every subsystem within btcd use its own logger instance so each
subsystem can have its own level specified independent of the others.
This is work towards #48.
This allows the provision of address/port pairs to be listened on instead
of just providing the port. e.g.:
btcd --listen 1.2.3.4:4321 --listen 127.0.0.01 --listen [::1]:5432
When --proxy and --connect are used, we disable listening *unless* any --listen
arguments have been provided, when we will listen on those addresses as
requested.
Initial code by davec, integration by myself.
Closes#33
allow listens to fail, but warn. error if all failed
fmt
This commit provides a new --cpuprofile flag that can be used to specify a
file path to write CPU profile data into. The resulting profile can then be
consumed by the 'go tool pprof' command.
Since the main SIGINT handler is running as a goroutine, the main
goroutine must be kept active long enough for it to finish or it will be
nuked when the main goroutine exits. This commit makes that happen by
slightly modifying how it waits for shutdown.
Rather than having the main goroutine only wait for the server to
shutdown, there is now a shutdown channel that is used to signal the main
goroutine to shutdown for all cases such as a graceful shutdown, a
scheduled shutdown, an RPC stop, or a SIGINT.
While here, also add a few prints to indicate a SIGINT was received and
the shutdown progress.
This commit is a first pass at improving the logging. It changes a number
of things to improve the readability of the output. The biggest addition
is message summaries for each message type when using the debug logging
level.
There is sitll more to do here such as allowing the level of each
subsystem to be independently specified, syslog support, and allowing the
logging level to be changed run-time.
