btcd/database/internal/treap/immutable_test.go

498 lines
15 KiB
Go
Raw Normal View History

database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
// Copyright (c) 2015-2016 The btcsuite developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.
package treap
import (
"bytes"
"crypto/sha256"
"testing"
)
// TestImmutableEmpty ensures calling functions on an empty immutable treap
// works as expected.
func TestImmutableEmpty(t *testing.T) {
t.Parallel()
// Ensure the treap length is the expected value.
testTreap := NewImmutable()
if gotLen := testTreap.Len(); gotLen != 0 {
t.Fatalf("Len: unexpected length - got %d, want %d", gotLen, 0)
}
// Ensure the reported size is 0.
if gotSize := testTreap.Size(); gotSize != 0 {
t.Fatalf("Size: unexpected byte size - got %d, want 0",
gotSize)
}
// Ensure there are no errors with requesting keys from an empty treap.
key := serializeUint32(0)
if gotVal := testTreap.Has(key); gotVal {
database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
t.Fatalf("Has: unexpected result - got %v, want false", gotVal)
}
if gotVal := testTreap.Get(key); gotVal != nil {
t.Fatalf("Get: unexpected result - got %x, want nil", gotVal)
}
// Ensure there are no panics when deleting keys from an empty treap.
testTreap.Delete(key)
// Ensure the number of keys iterated by ForEach on an empty treap is
// zero.
var numIterated int
testTreap.ForEach(func(k, v []byte) bool {
numIterated++
return true
})
if numIterated != 0 {
t.Fatalf("ForEach: unexpected iterate count - got %d, want 0",
numIterated)
}
}
// TestImmutableSequential ensures that putting keys into an immutable treap in
// sequential order works as expected.
func TestImmutableSequential(t *testing.T) {
t.Parallel()
// Insert a bunch of sequential keys while checking several of the treap
// functions work as expected.
expectedSize := uint64(0)
numItems := 1000
testTreap := NewImmutable()
for i := 0; i < numItems; i++ {
key := serializeUint32(uint32(i))
testTreap = testTreap.Put(key, key)
// Ensure the treap length is the expected value.
if gotLen := testTreap.Len(); gotLen != i+1 {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, i+1)
}
// Ensure the treap has the key.
if !testTreap.Has(key) {
t.Fatalf("Has #%d: key %q is not in treap", i, key)
}
// Get the key from the treap and ensure it is the expected
// value.
if gotVal := testTreap.Get(key); !bytes.Equal(gotVal, key) {
t.Fatalf("Get #%d: unexpected value - got %x, want %x",
i, gotVal, key)
}
// Ensure the expected size is reported.
expectedSize += (nodeFieldsSize + 8)
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
}
// Ensure the all keys are iterated by ForEach in order.
var numIterated int
testTreap.ForEach(func(k, v []byte) bool {
wantKey := serializeUint32(uint32(numIterated))
// Ensure the key is as expected.
if !bytes.Equal(k, wantKey) {
t.Fatalf("ForEach #%d: unexpected key - got %x, want %x",
numIterated, k, wantKey)
}
// Ensure the value is as expected.
if !bytes.Equal(v, wantKey) {
t.Fatalf("ForEach #%d: unexpected value - got %x, want %x",
numIterated, v, wantKey)
}
numIterated++
return true
})
// Ensure all items were iterated.
if numIterated != numItems {
t.Fatalf("ForEach: unexpected iterate count - got %d, want %d",
numIterated, numItems)
}
// Delete the keys one-by-one while checking several of the treap
// functions work as expected.
for i := 0; i < numItems; i++ {
key := serializeUint32(uint32(i))
testTreap = testTreap.Delete(key)
// Ensure the treap length is the expected value.
if gotLen := testTreap.Len(); gotLen != numItems-i-1 {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, numItems-i-1)
}
// Ensure the treap no longer has the key.
if testTreap.Has(key) {
t.Fatalf("Has #%d: key %q is in treap", i, key)
}
// Get the key that no longer exists from the treap and ensure
// it is nil.
if gotVal := testTreap.Get(key); gotVal != nil {
t.Fatalf("Get #%d: unexpected value - got %x, want nil",
i, gotVal)
}
// Ensure the expected size is reported.
expectedSize -= (nodeFieldsSize + 8)
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
}
}
// TestImmutableReverseSequential ensures that putting keys into an immutable
// treap in reverse sequential order works as expected.
func TestImmutableReverseSequential(t *testing.T) {
t.Parallel()
// Insert a bunch of sequential keys while checking several of the treap
// functions work as expected.
expectedSize := uint64(0)
numItems := 1000
testTreap := NewImmutable()
for i := 0; i < numItems; i++ {
key := serializeUint32(uint32(numItems - i - 1))
testTreap = testTreap.Put(key, key)
// Ensure the treap length is the expected value.
if gotLen := testTreap.Len(); gotLen != i+1 {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, i+1)
}
// Ensure the treap has the key.
if !testTreap.Has(key) {
t.Fatalf("Has #%d: key %q is not in treap", i, key)
}
// Get the key from the treap and ensure it is the expected
// value.
if gotVal := testTreap.Get(key); !bytes.Equal(gotVal, key) {
t.Fatalf("Get #%d: unexpected value - got %x, want %x",
i, gotVal, key)
}
// Ensure the expected size is reported.
expectedSize += (nodeFieldsSize + 8)
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
}
// Ensure the all keys are iterated by ForEach in order.
var numIterated int
testTreap.ForEach(func(k, v []byte) bool {
wantKey := serializeUint32(uint32(numIterated))
// Ensure the key is as expected.
if !bytes.Equal(k, wantKey) {
t.Fatalf("ForEach #%d: unexpected key - got %x, want %x",
numIterated, k, wantKey)
}
// Ensure the value is as expected.
if !bytes.Equal(v, wantKey) {
t.Fatalf("ForEach #%d: unexpected value - got %x, want %x",
numIterated, v, wantKey)
}
numIterated++
return true
})
// Ensure all items were iterated.
if numIterated != numItems {
t.Fatalf("ForEach: unexpected iterate count - got %d, want %d",
numIterated, numItems)
}
// Delete the keys one-by-one while checking several of the treap
// functions work as expected.
for i := 0; i < numItems; i++ {
// Intentionally use the reverse order they were inserted here.
key := serializeUint32(uint32(i))
testTreap = testTreap.Delete(key)
// Ensure the treap length is the expected value.
if gotLen := testTreap.Len(); gotLen != numItems-i-1 {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, numItems-i-1)
}
// Ensure the treap no longer has the key.
if testTreap.Has(key) {
t.Fatalf("Has #%d: key %q is in treap", i, key)
}
// Get the key that no longer exists from the treap and ensure
// it is nil.
if gotVal := testTreap.Get(key); gotVal != nil {
t.Fatalf("Get #%d: unexpected value - got %x, want nil",
i, gotVal)
}
// Ensure the expected size is reported.
expectedSize -= (nodeFieldsSize + 8)
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
}
}
// TestImmutableUnordered ensures that putting keys into an immutable treap in
// no paritcular order works as expected.
func TestImmutableUnordered(t *testing.T) {
t.Parallel()
// Insert a bunch of out-of-order keys while checking several of the
// treap functions work as expected.
expectedSize := uint64(0)
numItems := 1000
testTreap := NewImmutable()
for i := 0; i < numItems; i++ {
// Hash the serialized int to generate out-of-order keys.
hash := sha256.Sum256(serializeUint32(uint32(i)))
key := hash[:]
testTreap = testTreap.Put(key, key)
// Ensure the treap length is the expected value.
if gotLen := testTreap.Len(); gotLen != i+1 {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, i+1)
}
// Ensure the treap has the key.
if !testTreap.Has(key) {
t.Fatalf("Has #%d: key %q is not in treap", i, key)
}
// Get the key from the treap and ensure it is the expected
// value.
if gotVal := testTreap.Get(key); !bytes.Equal(gotVal, key) {
t.Fatalf("Get #%d: unexpected value - got %x, want %x",
i, gotVal, key)
}
// Ensure the expected size is reported.
expectedSize += nodeFieldsSize + uint64(len(key)+len(key))
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
}
// Delete the keys one-by-one while checking several of the treap
// functions work as expected.
for i := 0; i < numItems; i++ {
// Hash the serialized int to generate out-of-order keys.
hash := sha256.Sum256(serializeUint32(uint32(i)))
key := hash[:]
testTreap = testTreap.Delete(key)
// Ensure the treap length is the expected value.
if gotLen := testTreap.Len(); gotLen != numItems-i-1 {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, numItems-i-1)
}
// Ensure the treap no longer has the key.
if testTreap.Has(key) {
t.Fatalf("Has #%d: key %q is in treap", i, key)
}
// Get the key that no longer exists from the treap and ensure
// it is nil.
if gotVal := testTreap.Get(key); gotVal != nil {
t.Fatalf("Get #%d: unexpected value - got %x, want nil",
i, gotVal)
}
// Ensure the expected size is reported.
expectedSize -= (nodeFieldsSize + 64)
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
}
}
// TestImmutableDuplicatePut ensures that putting a duplicate key into an
// immutable treap works as expected.
func TestImmutableDuplicatePut(t *testing.T) {
t.Parallel()
expectedVal := []byte("testval")
expectedSize := uint64(0)
numItems := 1000
testTreap := NewImmutable()
for i := 0; i < numItems; i++ {
key := serializeUint32(uint32(i))
testTreap = testTreap.Put(key, key)
expectedSize += nodeFieldsSize + uint64(len(key)+len(key))
// Put a duplicate key with the expected final value.
database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
testTreap = testTreap.Put(key, expectedVal)
// Ensure the key still exists and is the new value.
if gotVal := testTreap.Has(key); !gotVal {
t.Fatalf("Has: unexpected result - got %v, want true",
database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
gotVal)
}
if gotVal := testTreap.Get(key); !bytes.Equal(gotVal, expectedVal) {
t.Fatalf("Get: unexpected result - got %x, want %x",
gotVal, expectedVal)
}
// Ensure the expected size is reported.
expectedSize -= uint64(len(key))
expectedSize += uint64(len(expectedVal))
if gotSize := testTreap.Size(); gotSize != expectedSize {
t.Fatalf("Size: unexpected byte size - got %d, want %d",
gotSize, expectedSize)
}
}
}
// TestImmutableNilValue ensures that putting a nil value into an immutable
// treap results in a key being added with an empty byte slice.
func TestImmutableNilValue(t *testing.T) {
t.Parallel()
key := serializeUint32(0)
// Put the key with a nil value.
testTreap := NewImmutable()
testTreap = testTreap.Put(key, nil)
// Ensure the key exists and is an empty byte slice.
if gotVal := testTreap.Has(key); !gotVal {
t.Fatalf("Has: unexpected result - got %v, want true", gotVal)
database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
}
if gotVal := testTreap.Get(key); gotVal == nil {
t.Fatalf("Get: unexpected result - got nil, want empty slice")
}
if gotVal := testTreap.Get(key); len(gotVal) != 0 {
t.Fatalf("Get: unexpected result - got %x, want empty slice",
gotVal)
}
}
// TestImmutableForEachStopIterator ensures that returning false from the ForEach
// callback on an immutable treap stops iteration early.
func TestImmutableForEachStopIterator(t *testing.T) {
t.Parallel()
// Insert a few keys.
numItems := 10
testTreap := NewImmutable()
for i := 0; i < numItems; i++ {
key := serializeUint32(uint32(i))
testTreap = testTreap.Put(key, key)
}
// Ensure ForEach exits early on false return by caller.
var numIterated int
testTreap.ForEach(func(k, v []byte) bool {
numIterated++
return numIterated != numItems/2
database: Major redesign of database package. This commit contains a complete redesign and rewrite of the database package that approaches things in a vastly different manner than the previous version. This is the first part of several stages that will be needed to ultimately make use of this new package. Some of the reason for this were discussed in #255, however a quick summary is as follows: - The previous database could only contain blocks on the main chain and reorgs required deleting the blocks from the database. This made it impossible to store orphans and could make external RPC calls for information about blocks during the middle of a reorg fail. - The previous database interface forced a high level of bitcoin-specific intelligence such as spend tracking into each backend driver. - The aforementioned point led to making it difficult to implement new backend drivers due to the need to repeat a lot of non-trivial logic which is better handled at a higher layer, such as the blockchain package. - The old database stored all blocks in leveldb. This made it extremely inefficient to do things such as lookup headers and individual transactions since the entire block had to be loaded from leveldb (which entails it doing data copies) to get access. In order to address all of these concerns, and others not mentioned, the database interface has been redesigned as follows: - Two main categories of functionality are provided: block storage and metadata storage - All block storage and metadata storage are done via read-only and read-write MVCC transactions with both manual and managed modes - Support for multiple concurrent readers and a single writer - Readers use a snapshot and therefore are not blocked by the writer - Some key properties of the block storage and retrieval API: - It is generic and does NOT contain additional bitcoin logic such spend tracking and block linking - Provides access to the raw serialized bytes so deserialization is not forced for callers that don't need it - Support for fetching headers via independent functions which allows implementations to provide significant optimizations - Ability to efficiently retrieve arbitrary regions of blocks (transactions, scripts, etc) - A rich metadata storage API is provided: - Key/value with arbitrary data - Support for buckets and nested buckets - Bucket iteration through a couple of different mechanisms - Cursors for efficient and direct key seeking - Supports registration of backend database implementations - Comprehensive test coverage - Provides strong documentation with example usage This commit also contains an implementation of the previously discussed interface named ffldb (flat file plus leveldb metadata backend). Here is a quick overview: - Highly optimized for read performance with consistent write performance regardless of database size - All blocks are stored in flat files on the file system - Bulk block region fetching is optimized to perform linear reads which improves performance on spindle disks - Anti-corruption mechanisms: - Flat files contain full block checksums to quickly an easily detect database corruption without needing to do expensive merkle root calculations - Metadata checksums - Open reconciliation - Extensive test coverage: - Comprehensive blackbox interface testing - Whitebox testing which uses intimate knowledge to exercise uncommon failure paths such as deleting files out from under the database - Corruption tests (replacing random data in the files) In addition, this commit also contains a new tool under the new database directory named dbtool which provides a few basic commands for testing the database. It is designed around commands, so it could be useful to expand on in the future. Finally, this commit addresses the following issues: - Adds support for and therefore closes #255 - Fixes #199 - Fixes #201 - Implements and closes #256 - Obsoletes and closes #257 - Closes #247 once the required chain and btcd modifications are in place to make use of this new code
2016-02-03 18:42:04 +01:00
})
if numIterated != numItems/2 {
t.Fatalf("ForEach: unexpected iterate count - got %d, want %d",
numIterated, numItems/2)
}
}
// TestImmutableSnapshot ensures that immutable treaps are actually immutable by
// keeping a reference to the previous treap, performing a mutation, and then
// ensuring the referenced treap does not have the mutation applied.
func TestImmutableSnapshot(t *testing.T) {
t.Parallel()
// Insert a bunch of sequential keys while checking several of the treap
// functions work as expected.
expectedSize := uint64(0)
numItems := 1000
testTreap := NewImmutable()
for i := 0; i < numItems; i++ {
treapSnap := testTreap
key := serializeUint32(uint32(i))
testTreap = testTreap.Put(key, key)
// Ensure the length of the treap snapshot is the expected
// value.
if gotLen := treapSnap.Len(); gotLen != i {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, i)
}
// Ensure the treap snapshot does not have the key.
if treapSnap.Has(key) {
t.Fatalf("Has #%d: key %q is in treap", i, key)
}
// Get the key that doesn't exist in the treap snapshot and
// ensure it is nil.
if gotVal := treapSnap.Get(key); gotVal != nil {
t.Fatalf("Get #%d: unexpected value - got %x, want nil",
i, gotVal)
}
// Ensure the expected size is reported.
if gotSize := treapSnap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
expectedSize += (nodeFieldsSize + 8)
}
// Delete the keys one-by-one while checking several of the treap
// functions work as expected.
for i := 0; i < numItems; i++ {
treapSnap := testTreap
key := serializeUint32(uint32(i))
testTreap = testTreap.Delete(key)
// Ensure the length of the treap snapshot is the expected
// value.
if gotLen := treapSnap.Len(); gotLen != numItems-i {
t.Fatalf("Len #%d: unexpected length - got %d, want %d",
i, gotLen, numItems-i)
}
// Ensure the treap snapshot still has the key.
if !treapSnap.Has(key) {
t.Fatalf("Has #%d: key %q is not in treap", i, key)
}
// Get the key from the treap snapshot and ensure it is still
// the expected value.
if gotVal := treapSnap.Get(key); !bytes.Equal(gotVal, key) {
t.Fatalf("Get #%d: unexpected value - got %x, want %x",
i, gotVal, key)
}
// Ensure the expected size is reported.
if gotSize := treapSnap.Size(); gotSize != expectedSize {
t.Fatalf("Size #%d: unexpected byte size - got %d, "+
"want %d", i, gotSize, expectedSize)
}
expectedSize -= (nodeFieldsSize + 8)
}
}