To avoid running a channel DB that was successfully migrated to another
system by accident, we check if there is a tombstone marker before we
open the DB for use.
We'll need to know whether a database was migrated to the latest version
in our upcoming data migration tool. To be able to determine the current
version of a DB and the total number of migrations in existence, we need
to export some of the functions in channeldb.
We also add some helper functions for adding tombstone and other markers
to a database.
This commit enables the db to run optional migrations that are specified
by config flags. To achieve this, an optional meta is introduced to
manage the optional migrations. We distinguish the two types of
migrations here so it's easier to manage them for the concern a future
migration can cause trouble for us to determine the db version if we
don't.
This introduces a BigSize migration that is used to expand the width
of the ChannelStatus and ChannelType fields. Three channel "types"
are added - ZeroConfBit, ScidAliasChanBit, and ScidAliasFeatureBit.
ScidAliasChanBit denotes that the scid-alias channel type was
negotiated for the channel. ScidAliasFeatureBit denotes that the
scid-alias feature bit was negotiated during the *lifetime* of the
channel. Several helper functions on the OpenChannel struct are
exposed to aid callers from different packages.
The RefreshShortChanID has been renamed to Refresh.
A new function BroadcastHeight is used to guard access to the
mutable FundingBroadcastHeight member. This prevents data races.
This commit adds a new migration to patch the two balance fields,
`InitialLocalBalance` and `InitialRemoteBalance` for the historical
channels. Because they are not saved previously, for historical channels
prior to the revocation log PR, these fields will be empty.
This can cause an intermittent panic otherwise if bbolt remaps itself
via munmap and mmap. From bbolt's documentation:
* Byte slices returned from Bolt are only valid during a transaction.
Once the transaction has been committed or rolled back then the memory
they point to can be reused by a new page or can be unmapped from
virtual memory and you'll see an unexpected fault address panic when
accessing it.
This commit was previously split into the following parts to ease
review:
- 2d746f68: replace imports
- 4008f0fd: use ecdsa.Signature
- 849e33d1: remove btcec.S256()
- b8f6ebbd: use v2 library correctly
- fa80bca9: bump go modules
For older nodes, this bucket was never created, so we'll get an error if
we try and query it. In this commit, we catch this error like we do when
a given channel doesn't have the information (but the bucket actually
exists).
Fixes#6155
In case the channeldb package is used as a library in external tools, it
can be useful to allow read-only access to a DB. This allows such a
tool to access a DB even if not all migrations were executed, which can
be useful for recovery purposes.
To make it possible to even start the DB with a read-only backend, we
need to disable the automatic migration step.
With this commit we forward the config option for disabling the channel
graph cache as a boolean to the channeldb. But we invert its meaning to
make the flag easier to understand.
To avoid the channel map needing to be re-grown while we fill the cache
initially, we might as well pre-allocate it with a somewhat sane value
to decrease the number of grow events.
To further separate the channel graph from the channel state, we
refactor the AddrsForNode method to use the graphs's public methods
instead of directly accessing any buckets. This makes sure that we can
have the channel state cached with just its buckets while not using a
kvdb level cache for the graph.
At the same time we refactor the graph's test to also be less dependent
upon the channel state DB.
The funding manager doesn't need to know the details of the underlying
storage of the opening channel state, so we move the actual store and
retrieval into the channel database.
Adds an optional tx parameter to ForAllOutgoingChannels and FetchChannel
so that data can be queried within the context of an existing database
transaction.
Depends on btcsuite/btcwallet#757.
Pulls in the updated version of btcwallet and walletdb that have the DB
interface enhanced by their own View() and Update() methods with the
reset callback/closure supported out of the box. That way the global
package-level View() and Update() functions now become pure redirects.
In this commit, we migrate all wire messages in the database from the
`legacy` to the `current` encoding.
This affects the way we write the `CommitDiff` and `LogUpdates` struct
to disk. We also need to migrate the network results bucket in the
switch as it includes a wire message without a length prefix.
Adds an outpoint index that stores a tlv stream. Currently the stream
only contains the outpoint's indexStatus. This should cut down on
big bbolt transactions in several places throughout the codebase.
Previously we wouldn't recreate some of the top level buckets that are
now considered expected with our migration logic. This bug was
preexisting, but never surfaced because the other TLB buckets were not
touched by this unit test.
* mod: bump btcwallet version to accept db timeout
* btcwallet: add DBTimeOut in config
* kvdb: add database timeout option for bbolt
This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.
* channeldb: add DBTimeout in db options
This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.
* contractcourt+routing: use DBTimeout in kvdb
This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.
* lncfg: add DBTimeout option in db config
The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.
* migration: add DBTimeout param in kvdb.Create/kvdb.Open
* keychain: update tests to use DBTimeout param
* htlcswitch+chainreg: add DBTimeout option
* macaroons: support DBTimeout config in creation
This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.
* walletunlocker: add dbTimeout option in UnlockerService
This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.
* watchtower/wtdb: add dbTimeout param during creation
This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.
* multi: add db timeout param for walletdb.Create
This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.
* lnd: pass DBTimeout config to relevant services
This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
- chainControlConfig
- walletunlocker
- wallet.NewLoader
- macaroons
- watchtower
In addition, the usage of wallet.Create is updated too.
* sample-config: add dbtimeout option
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
Since we will use peer flap rate to determine how we rate limit, we
store this value on disk per peer per channel. This allows us to
restart with memory of our peers past behaviour, so we don't give badly
behaving peers have a fresh start on restart. Last flap timestamp is
stored with our flap count so that we can degrade this all time flap
count over time for peers that have not recently flapped.
This commit moves makeTestDB to db.go and exports it so that we'll be
able to use this function in other unit tests to make them testable with
etcd if needed.
Add a new top level bucket which holds closed channels nested by chain
hash which contains additional information about channel closes. We add
resolver resolutions under their own key so that we can extend the
bucket with additional information if required.
This was initially done as there were a few assertions throughout the
codebase requiring a channel's policy to be known. Now that these have
been addressed, we no longer need to store restored channels in the
graph, as their policies where incomplete anyway.
Add an entry to a payments index bucket which maps sequence number
to payment hash when we initiate payments. This allows for more
efficient paginated queries. We create the top level bucket in its
own migration so that we do not need to create it on the fly.
When we retry payments and provide them with a new sequence number, we
delete the index for their existing payment so that we do not have an
index that points to a non-existent payment.
If we delete a payment, we also delete its index entry. This prevents
us from looking up entries from indexes to payments that do not exist.
