This commit adds an optional separate "native" SQL backend that will
host all tables supporting native SQL. For now since we don't have many
such tables we'll keep it in one file for SQLite, but it may be split up
if desired.
In this commit, a new `--db.no-rev-log-amt-data` flag is added. The
config option is passed though to everywhere that it will be used. Note
that it is still a no-op in this commit. An upcoming commit will make
use of the flag.
Use kvdb package v1.4.1. This update also forced the protobuf version to
be bumped which required `make rpc` to be run to update the generated
files. This also required a bump in the github pinned dependencies
config for the grpc and protobuf libs.
In this commit we add an extra config for enabling logging to an
external file when using embedded etcd. This can be useful when running
integration tests to see more details about etcd related issues.
Since we're now storing the content of multiple previously distinct
database files in etcd, we want to properly namespace them as not to
provoke any key collisions. We append a sub namespace to the given
global namespace in order to still support multiple lnd nodes using the
same etcd database simultaneously.
Because the btcwallet code uses the legacy walletdb interface we must
assume it is not fully concurrency safe. Therefore we make sure only a
single writer can be active at any given time for the wallet DB backend
when using etcd.
To have all the database backend related code in one place, we finally
also move the initialization of the wallet DB loader option into the
GetBackends() method.
The final database that needs to be made remote compatible is the
watchtower server and client database.
They are handled a bit differently because both of them are not always
active, only when specifically turned on in the config.
Even though the sphinx router's persistent replay log is not crucial in
the operation of lnd as its state can be re-created by creating a new
brontide connection, we want to make lnd fully stateless and therefore
have the option of not storing any state on disk.
The macaroon root keys should also be stored to the remote database if a
replicated backend such as etcd is used.
This commit refactors the macaroons service and wallet unlocker to
accept a kvdb backend directly instead of creating the bolt instance
automatically.
This commit gets rid of the concept of a local and remote database when
etcd is used. Instead the same backend is now used for both the
(previously renamed from local and remote DB) graph and channel state
databases.
This will make path finding extremely slow on etcd and will require
further optimizations and possibly a write-through cache for the graph
DB. But this is a requirement for making lnd itself fully stateless.
In order to separate our databases more clearly, we refactor the height
hint cache DB to use a kvdb backend instead of the channel DB instance
directly.
As a preparation to not have a local and remote version of the database
around anymore, we rename the variables into what their actual function
is. In case of the RPC server we even directly use the channel graph
instead of the DB instance. This should allow us to extract the channel
graph into its own, separate database (perhaps with better access
characteristics) in the future.
* mod: bump btcwallet version to accept db timeout
* btcwallet: add DBTimeOut in config
* kvdb: add database timeout option for bbolt
This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.
* channeldb: add DBTimeout in db options
This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.
* contractcourt+routing: use DBTimeout in kvdb
This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.
* lncfg: add DBTimeout option in db config
The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.
* migration: add DBTimeout param in kvdb.Create/kvdb.Open
* keychain: update tests to use DBTimeout param
* htlcswitch+chainreg: add DBTimeout option
* macaroons: support DBTimeout config in creation
This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.
* walletunlocker: add dbTimeout option in UnlockerService
This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.
* watchtower/wtdb: add dbTimeout param during creation
This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.
* multi: add db timeout param for walletdb.Create
This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.
* lnd: pass DBTimeout config to relevant services
This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
- chainControlConfig
- walletunlocker
- wallet.NewLoader
- macaroons
- watchtower
In addition, the usage of wallet.Create is updated too.
* sample-config: add dbtimeout option
In this commit, we modify the existing `GetBackend` method to now be
called `GetBackends`. This new method will populate a new `RemoteDB`
attribute based on if the replicated backend is active or not. As is,
the local backend is used everywhere. An upcoming commit will once again
re-enable the remote backend, in a hybrid manner.
In this commit, unify the old and new values for `sync-freelist`, and
also ensure that we don't break behavior for any users that're using the
_old_ value.
To do this, we first rename what was `--db.bolt.no-sync-freelist`, to
`--db.bolt.sync-freelist`. This gets rid of the negation on the config
level, and lets us override that value if the user is specifying the
legacy config option.
In the future, we'll deprecate the old config option, in favor of the
new DB scoped option.
This commit extends etcd db with namespaces without additional storage
space requirements. This is simply done by instead of using an all zero
root bucket id, we use the sha256 hash of the name space as our root
bucket id.
This commit separates all etcd related sources (sans a few stubs and
config) from the rest of the source tree and makes compilation conditional
depending on whether the kvdb_etcd build tag is specified.
This commit extends lncfg to support user specified database backend.
This supports configuration for both bolt and etcd (while only allowing
one or the other).
