In this commit, a migration is done that takes all the AckedUpdates of
all sessions and stores them in the RangeIndex pattern instead and
deletes the session's old AckedUpdates bucket. All the logic in the code
is also updates in order to write and read from this new structure.
In this commit, the code for migration 4 is added. This migration takes
all the existing session acked updates and migrates them to be stored in
the RangeIndex form instead. Note that this migration is not activated
in this commit. This is done in a follow up commit in order to keep this
one smaller.
In this commit, we add the ability to add a wtdb version migration that
does not get given a transaction but rather a whole db object. This will
be useful for migrations that are best done in multiple transaction in
order to use less RAM.
In preparation for an upcoming commit where some helper functions will
need access to the ClientDB's ackedRangeIndex member, this commit
converts those helper functions into methods on the ClientDB struct.
Refactor the putClientSessionBody to take in a session sub-bucket rather
than the top-level session bucket. This is mainly to make an upcoming
commit diff easier to parse.
In this commit, a new concept called a RangeIndex is introduced. It
provides an efficient way to keep track of numbers added to a set by
keeping track of various ranges instead of individual numbers.
Notably, it also provides a way to map the contents & diffs applied to
the in memory RangeIndex (which uses a sorted array structure) to a
persisted KV structure.
In this commit, a new channel-ID index is added to the tower client db
with the help of a migration. This index holds a mapping from a
db-assigned-ID (a uint64 encoded using BigSize encoding) to real channel
ID (32 bytes). This mapping will help us save space in future when
persisting references to channels.
In this commit a migration of the tower client db is done. The migration
creates a new top-level cChanDetailsBkt bucket and for each channel
found in the old cChanSummaryBkt bucket, it creates a new sub-bucket. In
the subbucket, the ClientChanSummary is then stored under the
cChannelSummary key. The reason for this migration is that it will be
useful in future when we want to store more easily accessible data under
a specific client ID.
This commit upgrades the wtclient package to make use of the new
`AddressIterator`. It does so by first creating new `Tower` and
`ClientSession` types. The new `Tower` type has an `AddressIterator`
instead of a list of addresses. The `ClientSession` type contains a
`Tower`.
In this commit, the new ListClientSession functional options and new
FetchSessionCommittedUpdates function are utilised in order to allow us
to completely remove the CommittedUpdates member from the ClientSession
struct.
In this commit, a new tower client db function is added that can be used
to fetch all the committed updates for a given session ID. This is done
in preparation for an upcoming commit where the CommittedUpdates will be
removed from the ClientSession struct.
In this commit, we start making use of the new ListClientSession
functional options added in the previous commit. We use the functional
options in order to calculate the max commit heights per channel on the
construction of the tower client. We also use the options to count the
total number of acked and committed updates. With this commit, we are
also able to completely remove the AckedUpdates member of the
ClientSession since it is no longer used anywhere in the code.
This commit adds functional options to the ListClientSessions call that
can be used to perform a variety of extra operations during the DB
query. These functional options are not yet used in this commit.
In this commit, the towerID-to-sessionID index added in the previous
commit is put to use in order to make session lookup more efficient in
certain places. In the process, 2 TODO's are also removed from the code.
This commit adds a new towerID-to-sessionID index to the wtclient DB.
The commit also contains the necessary migration required in order to
build the index for an existing client.
This index will greatly improve the lookup of sessions for a given tower
ID.
Before creating a new session, first check that the TowerID that the
ClientSession is referencing refers to an existing tower. This is done
to prevent the situation where RemoveTower is called right before
CreateClientSession is called which would, before this commit, lead to
the session being created with a tower ID that does not refer to any
existing tower.
This commit was previously split into the following parts to ease
review:
- 2d746f68: replace imports
- 4008f0fd: use ecdsa.Signature
- 849e33d1: remove btcec.S256()
- b8f6ebbd: use v2 library correctly
- fa80bca9: bump go modules
The final database that needs to be made remote compatible is the
watchtower server and client database.
They are handled a bit differently because both of them are not always
active, only when specifically turned on in the config.
* mod: bump btcwallet version to accept db timeout
* btcwallet: add DBTimeOut in config
* kvdb: add database timeout option for bbolt
This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.
* channeldb: add DBTimeout in db options
This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.
* contractcourt+routing: use DBTimeout in kvdb
This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.
* lncfg: add DBTimeout option in db config
The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.
* migration: add DBTimeout param in kvdb.Create/kvdb.Open
* keychain: update tests to use DBTimeout param
* htlcswitch+chainreg: add DBTimeout option
* macaroons: support DBTimeout config in creation
This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.
* walletunlocker: add dbTimeout option in UnlockerService
This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.
* watchtower/wtdb: add dbTimeout param during creation
This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.
* multi: add db timeout param for walletdb.Create
This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.
* lnd: pass DBTimeout config to relevant services
This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
- chainControlConfig
- walletunlocker
- wallet.NewLoader
- macaroons
- watchtower
In addition, the usage of wallet.Create is updated too.
* sample-config: add dbtimeout option
This commit introduces a change in the key format used to reserve/lookup
session-key-indexes. Currently the reservations are stored under the
tower id, however this creates issues when multiple clients are using
the same database since only one reservation is permitted per tower.
We fix this by appending the blob type to the session-key-index locator.
This allows multiple clients to reserve keys for the same tower, but
still limits each client to one outstanding reservation. The changes are
made in a way such that we fall back to the legacy format if the a
reservation under the new format is not found, but only if the blob type
matches blob.TypeAltruistCommit, which is so far the only actively
deployed blob type.
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
This addresses a potential panic when a tower has one of its candidate
sessions chosen, but its only reachable address was removed by a
user-initiated RPC before the fact.
In this commit, we create a new chainfee package, that houses all fee
related functionality used within the codebase. The creation of this new
package furthers our long-term goal of extracting functionality from the
bloated `lnwallet` package into new distinct packages. Additionally,
this new packages resolves a class of import cycle that could arise if a
new package that was imported by something in `lnwallet` wanted to use
the existing fee related functions in the prior `lnwallet` package.
We do this as a convenience for WatchtowerClient users so that they do
not need to re-add towers upon restarts. We ensure not to re-add towers
that have been previously removed by determining whether it has any
lingering active sessions.
These operations are currently unused, but will be integrated into the
TowerClient at a later point as future preparation for the
WatchtowerClient RPC subserver, which will allow users to add, remove,
and list the watchtowers currntly in use.
This currently takes O(N) time as there does not exist an index of
active client sessions for each watchtower within the client's database.
This index is likely to be added in the future.
