To avoid running a channel DB that was successfully migrated to another
system by accident, we check if there is a tombstone marker before we
open the DB for use.
We'll need to know whether a database was migrated to the latest version
in our upcoming data migration tool. To be able to determine the current
version of a DB and the total number of migrations in existence, we need
to export some of the functions in channeldb.
We also add some helper functions for adding tombstone and other markers
to a database.
This commit changes how we locate the next migration height by including
the scenario where `lnd@v0.15.0` is active. In the new version, we will
see a mixed of new and old logs under the same open channel bucket.
Hence, we need to alter how we locate the next un-migrated height.
This commit enables the db to run optional migrations that are specified
by config flags. To achieve this, an optional meta is introduced to
manage the optional migrations. We distinguish the two types of
migrations here so it's easier to manage them for the concern a future
migration can cause trouble for us to determine the db version if we
don't.
This commit adds the migration that's used to convert the old revocation
logs into the new format. The migration is fault-tolerant, meaning the
process can be interrupted and the migration will pick up what's left
when running again. We also cap how many records to be processed in each
db transaction to prevent OOM.
This commit adds supporting functions that will be used in the unit
test. The testing data are also added as hard-coded. We choose to copy
the most of the testing data from our itest results such that a) they
are "real" data that can be used to calculate scripts and b) we preserve
the result generated by the current code so a future change won't affect
our test.
This commit adds several utility functions to assist the migration. In
particular, an updateLocator is added to gives us the next un-migration
position in our buckets. This locator helps us to continue the job
in case of an interrupted migration. It also serves as an indicator on
whether the migration is finished or not.
This commit adds relevant code from the revocation_log.go and the
package lnwallet. The code is needed to migrate the data, and we choose
to copy the code instead of importing to preserve the version such that
a future change won't affect current migration. An alternative would be
tagging each of the packages imported.
feature-bit channels
This allows opening zero-conf chan-type, scid-alias chan-type, and
scid-alias feature-bit channels. scid-alias chan-type channels are
required to be private. Two paths are available for opening a zero-conf
channel:
* explicit chan-type negotiation
* LDK carve-out where chan-types are not used, LND is on the
receiving end, and a ChannelAcceptor is used to enable zero-conf
When a zero-conf channel is negotiated, the funding manager:
* sends a FundingLocked with an alias
* waits for a FundingLocked from the remote peer
* calls addToRouterGraph to persist the channel using our alias in
the graph. The peer's alias is used to send them a ChannelUpdate.
* wait for six confirmations. If public, the alias edge in the
graph is deleted and replaced (not atomically) with the confirmed
edge. Our policy is also read-and-replaced, but the counterparty's
policy won't exist until they send it to us.
When a scid-alias-feature channel is negotiated, the funding manager:
* sends a FundingLocked with an alias:
* calls addToRouterGraph, sends ChannelUpdate with the confirmed SCID
since it exists.
* when six confirmations occurs, the edge is deleted and re-inserted
since the peer may have sent us an alias ChannelUpdate that we are
storing in the graph.
Since it is possible for a user to toggle the scid-alias-feature-bit
to on while channels exist in the funding manager, care has been taken
to ensure that an alias is ALWAYS sent in the funding_locked message
if this happens.
An OptionalMsgField has been added that allows outside subsystems
to provide a short channel id we should insert into a ChannelUpdate
that we then sign and send to our peer.
When the gossiper receives a ChannelUpdate, it will query the
alias manager by the passed-in FindBaseByAlias function to determine
if the short channel id in the ChannelUpdate points to a known
channel. If this lookup returns an error, we'll fallback to using
the original id in the ChannelUpdate when querying the router.
The lookup and potential fallback must occur in order to properly
lock the multimutex, query the correct router channels, and rate
limit the correct short channel id. An unfortunate side effect of
receiving ChannelUpdates from our peer that reference on of our
aliases rather than the real SCID is that we must store this policy.
Yet it is not broadcast-able. Care has been taken to ensure the
gossiper does not broadcast *any* ChannelUpdate with an alias SCID.
The cachedNetworkMsg uses the new processedNetworkMsg struct. This
is necessary so that delete-and-reinsert in the funding manager
doesn't process a ChannelUpdate twice and end up in a deadlock since
the err chan is no longer being used.
This introduces a BigSize migration that is used to expand the width
of the ChannelStatus and ChannelType fields. Three channel "types"
are added - ZeroConfBit, ScidAliasChanBit, and ScidAliasFeatureBit.
ScidAliasChanBit denotes that the scid-alias channel type was
negotiated for the channel. ScidAliasFeatureBit denotes that the
scid-alias feature bit was negotiated during the *lifetime* of the
channel. Several helper functions on the OpenChannel struct are
exposed to aid callers from different packages.
The RefreshShortChanID has been renamed to Refresh.
A new function BroadcastHeight is used to guard access to the
mutable FundingBroadcastHeight member. This prevents data races.
When testing serializing revocation log, we need to also copy its
`HTLCEntries` as the serialization of the HTLC involves a writing to the
`htlc.amtTlv` field.
Previously, in `migration25.OpenChannel`, there was a private field
`chanStatus` used to keep track of the channel status. The following
migrations, `migration26` and `migration27` also have their own
`OpenChannel` defined, with `migration26` inherited from `migration25`,
and `migration27` inherited from `migration26`. The private field
`chanStatus`, however, is NOT inherited and each of the migrations uses
its own. This is fine for reading and writing as, under the hood, the
`chanStatus` is just a `uint8` value. Because each migration has its own
fetcher and putter, it can safely access its private field to read and
write it correctly.
The issue pops up when we use the method
`migration25.FundingTxPresent()`. Because it's evaluating its channel
status using its own private field `chanStatus`, this field would always
be the default value(`ChanStatusDefault`), leading the statement
`!c.hasChanStatus(ChanStatusRestored)` to always be true. Thus a
restored channel will be mistakenly considered to have funding tx
present, causing failures in reading the channel info in the following
migrations.
We fix this by exporting the `ChanStatus` field so its value can be set
by following migrations.
This commit adds a new migration to patch the two balance fields,
`InitialLocalBalance` and `InitialRemoteBalance` for the historical
channels. Because they are not saved previously, for historical channels
prior to the revocation log PR, these fields will be empty.
This can cause an intermittent panic otherwise if bbolt remaps itself
via munmap and mmap. From bbolt's documentation:
* Byte slices returned from Bolt are only valid during a transaction.
Once the transaction has been committed or rolled back then the memory
they point to can be reused by a new page or can be unmapped from
virtual memory and you'll see an unexpected fault address panic when
accessing it.
This commit uses bigsize record to encode the htlc amount, which could
save us 3 more bytes if the encoded value is no greater than roughly
0.043 bitcoin. The uint test has been updated with a more realistic
values to reflect the actual gain.
