This commit adds a new interface method `TerminalInfo` and changes its
implementation to return an `*HTLCAttempt` so it includes the route for
a successful payment. Method `GetFailureReason` is now removed as its
returned value can be found in the above method.
This commit removes the method `launchShard` and splits its original
functionality into two steps - first create the attempt, second send the
attempt. This enables us to have finer control over "which error is
returned from which system and how to handle it".
Finally, The LightningNode object is removed from ChannelEdgePolicy.
This is a step towards letting ChannelEdgePolicy reflect exactly the
schema that is on disk.
This is also nice because the `Node` object is not necessarily always
required when the ChannelEdgePolicy is loaded from the DB, so now it
only get's loaded when needed.
In preparation for the next commit which will remove the
`*LightningNode` from the `ChannelEdgePolicy` struct,
`FetchLightningNode` is modified to take in an optional transaction so
that it can be utilised in places where a transaction exists.
To prepare for the `kvdb.Backend` member of `ChannelEdgeInfo` being
removed, the `FetchOtherNode` method is moved from the `ChannelEdgeInfo`
to the `ChannelGraph` struct.
Having a `ForEachChannel` method on the `LightningNode` struct itself
results in a `kvdb.Backend` object needing to be stored within the
LightningNode struct. In this commit, this method is replaced with a
`ForEachNodeChannel` method on the `ChannelGraph` struct will perform
the same function without needing the db pointer to be stored within the
LightningNode. This change, the LightningNode struct more closely
represents the schema on disk.
The existing `ForEachNodeChannel` method on `ChannelGraph` is renamed to
`ForEachNodeDirectedChannel`. It performs a slightly different function
since it's call-back operates on Cached policies.
* multi: extend InvoiceDB methods with a context argument
This commit adds a context to InvoiceDB's methods. Along this refactor
we also extend InvoiceRegistry methods with contexts where it makes
sense. This change is essential to be able to provide kvdb and sqldb
implementations for InvoiceDB.
* channeldb: restrict invoice tests to only use an InvoiceDB instance
* docs: update release notes for 0.18.0
This commit adds the encrypted_data, blinding_point and total_amt_msat
tlvs to the known set of even tlvs for the onion payload. These TLVs
are added in two places (the onion payload and hop struct) because
lnd uses the same set of TLV types for both structs (and they
inherently represent the same thing).
Note: in some places, unit tests intentionally mimic the style
of older tests, so as to be more consistently readable.
This commit refactors the params used in lifecycle to prefer
`HTLCAttempt` over `HTLCAttemptInfo`. This change is needed as
`HTLCAttempt` also wraps settled and failure info, which is useful in
the following commits.
This commit turns `MPPayment` into an interface inside `routing`. Having
this interface gives us the benefit to write more granular unit tests
inside payment lifecycle. As seen from the modified unit tests, several
hacky ways of testing the `SendPayment` method is now replaced by a mock
over `MPPayment`.
This commit adds a new method, `NeedWaitAttempts`, to properly decide
whether we need to wait for the outcome of htlc attempts based on the
payment's current state.
This commit moves the struct `paymentState` used in `routing` into
`channeldb` and replaces it with `MPPaymentState`. In the following
commit we'd see the benefit, that we don't need to pass variables back
and forth between the two packages. More importantly, this state is put
closer to its origin, and is strictly updated whenever a payment is read
from disk. This approach is less error-prone comparing to the previous
one, which both the `payment` and `paymentState` need to be updated at
the same time to make sure the data stay consistant in a parallel
environment.
This commit applies the new method `Terminated`. A side effect from
using this method is, we can now save one query `fetchPayment` inside
`FetchInFlightPayments`.
This commit introduces more granular statuses to better determine a
payment's current state. Based on whether there are inflight HTLCs, the
state of each of the HTLCs, and whether the payment is failed, a total
of 5 states are derived, which can give a finer control over what action
to take based on a given state.
Also, `fetchPayment` now uses `decidePaymentStatus`. After applying the
new function, the returned payment status would be different,
```
| inflight | settled | failed | reason | previous -> now |
|:--------:|:-------:|:------:|:------:|:---------------------------------:|
| true | true | true | yes | StatusInFlight(unchanged) |
| true | true | true | no | StatusInFlight(unchanged) |
| true | true | false | yes | StatusInFlight(unchanged) |
| true | true | false | no | StatusInFlight(unchanged) |
| true | false | true | yes | StatusInFlight(unchanged) |
| true | false | true | no | StatusInFlight(unchanged) |
| true | false | false | yes | StatusInFlight(unchanged) |
| true | false | false | no | StatusInFlight(unchanged) |
| false | true | true | yes | StatusSucceeded(unchanged) |
| false | true | true | no | StatusSucceeded(unchanged) |
| false | true | false | yes | StatusSucceeded(unchanged) |
| false | true | false | no | StatusSucceeded(unchanged) |
| false | false | true | yes | StatusFailed(unchanged) |
| false | false | true | no | StatusInFlight(unchanged) |
| false | false | false | yes | StatusFailed(unchanged) |
| false | false | false | no | StatusInFlight -> StatusInitiated|
```
This commit adds a new method, `Registrable`, to help decide whether
adding new HTLCs to a given payment is allowed. A new unit test,
`TestRegistrable` is also added to test it.
This commit adds a new payment status, `StatusInitiated`, to properly
represent the state where a payment is newly created without attempting
any HTLCs. Since the `PaymentStatus` is a memory representation of a
given payment's status, the enum used for the statuses are directly
updated.
This commit changes `fetchPaymentStatus` to return an error when the
payment creation info bucket cannot be found. Instead of using the
ambiguous status `StatusUnknown`, we tell the callsite explictly that
there's no such payment. This also means the vague `StatusUnknown` can
now be removed as it has multiple meanings.
In this commit, we update the ChanSyncMsg to populate nonce information.
With this change, we can now hide nonce generation further down in the
pipeline and ensure that all callers will have the expected fields
populated.
In this commit, we update the genHtlcSigValidationJobs function to be
taproot aware. As we actually need a schnorr signature for the taproot
validation, we need to coerce the entire wire type into a schnorr sig
with the ForceSchnorr() method.
In this commit, we update the Sig type to support ECDSA and schnorr
signatures. We need to do this as the HTLC signatures will become
schnorr sigs for taproot channels. The current spec draft opts to
overload this field since both the sigs are actually 64 bytes in length.
The only consideration with this move is that callers need to "coerce" a
sig to the proper type if they need schnorr signatures.
We know that onion blobs in lightning are _exactly_ 1366 bytes in
lightning, but they are currently expressed as a byte slice in
channeldb's HTLC struct. Blobs are currently serialized as var bytes,
so we can take advantage of this known length and variable length
to add additional data to the inline serialization of our HTLCs, which
are otherwise not easily extensible (without creating a new bucket).
We add a Memo field to the OpenChannel DB struct. We also persist
it using a tlv record. We then pass the Memo value from the
InitFundingReserveMsg when creating a new reservation for the channel.
Finally, we also read Memo field when fetching channel from DB.
This commit introduces a new `channelSelector` method and moves all
generic logic from `FetchChannel` to it. This refactor will make it
easier to add new methods that require the same open-channel db
traversal with slightly different channel selection logic.
This commit adds a small optimisation to the FetchChannel method.
Instead of iterating over each channel bucket, an identifiable error is
thrown once the wanted channel is found so that the iteration can stop
early.
This commit replaces `FundingLocked` found in docs using the following
command,
```shell
find . -name "*.go" -exec sed -i '' 's/FundingLocked/ChannelReady/g' {} \;
find . -name "*.go" -exec sed -i '' 's/FundingLock/ChannelReady/g' {} \;
```
All the updates type are now catched in a switch statement, we can
delete the rest of the body of this function + add a default path for
not matched flows.
Previous to this commit we were able to call `UpdateInvoice` with an
updated that added and cancelled htlcs at the same time. The function
returned an error if there was overlapping between the two htlc set.
However, that behavior was not used in the LND code itself.
Eventually we want to split this method in multiple ones, among them one
for canceling htlcs and another one for adding them. For that reason,
this behavior is not supported anymore.
We rename `ChanUpdateOptionMaxHtlc` to `ChanUpdateRequiredMaxHtlc`
as with the latest changes it is now required.
Similarly, rename `validateOptionalFields` to
`ValidateChannelUpdateFields`, export it to use it in a later commit.
This commit re-adds the LocalBalance and RemoteBalance fields to the
RevocationLog. The channeldb/migration30 is also adjusted so that anyone
who has not yet run the optional migration will not lose these fields if
they run the migration after this commit.
The reason for re-adding these fields is that they are needed if we want
to reconstruct all the info of the lnwallet.BreachRetribution without
having access to the breach spend transaction. In most cases we would
have access to the spend tx since we would see it on-chain at which time
we would want to reconstruct the retribution info. However, for the
watchtower subsystem, we sometimes want to construct the retribution
info withouth having access to the spend transaction.
A user can use the `--no-rev-log-amt-data` flag to opt-out of storing
these amount fields.
In this commit, a new `--db.no-rev-log-amt-data` flag is added. The
config option is passed though to everywhere that it will be used. Note
that it is still a no-op in this commit. An upcoming commit will make
use of the flag.
Define a MigrationConfig interface that should be used to pass the
config of an optional migration function. An implementation of this
interface is added for migration30 (the only current optional
migration).
When a block is disconnected due to a reorg, DisconnectBlockAtHeight
is called for the block height. Prior to this patch, it would delete
every SCID in the graph with a block height greater than the
disconnected height. This meant that a reorg would delete every
zero-conf channel edge from the graph. The fix simply iterates up
until the StartingAlias and deletes every SCID between the
disconnected height and the StartingAlias height.
The only way to know if an invoice is AMP, Keysend, etc is to look at
its shape/characteristics. This commit adds a couple of helper functions
to encapsulate the logic of these checks.
If all these types cannot intersect: an invoice cannot be AMP and
Keysend or Keysend and Bolt12, etc it could be useful to add an extra
field to store this information instead of relying on checking how the
invoice looks like.
Now that we have the new package `lnd/channeldb/models` we can invert the
depenency between `channeldb` and `invoices`.
- Move all the invoice related types and errors to the
`invoices` package.
- Ensure that all the packages dealing with invoices use the types and
interfaces defined in the `invoices` package.
- Implement the InvoiceDB interface (defined in `lnd/invoices`) in
channeldb.
- Add new mock for InterfaceDB.
- `InvoiceRegistery` tests are now in its own subpacakge (they need to
import both invoices & channeldb). This is temporary until we can
decouple them.
Add a new subpackage to `lnd/channeldb` to hold some of the types that
are used in the package itself and in other packages that should not
depend on `channeldb`.
This commit moves the `HeightHintCache` implementation to the
`channeldb` package and inverts the dependency relation between
`chainntnfs` and `channeldb`.
Many packages depend on channeldb for type definitions,
interfaces, etc. `chainntnfs` is an example of that. `chainntnfs`
defines the `SpendHintCache` and `ConfirmHintCache` interfaces but
it also implments them (`HeightHintCache` struct). The implementation
uses logic that should not leak from channeldb (ex: bucket paths).
This makes our code highly coupled + it would not allow us to use any
of these interfaces in a package that is imported by `channeldb`
(circular dependency).
To avoid running a channel DB that was successfully migrated to another
system by accident, we check if there is a tombstone marker before we
open the DB for use.
We'll need to know whether a database was migrated to the latest version
in our upcoming data migration tool. To be able to determine the current
version of a DB and the total number of migrations in existence, we need
to export some of the functions in channeldb.
We also add some helper functions for adding tombstone and other markers
to a database.
This commit changes how we locate the next migration height by including
the scenario where `lnd@v0.15.0` is active. In the new version, we will
see a mixed of new and old logs under the same open channel bucket.
Hence, we need to alter how we locate the next un-migrated height.
This commit enables the db to run optional migrations that are specified
by config flags. To achieve this, an optional meta is introduced to
manage the optional migrations. We distinguish the two types of
migrations here so it's easier to manage them for the concern a future
migration can cause trouble for us to determine the db version if we
don't.
This commit adds the migration that's used to convert the old revocation
logs into the new format. The migration is fault-tolerant, meaning the
process can be interrupted and the migration will pick up what's left
when running again. We also cap how many records to be processed in each
db transaction to prevent OOM.
This commit adds supporting functions that will be used in the unit
test. The testing data are also added as hard-coded. We choose to copy
the most of the testing data from our itest results such that a) they
are "real" data that can be used to calculate scripts and b) we preserve
the result generated by the current code so a future change won't affect
our test.
This commit adds several utility functions to assist the migration. In
particular, an updateLocator is added to gives us the next un-migration
position in our buckets. This locator helps us to continue the job
in case of an interrupted migration. It also serves as an indicator on
whether the migration is finished or not.
This commit adds relevant code from the revocation_log.go and the
package lnwallet. The code is needed to migrate the data, and we choose
to copy the code instead of importing to preserve the version such that
a future change won't affect current migration. An alternative would be
tagging each of the packages imported.
feature-bit channels
This allows opening zero-conf chan-type, scid-alias chan-type, and
scid-alias feature-bit channels. scid-alias chan-type channels are
required to be private. Two paths are available for opening a zero-conf
channel:
* explicit chan-type negotiation
* LDK carve-out where chan-types are not used, LND is on the
receiving end, and a ChannelAcceptor is used to enable zero-conf
When a zero-conf channel is negotiated, the funding manager:
* sends a FundingLocked with an alias
* waits for a FundingLocked from the remote peer
* calls addToRouterGraph to persist the channel using our alias in
the graph. The peer's alias is used to send them a ChannelUpdate.
* wait for six confirmations. If public, the alias edge in the
graph is deleted and replaced (not atomically) with the confirmed
edge. Our policy is also read-and-replaced, but the counterparty's
policy won't exist until they send it to us.
When a scid-alias-feature channel is negotiated, the funding manager:
* sends a FundingLocked with an alias:
* calls addToRouterGraph, sends ChannelUpdate with the confirmed SCID
since it exists.
* when six confirmations occurs, the edge is deleted and re-inserted
since the peer may have sent us an alias ChannelUpdate that we are
storing in the graph.
Since it is possible for a user to toggle the scid-alias-feature-bit
to on while channels exist in the funding manager, care has been taken
to ensure that an alias is ALWAYS sent in the funding_locked message
if this happens.
An OptionalMsgField has been added that allows outside subsystems
to provide a short channel id we should insert into a ChannelUpdate
that we then sign and send to our peer.
When the gossiper receives a ChannelUpdate, it will query the
alias manager by the passed-in FindBaseByAlias function to determine
if the short channel id in the ChannelUpdate points to a known
channel. If this lookup returns an error, we'll fallback to using
the original id in the ChannelUpdate when querying the router.
The lookup and potential fallback must occur in order to properly
lock the multimutex, query the correct router channels, and rate
limit the correct short channel id. An unfortunate side effect of
receiving ChannelUpdates from our peer that reference on of our
aliases rather than the real SCID is that we must store this policy.
Yet it is not broadcast-able. Care has been taken to ensure the
gossiper does not broadcast *any* ChannelUpdate with an alias SCID.
The cachedNetworkMsg uses the new processedNetworkMsg struct. This
is necessary so that delete-and-reinsert in the funding manager
doesn't process a ChannelUpdate twice and end up in a deadlock since
the err chan is no longer being used.
This introduces a BigSize migration that is used to expand the width
of the ChannelStatus and ChannelType fields. Three channel "types"
are added - ZeroConfBit, ScidAliasChanBit, and ScidAliasFeatureBit.
ScidAliasChanBit denotes that the scid-alias channel type was
negotiated for the channel. ScidAliasFeatureBit denotes that the
scid-alias feature bit was negotiated during the *lifetime* of the
channel. Several helper functions on the OpenChannel struct are
exposed to aid callers from different packages.
The RefreshShortChanID has been renamed to Refresh.
A new function BroadcastHeight is used to guard access to the
mutable FundingBroadcastHeight member. This prevents data races.
When testing serializing revocation log, we need to also copy its
`HTLCEntries` as the serialization of the HTLC involves a writing to the
`htlc.amtTlv` field.
Previously, in `migration25.OpenChannel`, there was a private field
`chanStatus` used to keep track of the channel status. The following
migrations, `migration26` and `migration27` also have their own
`OpenChannel` defined, with `migration26` inherited from `migration25`,
and `migration27` inherited from `migration26`. The private field
`chanStatus`, however, is NOT inherited and each of the migrations uses
its own. This is fine for reading and writing as, under the hood, the
`chanStatus` is just a `uint8` value. Because each migration has its own
fetcher and putter, it can safely access its private field to read and
write it correctly.
The issue pops up when we use the method
`migration25.FundingTxPresent()`. Because it's evaluating its channel
status using its own private field `chanStatus`, this field would always
be the default value(`ChanStatusDefault`), leading the statement
`!c.hasChanStatus(ChanStatusRestored)` to always be true. Thus a
restored channel will be mistakenly considered to have funding tx
present, causing failures in reading the channel info in the following
migrations.
We fix this by exporting the `ChanStatus` field so its value can be set
by following migrations.
This commit adds a new migration to patch the two balance fields,
`InitialLocalBalance` and `InitialRemoteBalance` for the historical
channels. Because they are not saved previously, for historical channels
prior to the revocation log PR, these fields will be empty.
This can cause an intermittent panic otherwise if bbolt remaps itself
via munmap and mmap. From bbolt's documentation:
* Byte slices returned from Bolt are only valid during a transaction.
Once the transaction has been committed or rolled back then the memory
they point to can be reused by a new page or can be unmapped from
virtual memory and you'll see an unexpected fault address panic when
accessing it.
This commit uses bigsize record to encode the htlc amount, which could
save us 3 more bytes if the encoded value is no greater than roughly
0.043 bitcoin. The uint test has been updated with a more realistic
values to reflect the actual gain.
This commit adds `revocationLogBucket` using the new sub-bucket key
"revocation-log" to store the minimal info needed. Two structs,
`RevocationLog` and `HTLCEntry` are created to represent the disk
records.
This commit deletes the function `BalancesAtHeight` since its only
usague is to find the push amount, which can be achieved by saving the
initial balances.
Another reason to remove it is to pave the way to incooperate our new
revocation log. If we ever need this function again, we can add it back
by visiting all the revocation logs to calculate the balances at a given
height.
This commit replaces the method RevocationLogTail which in used in our
unit test with a private method revocationLogTailCommitHeight. The new
method returns the commit height only since that's what's needed in the
test.
This commit was previously split into the following parts to ease
review:
- 2d746f68: replace imports
- 4008f0fd: use ecdsa.Signature
- 849e33d1: remove btcec.S256()
- b8f6ebbd: use v2 library correctly
- fa80bca9: bump go modules
