It may happen that we do pathfinding while also attempt to change the
graph. In this case changing the database, channel state, graph while
reading from the same sources may create deadlocks. To resolve this we
change locking policy in the graph cache when pathfinding.
This commit, adds a new ForEachNode method to the channel graph cache
that assumes the contents won't be modified. This is generally useful,
and will be used in a later commit to optimize some heavy RPC calls.
In this commit, we update the logic in `updateInvoice` to allow callers
to pass in either a hint, or the setID in the update callback. This
makes things more efficient for AMP invoices with thousands of recurring
payments, as we no longer need to read out _all_ the invoices each time
we go to update the state of a few HTLCs.
This change allows us to deliver notifications to a user of all the
settled recurring payments to the same payment_addr in the order that
they occurred.
In this commit, we modify the way we handle state updates for AMP
invoices. With the current logic, once an invoice is settled, we'll
update the primary invoice state. This means that a user can't take the
same payment_addr, w/ a new set_id and pay the invoice again.
To remedy this, we'll move to instead _not_ updating the main invoice
state each time a new htlc Set (group of HTLCs according to setID) is
added. Instead, given that each HTLC stores an individual state, we'll
instead just use that directly from now on.
We also update TestSetIDIndex to account for new repeated settle AMP
logic.
In this commit, we modify the HTLC storage for AMP invoice only to be
stored within a new key prefix next to the main invoice data. We do this
as otherwise each time an AMP invoice is settled (enabled by the
following commits), we need to continually encode+decode the _entire_
set of invoices.
Instead, we store the AMP invoices within the main invoice bucket, using
the invoice number as a key prefix, with the final key being:
`invoiceNum || setID`
In this commit, we add a new type `AMPInvoiceState` that's used to store
AMP sub-invoice meta data alongside the main invoice. This will be used
to allow changes to be made to an AMP invoices without reading out all
the HTLCs. In addition, callers can use this metadata to look up
information about the current sub-invoice state of AMP HTLCs.
This commit partially reverts bf27d05a.
To avoid creating multiple database transactions during a single path
finding operation, we create an explicit transaction when the cached
graph is instantiated.
We cache the source node to avoid needing to look that up for every path
finding session.
The database transaction will be nil in case of the in-memory graph.
With this commit we forward the config option for disabling the channel
graph cache as a boolean to the channeldb. But we invert its meaning to
make the flag easier to understand.
Fixes#5830.
When a channel for a node is announced before the node itself is
announced on the network, it's possible that we have channels for a node
but no features defined yet. This was previously logged as a warning
which spammed the log unnecessarily.
With this commit we use an optimized version of the node iteration that
causes fewer memory allocations by only loading the part of the graph
node that we actually need to know for the cache.
To avoid the channel map needing to be re-grown while we fill the cache
initially, we might as well pre-allocate it with a somewhat sane value
to decrease the number of grow events.
To further separate the channel graph from the channel state, we
refactor the AddrsForNode method to use the graphs's public methods
instead of directly accessing any buckets. This makes sure that we can
have the channel state cached with just its buckets while not using a
kvdb level cache for the graph.
At the same time we refactor the graph's test to also be less dependent
upon the channel state DB.
The funding manager doesn't need to know the details of the underlying
storage of the opening channel state, so we move the actual store and
retrieval into the channel database.
Adds `DeletePayment` to the channeldb, which allows to delete a single payment. If only failed HTLCs for this payment should be deleted it can be specified by the bool `failedHtlcsOnly`.
Adds an optional tx parameter to ForAllOutgoingChannels and FetchChannel
so that data can be queried within the context of an existing database
transaction.
This commit changes the WriteElement and WriteElements methods to take a
write buffer instead of io.Writer. The corresponding Encode methods are
changed to use the write buffer.
Depends on btcsuite/btcwallet#757.
Pulls in the updated version of btcwallet and walletdb that have the DB
interface enhanced by their own View() and Update() methods with the
reset callback/closure supported out of the box. That way the global
package-level View() and Update() functions now become pure redirects.
The bucket sequence we use as payment sequence makes the DB update in
InitPayment conflict and therefore queue up when many concurrent
payments happen. This change allocates payment sequences in blocks
of 1000 to avoid these conflicts.
In this commit, we add a new method that allows us to mark a channel as
being a zombie on the fly without needing to go through the normal
channel deletion process.
Since we want to support AMP payment using a different unique payment
identifier (AMP payments don't go to one specific hash), we change the
nomenclature to be Identifier instead of PaymentHash.
Move our more generic terminal check forward so that we only
need to handle a single class of expected errors. This change
is mirrored in our mock, and our reproducing tests are updated
to assert that this move catches both classes of errors we get.
In this commit, we add strict zombie pruning as a config level param.
This allow us to add the option for those that want a tighter graph, and
not change the default composition of the channel graph for most users
over night.
In addition, we expand the test case slightly by testing that the self
node won't be pruned, but also that if there's a node with only a single
known stale edge, then both variants will prune that edge.
This commit relaxes DeleteInvoice failure cases by removing the
requirement that the invoice needs to be indexed by the payment address.
Since payment address index was introduced with an empty migration it
is possible that users have old invoices which were never added to
this index causing invoice garbage collection to get stuck.
Without this check, it's possible to send a probe HTLC w/ a valid
payment address but invalid payment hash that gets settled. Because
there was no assertion that the HTLC's payment hash matches the
invoice, the link would fail when it receives an invalid preimage for
the HTLC on its commitment.
This PR was created in order to help to debug the failures in:
https://github.com/lightningnetwork/lnd/issues/5113
We add some more contextual errors so we can figure out where the
assumptions of the current scan to delete function in the invoice
registry is incorrect.
This commit adds a RevocationKeyLocator field to the OpenChannel
struct so that the SCB derivation doesn't have to brute-force the
sha chain root key and match the public key. ECDH derivation is now
used to derive the key instead of regular private key derivation a
la DerivePrivKey. The legacy can still be used to recover old
channels.
Prior to AMP, there could only be one HTLC set. Now that there can be
multiple, we introduce the inherent risk that we might be persisting a
settle/accept of the wrong HTLC set. To migitigate this risk, we add a
check to assert that an invoice can only be transitioned into an
Accepted or Settled state if the specified HTLC set is non-empty, i.e.
has accepted HTLCs.
To do this check, we unroll the loops in UpdateInvoice to first process
any added or canceled HTLCs. This ensures that the set of accepted HTLCs
is up-to-date when we got to transition the invoice into a new state, at
which point we can accurately check that the HTLC set is non-empty.
Previously this sort of check wasn't possible because a newly added HTLC
wouldn't be populated on the invoice when going to call
updateInvoiceState.
Once the invoice-level transition checks have completed, we complete a
final pass to move the HTLCs into their final states and recompute the
amount paid.
With this refactor, it is now possible to make stronger assertions
about the invoice and the state transitions being made by the
invoiceregistry before those changes are ultimately persisted.
Messages:
- UpdateFulfillHTLC
- UpdateFee
- UpdateFailMalformedHTLC
- UpdateFailHTLC
- UpdateAddHTLC
- Shutdown
- RevokeAndAck
- ReplyShortChanIDsEnd
- ReplyChannelRange
- QueryShortChanIDs
- QueryChannelRange
- NodeAnnouncement
- Init
- GossipTimestampRange
- FundingSigned
- FundingLocked
- FundingCreated
- CommitSig
- ClosingSigned
- ChannelUpdate
- ChannelReestablish
- ChannelAnnouncement
- AnnounceSignatures
lnwire: update quickcheck tests, use constant for Error
multi: update unit tests to pass deep equal assertions with messages
In this commit, we update a series of unit tests in the code base to now
pass due to the new wire message encode/decode logic. In many instances,
we'll now manually set the extra bytes to an empty byte slice to avoid
comparisons that fail due to one message having an empty byte slice and
the other having a nil pointer.
In this commit, we migrate all wire messages in the database from the
`legacy` to the `current` encoding.
This affects the way we write the `CommitDiff` and `LogUpdates` struct
to disk. We also need to migrate the network results bucket in the
switch as it includes a wire message without a length prefix.
We copy the new serialization togic to a new package 'current' and
export the methods we will need from the migration. This ensures that we
have isolation between the legacy and current serialization methods,
cleanly inidcating what type we are using during the migration.
In this commit, we modify the way we write wire messages across the
entire database. We'll now ensure that we always write wire messages
with a length prefix. We update the `codec.go` file to always write a 2
byte length prefix, this affects the way we write the `CommitDiff` and
`LogUpdates` struct to disk, and the network results bucket in the
switch as it includes a wire message.
The legacy encoding depends on the lnwire21 version of lnwire, so it will
let us change lnwire after the migration. To make sure it is separated
from the new encoding, we add it to a new package 'legacy'.
We also put common types in a new package 'common', which will house
types that won't change during the migration, and can be used by both
legacy and current serialization code.
To avoid code changing underneath the static migrations, we copy the
lnwire dependency in its current form into the migration directory.
Ideally the migrations doesn't depend on any code that might change,
this takes us a step closer.
This commit adds support for etcd namespaces. This is useful when
using a shared etcd database where separate users have access to
separate namespaces.
Adds an outpoint index that stores a tlv stream. Currently the stream
only contains the outpoint's indexStatus. This should cut down on
big bbolt transactions in several places throughout the codebase.
Previously we wouldn't recreate some of the top level buckets that are
now considered expected with our migration logic. This bug was
preexisting, but never surfaced because the other TLB buckets were not
touched by this unit test.
* mod: bump btcwallet version to accept db timeout
* btcwallet: add DBTimeOut in config
* kvdb: add database timeout option for bbolt
This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.
* channeldb: add DBTimeout in db options
This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.
* contractcourt+routing: use DBTimeout in kvdb
This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.
* lncfg: add DBTimeout option in db config
The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.
* migration: add DBTimeout param in kvdb.Create/kvdb.Open
* keychain: update tests to use DBTimeout param
* htlcswitch+chainreg: add DBTimeout option
* macaroons: support DBTimeout config in creation
This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.
* walletunlocker: add dbTimeout option in UnlockerService
This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.
* watchtower/wtdb: add dbTimeout param during creation
This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.
* multi: add db timeout param for walletdb.Create
This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.
* lnd: pass DBTimeout config to relevant services
This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
- chainControlConfig
- walletunlocker
- wallet.NewLoader
- macaroons
- watchtower
In addition, the usage of wallet.Create is updated too.
* sample-config: add dbtimeout option
Similar to what we did for the local state handling, we extract handling
all known remote states we can act on (breach, current, pending state)
into its own method.
Since we want to handle the case where we lost state (both in case of
local and remote close) last, we don't rely on the remote state number
to check which commit we are looking at, but match on TXIDs directly.
This change was largely motivated by an increase in high disk usage as a
result of channel update spam. With an in memory graph, this would've
gone mostly undetected except for the increased bandwidth usage, which
this doesn't aim to solve yet. To minimize the effects to disks, we
begin to rate limit channel updates in two ways. Keep alive updates,
those which only increase their timestamps to signal liveliness, are now
limited to one per lnd's rebroadcast interval (current default of 24H).
Non keep alive updates are now limited to one per block per direction.
This commit adds the compaction feature of the bbolt compact to our bolt
backend. This will try to open the DB in read-only mode and create a
compacted copy in a temporary file. Once the compaction was successful,
the temporary file and the old source file are swapped atomically.
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
This commit adds commitQueue which is a lightweight contention manager
for STM transactions. The queue attempts to queue up transactions that
conflict for sequential execution, while leaving all "unblocked"
transactons to run freely in parallel.
Since we will use peer flap rate to determine how we rate limit, we
store this value on disk per peer per channel. This allows us to
restart with memory of our peers past behaviour, so we don't give badly
behaving peers have a fresh start on restart. Last flap timestamp is
stored with our flap count so that we can degrade this all time flap
count over time for peers that have not recently flapped.
