Splt the TxPublisher in many smaller actors with clear responsibilities.
Each tx publishing attempt is its own actor and watches the tx until it
either confirms or becomes evicted, and reports the result to its parent.
The TxPublisher (one per channel) orchestrates publishing attempts and
will in the future decide to RBF txs based on deadline information.
The `payment_secret` feature was made mandatory in #1810 and is the default
in other implementations as well. We can thus force it to be available when
decoding onion payloads, which simplifies downstream components (no need
to handle the case where a `payment_secret` may be missing anymore).
We also rename messages in `PaymentInitiator` to remove the confusion with
Bolt 11 payment requests.
There are no functional changes, but bitcoin-lib 0.19 is based on secp256k1-kmp (instead of our own fork of secp256k1's JNI wrapper) which is cleaner, easier to maintain and used in our mobile apps.
With the move to akka _typed_, we will be using more and more
scalatest's `eventually` as a replacement for akka's `awaitCond`
(which isn't available in `testkit.typed`).
But there is a catch:
- `awaitCond` expects a boolean
- `eventually` expects a non-failure
Which means that we must use `eventually(assert(cond))`, and not
`eventually(cond)`.
Make `Setup.datadir` visible to code that receives an instance of
`Setup`. This allows plugin to know where the eclair data directory
is and potentially enrich it.
The spec defines `max_accepted_htlcs` and `max_htlc_value_in_flight_msat`
to let nodes reduce their exposure to pending HTLCs. This only applies to
received HTLCs, and we use the remote peer's values for outgoing HTLCs.
But when we're more restrictive than our peer, it makes sense to apply our
limits to outgoing HTLCs as well.
This allows us to use the full power of scala collections, to iterate
over results, convert to options, etc. while staying purely functional
and immutable.
There is a catch though: the iterator is lazy, it must be materialized
before the result set is closed, by converting the end result in a
collection or an option. In other words, database methods must never
return an `Iterable` or `Iterator`.
* Reduce log level for explorer API errors
* Reduce log level for remote peer invalid open_channel
* Don't send duplicate commands in PostRestartHtlcCleaner: if there
is already a pending HTLC settlement command in the DB, the post
restart handler should let the channel replay it instead of sending
a conflicting command.
* Workaround for lnd bug in reestablish: sometimes lnd sends
announcement_signatures before sending their channel reestablish.
This is a minor spec violation, we can simply delay the message and
handle it later (hopefully once we've received their reestablish).
* Log shared secrets in Sphinx error: Breez sometimes returns errors
that we fail to parse. Unfortunately we didn't correctly log the shared
secrets because the variable was shadowed, so we can't investigate
further for now.
* Fix utxo metric checks: if we're unable to fetch the number of
unconfirmed parents for a utxo, this shouldn't cause the global utxo
check to fail. We log a warning and let operations continue to ensure
the metric is updated.
* Handle ChannelIdAssigned when disconnected: there may be a race
condition where a peer disconnect in the middle of a channel id assignment.
In that case, we still want to record the up-to-date mapping.
Naming was confusing because it led to believe messages were related to
htlcs that have not yet been relayed, whereas those are settlement
messages, meaning that htlcs have relayed and are pending resolution
upstream.
The database has been renamed to a more generic `PendingCommandsDb`
because we may store other types of commands for which we need reliable
delivery.
In case of catastrophic failures of the `SecureRandom` instance, we add
a secondary randomness source that we mix into the random stream.
This is a somewhat weak random source and should not be used on its own,
but it doesn't hurt to xor it with the output of `SecureRandom`.
We use an actor that listens to events in the system and inject them
in our weak pseudo-RNG.
One of ZMQ's drawbacks is that subscribers on an unreliable network may
silently disconnect from publishers in case of network failures.
In our case, we want to reconnect immediately when that happens, so we set
a tcp keep-alive to ensure this.
Fixes#1789
This is a security feature that has been introduced a long time ago and is
widely supported across the network.
We can safely make it mandatory which closes probing attack vectors.
* Update default path-finding weight ratios
* The two months window for channel age was too small for today's network
* CLTV is less of an issue nowadays: there are fewer stuck payments and
we're encouraging nodes to increase their CLTV because of the recent
mempool congestions
There are two cases depending on whether you use weight ratios or not.
They used to behave very differently:
* Without weight ratios, the weight would be the total amount to send (base amount + fees)
* With weight ratios, the ratios would apply to the total amount, not just the fees.
The effect is that only the number of hops and then the weight factors matter as
the fee itself is negligible compared to the total amount.
The code is now shared and the weight is now the sum of the fees
(multiplied by a factor if using weight ratios).
We use one actor per topic, but each actor previously registered to multiple
topics so we received duplicate events and consumed twice the necessary
bandwidth.
We don't need `SERIALIZABLE` consistency guarantees when all we do is
updating timestamp columns. This happens concurrently to channel data
update and raised serialization errors in postgres.
Fixed#1786.
Co-authored-by: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
We need to group incoming HTLCs together by payment_hash and payment_secret,
otherwise we will reject valid payments that are split into multiple distinct
trampoline parts (same payment_hash but different payment_secret).
Fixes#1723
Did some refactoring in tests and introduced a new `migrationCheck`
helper method.
Note that the change of data type in sqlite for the `commitment_number`
field (from `BLOB` to `INTEGER`) is not a migration. If the table has
been created before, it will stay like it was. It doesn't matter due to
how sqlite stores data, and we make sure in tests that there is no
regression.
* rework the db version management
The way our `getVersion`/`setVersion` was very unintuitive and led to
comments like the following in tests:
```scala
dbs.getVersion(statement, "network", 1) // this will set version to 1
```
The reason is that we treat unitialized databases and up-to-date
databases exactly the same way. That is why a `getVersion` will set the
version to the most up-to-date if none is found. It's also why we use
`CREATE IF NOT EXISTS` statements for tables and indices.
With this change, the `getVersion` now only _gets_ the version, or
returns `None` if no version exists.
Since we now differentiate between uninitialized and up-to-date db, we
don't need to make the create statements idempotent. This makes the code
more strict, and we will see errors if our versioning system has bugs.
Internal tables (used for versioning, locking, etc.) have been left
unchanged.
Co-authored-by: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
* use prepared statements for pruning
* optional read-only user
It is good practice to create a dedicated read-only user to browse the
database safely. But since the app itself creates its tables, the
postgres user is the owner and a manual `GRANT` is required everytime a
new table is added.
This PR makes it possible to specify an arbitrary username, that will be
granted read-only access to all tables in the `public` schema.
NB: The assumption here is that eclair is the only app using the
eclair database (in the `CREATE DATABASE eclair` sense), which I believe
is reasonable.
* set timezone on lease table
This only affects newly created table, there is no migration.
Users that are already using postgres will keep the previous column
type, it doesn't change anything for them.
* put back lock timeout on lease table
We use a timeout, because due to concurrency we may not be able to
obtain a lock immediately.
The timeout has been set to its original value of 5s and made
configurable.
* obtain lock before initializing tables
* Add trampoline info to auditDB
Add a new table containing the recipient and amount sent to the recipient in case of trampoline relaying.
When using trampoline, the recipient may not be the next node on the path.
We don't want a database error to cause force close of channels.
Database errors are more likely to happen when using Postgres, but can
also happen with Sqlite in case of e.g. full disk.
Since we always write to disk before sending messages, we should be able
to recover gracefully after the db issue is fixed and eclair is
restarted.
The `ReconnectionTask` was only catching
`ConnectionResult.Failure.ConnectionFailed`, which is a subset of
possible failures. It should instead have caught
`ConnectionResult.Failure`.
All authentication and initialization failures were not caught and
didn't trigger reconnections.
Co-authored-by: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
Introduce a `TxPublisher` actor to publish channel txs.
Move logic from watcher to this new actor.
Remove the `TxSigningKit` abstraction that was introduced a bit too early.
The `TxPublisher` will hold all the logic so we'll start by providing the full
commitments, and we'll extract more compact objects later.
We also now publish the commit-tx and its anchor-tx independently.
* preserve pg lock exception cause
* specialize connections by backend type
* added concurrency test on channels table
This test unveils a concurrency issue in the upsert logic of the local
channels db, with the following error being thrown when we update many
channels concurrently:
```
Canceled on identification as a pivot, during conflict out checking
```
* use pg upsert construct
This is the recommended pattern according to postgres doc
(https://www.postgresql.org/docs/current/plpgsql-control-structures.html#PLPGSQL-UPSERT-EXAMPLE):
> It is recommended that applications use INSERT with ON CONFLICT DO
UPDATE rather than actually using this pattern.
* reproduce and fix same issue in peers db
* proper formatting
* prefix sqlite-specific tests
* fix double init tests
Due to how the `TestDatabases` is instantiated, calling `dbs.dbName`
twice was a no-op.
* add jdbcurl files to gitignore
Electrum support was provided for mobile wallets, server nodes should always
run a bitcoind node as this provides more control (especially for utxo
management for anchor outputs channels).
Since wallets will use https://github.com/acinq/eclair-kmp instead of eclair,
we can now remove Electrum and API fee providers from eclair.
We also removed 3rd-party fee API providers that were only used on wallets.
We need to wait for the channel relayer to be registered to the event stream
before publishing an event, otherwise the channel relayer will never receive it.
We send it a first message and wait for its response to ensure it had time
to correctly initialize.
More symmetry between postgres and sqlite init.
* define SqliteDatabases and PostgresDatabase
Those classes implement traits `Databases`, `FileBackup` and
`ExclusiveLock`.
The goal is to have access to backend-specific attributes, particularly
in tests. It arguably makes the `Databases` cleaner and simpler, with a
nice symmetry between the `apply methods`.
* replace 5s lock timeout by NOLOCK
* use chaindir instead of datadir for jdbcurl file
It is more consistent with sqlite, and makes sense because we don't want
to mix up testnet and mainnet databases.
* add tests on locks and jdbc url check
Reverse the flow of yen's k-shortest path: go backwards like
we do in dijkstra.
Better tracking of already explored spur paths which improves
performance (especially tail latency).
An interesting side-effect of anchor outputs is that htlc txs can be merged
when they have the same lockTime (thanks to sighash flags).
We're not currently doing that, but our peers may do it, so we need to handle
it in the revoked commit tx case and correctly claim multiple outputs if
necessary.
Since anchor outputs, we not only deduce the commit tx fee from the funder's
main output but the cost of the anchors as well.
We rename the function that does that for more clarity.
This commit clarifies some parts of the code on which we regularly have
questions during pull requests.
We also add a test for an edge case in shutdown that was correctly handled,
but not properly tested, to ensure non-regression.
Re-work the `CommitPublished` types to work better with anchor outputs.
We previously stored the txs spending utxos that we could claim: this
doesn't make sense anymore if these txs may be RBF-ed, because the final
tx will be different from the initial one.
We instead track what `OutPoint`s we can claim, and the information
necessary to claim them. This way we can in the future let a different
actor finalize the txs that spend these outpoints (set the fees and sign).
We also add information on mutual close txs to immediately identify our
output and its amount: this makes auditing how much sats we'll get back
very easy from the API when we have many channels to watch.
This commit contains a DB migration of the channel data types, but in a
backwards-compatible way: we can still read from old data. The only
scenario impacted is channels that started force-closing before the migration.
They need special care to handle the fact that they had less data than
migrated channels, which is why we keep some legacy code around.
The goal is to ensure maximum safety when migrating data.
To achieve this, we strictly segregate each codec version (to make sure
that we don't accidentally mix codec versions), while still letting
tests access each unitary codecs (which using inner `private classes`
would have prevented). Relevant tests only need to be moved to the
appropriate package.
The package structure is now:
```
wire
|
`-- internal
| |
| `-- channel
| | |
| | `-- version0
| | | |
| | | `-- ChannelCodecs0
| | |
| | `-- version1
| | | |
| | | `-- ChannelCodecs1
| | |
| | `-- ChannelCodecs
| |
| `-- CommandCodecs
|
`-- others
```
Co-authored-by: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
* rename `Auditor` to `DbEventHandler`
Move it to the `db` package (it was in the `payments` package for
historical reasons but doesn't deal only with payment anymore).
Better typing for channel lifecycle event.
* add meta info to local_channels table
Here is the rationale for implementing channel metadata as additional
columns in the `local_channels` table of the `channels` db, as opposed
to a dedicated `channel_metadata` table of a `audit` db:
1) There is a migration to do (in the `local_channels` table, no less!),
but it's just a table migration, as opposed to a data migration, if we
had to populate a new table in a separate database.
2) We don't need to worry about creating a new metadata line when a new
channel is created (compared to doing add-or-update stuff). It's only
_updating_ optional columns in a best-effort manner.
3) We don't need to worry about inconsistencies between two tables
located in two separated databases (that's a big one).
4) We may want to use the metadata during operations, not just for
audit purposes. For example to close channels that have stayed unused
for a long time.
5) The audit db is an append-only log of events and shouldn't be used
for anything else. There is no `UPDATE` sql statement in
`*AuditDb.scala`. The `channel_metadata` would break that heuristic.
When a node returns a TemporaryChannelFailure, we should ignore this channel
when retrying. In some cases (such as channels from routing hints) this was
not correctly handled.
Fixes#1725
When we receive a trampoline payment asking us to relay with an
expiry that is already below the current chain height, we know that
this payment will fail when it reaches the next trampoline node.
Instead of waiting for the next trampoline node to reject it, we reject
it immediately.
Refactor the API handlers.
Split handlers and directives in several files to make them more composable.
Co-authored-by: Pierre-Marie Padiou <pm47@users.noreply.github.com>
Co-authored-by: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
We previously relied on `context.child` to check whether we already had a
relay handler for a given payment_hash.
Unfortunately this could return an actor that is currently stopping itself.
When that happens our relay command can end up in the dead letters and the
payment will not be relayed, nor be failed upstream.
We fix that by maintaining the list of current relay handlers in the
NodeRelayer and removing them from the list before stopping them.
This is similar to what's done in the MultiPartPaymentFSM.
It makes sense to allow node operators to configure the value they want
to use as a maximum threshold for anchor outputs commitment tx feerate.
This allows node operators to raise this value when mempools start getting
full in anticipation of a potential rise of the min-relay-fee.
This value can also be overridden for specific nodes.
We previously used a Set, which means you could theoretically have a feature
that is both activated as `optional` and `mandatory`.
We change that to be a Map `feature -> support`.
* add tests on funding mindepth
We verify that when using wumbo channels:
- if we are funder we keep our regular min_depth
- if we are fundee we use a greater min_depth
* use lenses to simplify tags handling
Co-authored-by: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
When using anchor outputs, the commitment feerate is kept low (<10 sat/byte).
When we need to force-close a channel, we must ensure the commit tx and htlc
txs confirm before a given deadline, so we need to increase their feerates.
This is currently done only once, at broadcast time.
We use CPFP for the commit tx and RBF for the htlc txs.
If publishing fails because we don't have enough utxos available, it will
be retried after the next block is confirmed.
Note that it's still not recommended to activate anchor outputs.
More work needs to be done on this fee bumping logic and utxos management.
If a channel closes when we've received an UpdateFailHtlc, signed it but
not yet received our peer's revocation, we need to fail the htlc upstream.
That specific scenario was not correctly handled, resulting in upstream
htlcs that were not failed which would force our upstream peer to close
the channel.
Since we almost always know which transactions will spend the utxos that we are watching, we can optimize the watcher to look for those instead of starting from scratch.
Our previous timeout was based on timestamps, mostly because blockCount
could be 0 on mobile using Electrum until a new block was received.
Now that we're diverging from the mobile wallet codebase, we can use block
heights instead which is more accurate.
See lightningnetwork/lightning-rfc#839
In some places of the codebase we relied on the fact that lightning transactions
had a single input. That was correct with the standard commitments format,
but will not be the case with anchor outputs: 2nd-stage txs (htlc-txs) and
3rd-stage txs (claim-htlc-txs) can be RBF-ed and have any number of inputs
and outputs.
With anchor outputs, the actual feerate for the commit tx can be decided
when broadcasting the tx by using CPFP on the anchor.
That means we don't need to constantly keep the channel feerate close to
what's happening on-chain. We just need a feerate that's good enough to get
the tx to propagate through the bitcoin network.
We set the upper threshold to 10 sat/byte, which is what lnd does as well.
We let the feerate be lower than that when possible, but do note that
depending on your configured `feerate-tolerance`, that means you can still
experience some force-close events because of feerate mismatch.
Fix anchor outputs closing fee requirements: when using anchor outputs,
the mutual close fee is allowed to be greater than the commit tx fee,
because we're targeting a specific confirmation window.
Fix fee mismatch without htlc: we allow disagreeing on fees while the channel
doesn't contain any htlc, because no funds can be at risk in that case.
But we used the latest signed fee when adding a new HTLC, whereas we must
also take into account the latest proposed (unsigned) fee.
Added additional method to Eclair like findRoute but allowing for 2 nodeIds.
Also added a new endpoint to the http Api "findroutebetweennodes" which
takes sourceNode and targetNode as params.
Fixes#1068
When our mempool is full, its min-relay-fee may be constantly changing.
To ensure our txs can be published, we need to check the min-relay-fee when
we fund the transaction, and raise it if necessary.
We are restoring the previous behavior of using the `sync_complete` field
to signal the end of a `channel_range_query` sync.
The first step is to correctly set that field, before we can read it and
interpret it to mark the end of sync.
See https://github.com/lightningnetwork/lightning-rfc/pull/826
When a peer is disconnected, the register will return a forward failure.
This can happen if the peer is connected when we start the payment FSM and
then disconnects before we send them an HTLC.
Obviously the route amount must be strictly positive.
We don't control htlcMinimumMsat (it is set by our peer) and for backwards
compatibility reasons we allow it to be 0 msat (even though it doesn't make
much sense), so we need to enrich our condition to detect empty channels.
We keep the GetRoutingState API available in the router as it's useful to
query network information locally (or between actors), but we stop sending
that data to remote nodes.
It's useful to separate channel state test methods in a dedicated trait
instead of always bundling it with `FixtureTestSuite`.
In particular, it was previously impossible to use both `BitcoindService`
and `StateTestsHelperMethods` because `BitcoindService` doesn't work with
fixtures (it leverages `beforeAll` and `afterAll` instead because launching
one bitcoind instance per-test would be too expensive and useless).
Front-end logs can produce a huge amount of logs, with significant
duplication. In order to reduce the log volume, we truncate `nodeId` and
`channelId` in the MDC to only keep the first 8 hexadecimal characters.
Also, override a few `toString` because some channel-queries-related
case classes produce huge strings.
