Codecs shouldn't throw exceptions and should rather return an explicit
`Attempt.Failure` when decoding invalid input. We fix a few codecs that
create public keys to do avoid throwing.
We also wrap onion TLV streams into a `Try`: even though in theory none
of our codecs should throw, since we don't have good enough fuzzing in
place to guarantee that it, it's safer to catch exceptions in case a
codec isn't correctly implemented.
For single-funded channels where we are the funder, we previously only
waited for 1 confirmation before marking the funding transaction as
confirmed. This is fine because we trust ourselves to not double-spend
our own channels.
However, in #2969, we started assigning the `short_channel_id` when we
consider the funding transaction confirmed, and not after 6 blocks. We
thus now risk creating a `short_channel_id` after only 1 confirmation,
which is too early in case a reorg happens. This wouldn't cause loss
of funds because we would create invalid `channel_update`s and the
channel would likely not be usable.
Update Bitcoin Core to v28.x. This releases contains the following
interesting features:
- automatic utxo unlocking on wallet conflicts
- support for testnet4
- 1-parent 1-child package relay (channel force-close)
- `max_tx_weight` parameter to `fundrawtransaction`
We add support for the latest channel closing protocol described in
https://github.com/lightning/bolts/pull/1205. This is a prerequisite
for taproot channels.
We introduce a new `NEGOTIATING_SIMPLE` state where we exchange the
`closing_complete` and `closing_sig` messages, and allow RBF-ing previous
transactions and updating our closing script.
We stay in that state until one of the transactions confirms, or a force
close is detected. This is important to ensure we're able to correctly
reconnect and negotiate RBF candidates.
We keep this separate from the previous `NEGOTIATING` state to make it
easier to remove support for the older mutual close protocols once we're
confident the network has been upgraded.
Before relaying a trampoline payment, we check if the next node is one
of our direct peers. If that's the case, we check their features to see
if they support waking up through notifications, in which case we go
through the wake-up flow.
Note that we don't need this for channel relay, because:
- if it is a Bolt 12 payment, the recipient will use a `wallet_node_id`
to let us know that they support wake-up notifications
- if it is a Bolt 11 payment, we use a custom `short_channel_id` for
phoenix that also lets us know that they support wake-up notifications
We only store information about our peers when we succeed in making an
outgoing connection to them. The only information we stored was the
address that we used when connecting. We now also store the features
supported by our peer when we last connected to them.
Once we have a channel with a peer that connected to us, we store their
details in our DB. We don't store the address they're connecting from,
because we don't know if we will be able to connect to them using this
address, but we store their features.
When a new block is found, we want to check its confirmed transactions
to potentially trigger watches. This is especially important when a
channel is spent and we haven't seen the spending transaction in our
mempool before receiving it in a block. This is already supposed to be
handled through the ZMQ `rawtx` topic, where bitcoind should send us
every transaction it receives (either in the mempool or in a block).
But when using remote `bitcoind` instances, ZMQ seems to sometimes be
unreliable and silently drop some events. That's why we add another
mechanism for extra safety, where whenever a new block is found, we
fetch the last `N` blocks and re-process their transactions. We keep
a cache of the processed blocks to ensure that we don't needlessly
re-process them multiple times.
We were previously using a composite index on both the `channel_id` and
the `commitment_number` in the `htlc_infos` table. This table is used to
store historical HTLC information to be able to publish penalty txs when
a malicious peer publishes a revoked commitment. This table is usually
the largest DB table on nodes that relay a lot of payments, because it
grows with the number of HTLCs received and sent (and only shrinks when
channels are closed or spliced).
Using a composite index makes sense, but leads to increased memory usage
compared to separate indices, thus reducing performance because this is
a table on which we write a lot, but almost never read (only when we
detect a revoked force-close). The read performance doesn't seem to be
negatively impacted anyway when splitting the indices, and the memory
usage is greatly improved.
The migration may take some time depending on the size of the table,
but we cannot get around this.
Thanks to @DerEwige for the performance investigation that lead to
this change (see #2932 for more details).
When using dual-funding, both peers may contribute to the funding amount
and it usually cannot be known ahead of time how much the remote peer
will contribute, which usually leads to underestimating the channel
capacity and thus using a lower `max_htlc_value_in_flight` than what
should be used.
However, when we use liquidity ads, we will:
- contribute to the funding transaction if we're not the opener
- cancel the funding attempt if we're the opener and our peers doesn't
contribute at least the amount we requested
So in that case, we can use a better estimate of the channel capacity
when computing our `max_htlc_value_in_flight`.
This PR:
- uses _outpoints_ instead of _txids_ to track on-chain utxos, the later being insufficient when transactions have more than one output that belongs to us
- prints a detailed diff between two balance checks, by utxos and channels.
A splice tx could involve more than one parent channel. The Router must track the set of channels spent by a given spend tx until either:
1) If it is a splice, then matching channel announcements are received for each parent and the channels are updated in the routing graph.
2) If the spend tx is deeply buried without receiving matching channel announcements for a parent channel, then it can be removed from the routing graph.
If a splice tx spends more than one parent channel between the same nodes, then there's no way to know which new channel announcement corresponds to which parent channel. We simply update the first one found.
By default we sync with every peer when reconnecting, which can be a lot. We now only sync on reconnection with our top peers (by capacity of our shared channels).
We now support splicing on public channels: once the splice transaction
is confirmed and locked on both sides, nodes will exchange announcement
signatures that allows them to create a `channel_announcement` that they
then broadcast to the network.
This requires reworking the data model to include the announcement and
the real `short_channel_id` in each commitment, which lets us cleanly
distinguish real `short_channel_id`s from aliases (which are set at the
channel level regardless of the current commitments).
The flow now becomes:
- when the funding transaction of a commitment confirms, we set the
corresponding real `short_channel_id` in that commitment
- if the channel is public and we've received `channel_ready` or
`splice_locked`, we send our `announcement_signatures`
- if we receive `announcement_signatures` for a commitment for which
the funding transaction is unconfirmed, we stash it and replay it
when the transaction confirms
- if we receive `announcement_signatures` for a confirmed commitment,
and we don't have a more recently announced commitment, we generate
a `channel_announcement`, store it with the commitment and update
our router data
When creating a `channel_update` for a public channel, we always use
the `short_channel_id` that matches the latest announcement we created.
This is very important to guarantee that nodes receiving our updates
will not discard them because they cannot match it to a channel.
For private channels, we stop allowing usage of the `short_channel_id`
for routing: `scid_alias` MUST be used, which ensures that the channel
utxo isn't revealed.
Note that when migrating to taproot channels, `splice_locked` will be
used to transmit nonces for the announcement signatures, which will be
compatible with the existing flow (and similarly, `channel_ready` will
be used for the initial funding transaction). They are retransmitted
on reconnection to ensure that the announcements can be generated.
Now that we wait for at least 6 confirmations before considering a
channel confirmed, we can simplify our channel announcement logic.
Whenever a channel reaches the confirmed state, it can be announced
to the network (if nodes wish to announce it). We thus don't need
the "deeply buried" state and the "temporary" scid anymore.
The logic is much simpler to follow: when the channel confirms, we
internally update the real scid to match the confirmed funding tx
and send our `announcement_signatures`. When we receive our peer's
`announcement_signatures`, we stash them if the funding tx doesn't
have enough confirmations yet, otherwise we announce the channel and
create a new `channel_update` that uses the real scid.
Whenever we create a `channel_update`, we simply look at whether the
channel is announced or not to choose which scid to use.
This will make it much simpler to announce splice transactions, which
don't need a "deeply buried" state either and will instead simply rely
on whether the splice transaction is confirmed or not to generate
`announcement_signatures`.
* Allow funding without locks for liquidity griefing
When using liquidity ads, seller nodes may be vulnerable to liquidity
griefing attacks. They may disable utxo locking to protect against such
attacks, but the trade-off is that honest peers may also be affected by
having their funding transaction double-spent.
We thus expose a configuration flags to let node operators decide which
trade-off they choose, depending on how likely they think someone will
target them vs the UX they want to provide to their customers.
* Abort incoming channel after timeout
If a remote node starts opening a channel to us and then becomes
unresponsive, we abort the channel. This is particularly useful
when they're purchasing liquidity and we've locked utxos.
We previously handled `recommended_feerates` in our `whenUnhandled`
handler, but that doesn't work since we have a catch-all handler in
the `CONNECTED` state for all known lightning messages. We thus move
this handler into the `CONNECTED` state to avoid warnings in the logs.
* Watch explicit event for channel actor termination
We use an explicit event in the `Peer` when watching for child channel
actors being terminated. This should get rid of the warnings in the logs
about the `Terminated` event not being handled, or at least provide us
more hints of which actors aren't properly being tracked.
* Fix disconnect response in channel actors
We weren't watching for the right event, which creates a lot of log
lines saying that the `Disconnecting` event is unhandled.
* Ignore closing negotiation if closing or closed
If we receive an outdated closing message, we ignore it without creating
a warning in the logs.
* Ignore HTLC settlement commands while disconnected
We ignore HTLC settlement commands while we're disconnected, they will
be retried once the channel has been reestablished. We also ignore
commands asking us to sign the latest state or update fees.
* Get ready to store partial signatures
We currently store our peer's signature for our remote commit tx, so we can publish it if needed.
If we upgrade funding tx to use musig2 instead of multisig 2-of-2 we will need to store a partial signature instead.
For this, we add specific types for standard signature (64 bytes) and musig2 partial signatures + nonce.
We already use a minimum depth of 6 before announcing channels to
protect against reorgs. However, we allowed using the channel for
payments after only 3 confirmations (for small channels). A reorg
of 3 blocks that invalidates the funding transaction would allow
our peer to potentially steal funds. It's more consistent to use
the same depth for announcing the channel and actually using it.
Note that for wumbo channels, we already scaled the number of
confirmations based on the size of the channel.
For closing transaction, we don't need the same reorg safety, since
we keep watching the funding output for any transaction that spends
it, and concurrently spend any commitment transaction that we detect.
We thus keep a minimum depth of 3 for closing transactions.
We also update our confirmation scaling factor post-halving.
We were still using values from before the halving. We update those
values and change the scaling factor to a reasonable scaling. This
protects channels against attackers with significant mining power.
As pointed out in https://github.com/lightning/bolts/pull/1214, when
reconnecting a partially signed `interactive-tx` session, we should
set `next_commitment_number` to the current commitment number if we
haven't received our peer's `commit_sig`, which tells them they need
to retransmit it.
That's not what we're currently doing: we're currently setting this
value to the next commitment number, regardless of whether or not
we have received our peer's `commit_sig`. And we always retransmit
our `commit_sig` if our peer is setting `next_funding_txid`, even
if they have already received it.
More importantly, if our peer behaves correctly and sends us the
current commitment number, we will think that they're late and will
halt, waiting for them to send `error`. This commit fixes that by
allowing our peers to use the current commitment number when they
set `next_funding_txid`.
Note that this doesn't yet make us spec-compliant, but in order to
guarantee backwards-compatibility, we must first deploy that change
before we can start removing spurious `commit_sig` retransmissions.
* Refactor tx signing (no functional changes)
* Upgrade input info class to allow spending from taproot transactions
Our InputInfo class contains a tx output and the matching redeem script, which is enough to spend segwit v0 transactions.
For taproot transactions, instead of a redeem script, we need a script tree instead, and the appropriate internal pubkey.
* Use specific segwit and taproot input info types
We now use specific subtypes for segwit inputs (which include a redeem script) and taproot inputs (which include a script tree and an internal key).
Older codecs have been modified to always return a SegwitInput.
v4 codec is modified and uses an empty redeem script as a marker to specify that a script tree is being used, which makes it compatible with the current v4 codec.
Current (v4) codecs only handle segwit inputs. Support for taproot inputs will be added to v5 codecs.
---------
Co-authored-by: Pierre-Marie Padiou <pm47@users.noreply.github.com>
* Update kamon and kanela-agent
Previous version was not compatible with JDK21, see https://github.com/kamon-io/kanela/issues/150.
* Add and configure maven wrapper to use maven 3.9.9
This will make it easier to control which version of maven is used to build eclair, which in turns makes deterministic builds easier, as well as using recent compiler options (to target newer JDKs for example).
For example, even recent versions of Github runner images use an old version of maven and there is no easy way to upgrade.
* Update Dockerfile
We now use multiarch (amd64/arm64) base images.
* Run CI tests with JDK21
* Update doc to recommend Adoptium OpenJDK21
* Target Java 21
Eclair now targets Java 21 and will require a compatible Java Runtime Environment.
It will no longer work on JRE 11 and JRE 17.
Fixes Issue #2437
When an external channel is spent, add it to the `spentChannels` list instead of immediately removing it from the graph.
RBF attempts can produce multiple spending txs in the mempool for the same channel.
The `spendChannels` list maps the txid of the spending tx to the scid of the spent channel.
When a channel announcement is validated with a funding tx on the `spentChannels` list, consider the new channel a splice of the corresponding spent channel.
A splice updates the graph edges to preserve balance estimate information in the graph.
If a spending tx from the `spentChannels` list is deeply buried before appearing in a valid channel announcement, remove the corresponding spent channel from the graph.
The integration test demonstrates that local channels update their capacity, but we can not test the remote node (carol) because the ChannelAnnouncements are ignored because it has a duplicate scid. After PR #2941 we can fix this test.
---------
Co-authored-by: t-bast <bastien@acinq.fr>
We add more tests around disconnection in the middle of signing an RBF
attempt, and verify more details of the `channel_reestablish` message
sent on reconnection.
When using RBF for a dual-funded channel or a splice, we set multiple
`WatchFundingConfirmed` for conflicting transactions. When one of those
transactions confirms, the others will never confirm: it is wasteful to
keep watching for their confirmation.
The watcher doesn't have enough information on its own to efficiently
detect that some watches are double-spent: we instead rely on the
consumer of the watch to tell the watcher to stop watching the RBF
attempts.
Fixes#2954
While we ignore the actual failure and always return a temporary failure
upstream, it's useful to decrypt the recipient failure to log it. When
we add support for trampoline errors, we will need the decrypted failure
to be able to re-wrap it with trampoline onion secrets.
Otherwise it gets confusing quite quickly, because there are a lot of
onion-related secrets (sphinx shared secret, path key, invoice payment
secret...).
* Return unwrapped decryption failure onion
When we fail to decrypt an onion failure packet, we should return the
result of the unwrapping process. When using trampoline, this will let
us properly re-encrypt the failure and relay it upstream to the previous
trampoline node, until it reaches the sender.
* Refactor HTLC failure creation
We refactor the shared secret extraction to a dedicated function.
* Refactor HTLC failure reason
We previously used an `Either[ByteVector, FailureMessage]` to encode:
- a downstream error that we couldn't decrypt and must re-wrap (left)
- a local error that we must encrypt (right)
This won't be sufficient for trampoline, because we will need to handle
the following cases:
- a downstream error that we couldn't decrypt and must re-wrap
- a local error for the node who created the *outer* onion (which we
encrypt with the sphinx shared secret of the outer onion)
- a local error for the node who created the *trampoline* onion (which
we encrypt with the sphinx shared secret of the trampoline onion and
then with the shared secret of the outer onion)
We thus introduce a trait, which currently only contains the first two
cases. We will extend this trait when adding support for trampoline
failures. This is a pure refactoring without any behavior changes so
far, which will simplify the future trampoline changes.
* Include trampoline onion in final trampoline payloads
When we receive a (non-blinded) payment that uses trampoline, we keep
the trampoline onion to be able to distinguish this payment from a
non-trampoline payment.
When a channel force-closes, we publish our commit tx and HTLC txs.
HTLC transactions conflict with our peer's transactions that also spend
the HTLC outputs. If our peer is able to get their transaction confirmed
before ours, we should stop retrying to publish our HTLC transaction as
that will never succeed.
Since we didn't check the output status, we kept retrying until the
channel was closed (which requires waiting for the `to_self_delay`).
The retries always fail at funding time: `bitcoind` returns an error
saying that the UTXO cannot be found (which is expected because it has
already been spent by our peer). This creates a lot of unnecessary
retries and a lot of noise in the logs.
This scenario usually happens when our peer didn't send the preimage
before force-closing the channel, but reveals it on-chain before the
HTLC timeout: when that happens we kept retrying to publish our HTLC
timeout transaction, which cannot succeed.
We now check the output status in our publishing preconditions, and
immediately abort if the output has already been spent by a confirmed
transaction.
* Fix offer description documentation
And remove the `currency` fields as we have no short-term plans to
support currency conversion in `eclair`.
* Relax `payment_constraints` requirement in final blinded payload
We don't always need to include a `payment_constraints` field for
ourselves: it's fine to accept payment that don't contain one as
long as we created the `encrypted_recipient_data`, which we can
verify using the `path_id`. We were too restrictive for no good
reason.
* Allow omitting `total_amount` in blinded payments
If the `total_amount` field isn't provided, we can safely default to
using the `amount`, which saves space in the onion. Note that we keep
always encoding it in the outgoing payments we send, we're simply more
permissive when receiving payments.
* Refactor `decryptEncryptedRecipientData`
We extract a helper method for decrypting encrypted recipient data
which will be used when decrypting trampoline blinded paths.
* Use relay methods in `PaymentOnion.IntermediatePayload.NodeRelay`
In order to support blinded trampoline payments, we won't have access to
a direct `amount_to_forward` field, but will use a `payment_relay` TLV
instead, which only allows calculating the outgoing amount from the
incoming amount (same thing for the expiry).
We refactor this to simplify the diff when introducing blinded trampoline
payments.
We refactor trampoline-to-legacy payments to use a dedicated class, like
what we do for trampoline-to-blinded-paths payments. This allows us to
supports two encodings for those payments:
- one where the trampoline onion contains a dummy payload for the
recipient that must be ignored (current Phoenix wallets), which
wastes space in the onion for legacy reasons
- one where we don't include a dummy payload for the recipient, which
is more efficient and similar to trampoline-to-blinded-paths
When we're the introduction node of a trampoline blinded path, we
previously only allowed our custom `wallet_node_id` format when a
`short_channel_id` was not included. But we actually can allow plain
`node_id`s as well, as we only need to know the public key.
We've rejected some payments in the past months because they included an
`outgoing_node_id` that didn't use the wallet format: by removing that
limitation we ensure that those payments will be correctly relayed in
the future.
We previously supported sending arbitrary trampoline payments in eclair,
which added a bunch of complexity to many of our payment components.
This was necessary because the first version of Phoenix was based on the
eclair codebase. But Phoenix isn't based on eclair anymore and is now
using https://github.com/acinq/lightning-kmp, which is the library that
mobile wallets should use.
Eclair is only meant to be used for server nodes that relay payments and
have access to the full network graph, so it doesn't make sense ever to
send trampoline payments (full source-routing will always be better). We
thus refactor our trampoline client code to only be used for tests (to
ensure that trampoline relay and trampoline receive are implemented
correctly). We isolate the test payment lifecycle in a dedicated actor
(`TrampolinePaymentLifecycle`) which simplifies other payment components
(such as `PaymentInitiator`).
This refactoring will make it easier to support the official version of
trampoline while maintaining backwards-compatibility for older Phoenix
wallets that use the non-official trampoline version.
Bitcoin Core 27.2 is out with some bug fixes. None of the bugs fixed
seem to impact us, but there's always a possibility that a security
issue was also fixed, so it's better to upgrade!
