When either the amount or the `max_total_cltv_expiry_delta` are
set to max-value, `set_max_path_length` can trigger overflows in
`build_onion_payloads_callback`, leading to debug-panics.
With the `Confirm` interface, transaction confirmations can come
in at any time, so asserting that a confirmation is more recent
than the last time we broadcasted a transaction can lead to
spurious assertion failures.
Lack of bindings support was because the method used to return a slice
of tuples, it seems. Now that it returns &[BlindedPaymentPath], bindings
should be possible given that they can be generated for
Bolt12Invoice::message_paths.
Rust 1.84.0 was recently released along with some new clippy lints, one
of which is `unnecessary_map_or`. Unfortunately this lint suggests using
`Option::is_some_and` as a fix, but this is only available in Rust
version >= 1.70, while we still have an MSRV of 1.63. So we silence that
lint for now.
We'd still like our lint CI to use stable Rust so that we can benefit from
new lint checks which may be helpful and don't require an MSRV bump, but
sometimes new lints (like in this case) do.
See:
https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_map_orhttps://doc.rust-lang.org/std/option/enum.Option.html#method.is_some_and
Pending v2 channels will need to be broken up into separate phases for
constructing and signing the funding transaction. To avoid increasing
the number of phases, combine the InboundV2Channel and OutboundV2Channel
types so that the can be used in one phase. Whether the channel is
inbound or outbound can be inferred from the ChannelContext.
We recently introduced `TRACE`-level logging for event handling.
However, in onion messenger we'd now log (twice, actually) every time
`process_events_async` is called, which is very very spammy. Here we fix
this by short-cutting to only proceed when we actualy have any event
futures to poll.
Previously, we would fail parsing `Offer`s if the HRP didn't match our
expected (lowercase) HRP. Here, we relax this check in accordance with
the spec to also allow all-uppercase HRPs.
Utilizing the results of probes sent once a minute to a random node
in the network for a random amount (within a reasonable range), we
were able to analyze the accuracy of our resulting success
probability estimation with various PDFs across the historical and
live-bounds models.
For each candidate PDF (as well as other parameters, including the
histogram bucket weight), we used the
`min_zero_implies_no_successes` fudge factor in
`success_probability` as well as a total probability multiple fudge
factor to get both the historical success model and the a priori
model to be neither too optimistic nor too pessimistic (as measured
by the relative log-loss between succeeding and failing hops in our
sample data).
We then compared the resulting log-loss for the historical success
model and selected the candidate PDF with the lowest log-loss,
skipping a few candidates with similar resulting log-loss but with
more extreme constants (such as a power of 11 with a higher
`min_zero_implies_no_successes` penalty).
Somewhat surprisingly (to me at least), the (fairly strongly)
preferred model was one where the bucket weights in the historical
histograms are exponentiated. In the current design, the weights
are effectively squared as we multiply the minimum- and maximum-
histogram buckets together before adding the weight*probabilities
together.
Here we multiply the weights yet again before addition. While the
simulation runs seemed to prefer a slightly stronger weight than
the 4th power we do here, the difference wasn't substantial
(log-loss 0.5058 to 0.4941), so we do the simpler single extra
multiply here.
Note that if we did this naively we'd run out of bits in our
arithmetic operations - we have 16-bit buckets, which when raised
to the 4th can fully fill a 64-bit int. Additionally, when looking
at the 0th min-bucket we occasionally add up to 32 weights together
before multiplying by the probability, requiring an additional five
bits.
Instead, we move to using floats during our histogram walks, which
further avoids some float -> int conversions because it allows for
retaining the floats we're already using to calculate probability.
Across the last handful of commits, the increased pessimism more
than makes up for the increased runtime complexity, leading to a
40-45% pathfinding speedup on a Xeon Silver 4116 and a 25-45%
speedup on a Xeon E5-2687W v3.
Thanks to @twood22 for being a sounding board and helping analyze
the resulting PDF.
In the next commit we'll want to return floats or ints from
`success_probability` depending on the callsite, so instead of
duplicating the calculation logic, here we split the linear (which
always uses int math) and nonlinear (which always uses float math)
into separate methods, allowing us to write trivial
`success_probability` wrappers that return the desired type.
Utilizing the results of probes sent once a minute to a random node
in the network for a random amount (within a reasonable range), we
were able to analyze the accuracy of our resulting success
probability estimation with various PDFs across the historical and
live-bounds models.
For each candidate PDF (as well as other parameters, to be tuned in
the coming commits), we used the `min_zero_implies_no_successes`
fudge factor in `success_probability` as well as a total
probability multiple fudge factor to get both the historical
success model and the a priori model to be neither too optimistic
nor too pessimistic (as measured by the relative log-loss between
succeeding and failing hops in our sample data).
Across the simulation runs, for a given PDF and other parameters,
we nearly always did better with a shorter half-life (even as short
as 1ms, i.e. only learning per-probe rather than across probes).
While this likely makes sense for nodes which do live probing, not
all nodes do, and thus we should avoid over-biasing on the dataset
we have.
While it may make sense to only learn per-payment and not across
payments, I can't fully rationalize this result and thus want to
avoid over-tuning, so here we reduce the half-life from 6 hours to
30 minutes.
If the liquidity penalty multipliers in the scoring config are both
0 (as is now the default), the corresponding liquiditiy penalties
will be 0. Thus, we should avoid doing the work to calculate them
if we're ultimately just gonna get a value of zero anyway, which we
do here.
Utilizing the results of probes sent once a minute to a random node
in the network for a random amount (within a reasonable range), we
were able to analyze the accuracy of our resulting success
probability estimation with various PDFs across the historical and
live-bounds models.
For each candidate PDF (as well as other parameters, to be tuned in
the coming commits), we used the `min_zero_implies_no_successes`
fudge factor in `success_probability` as well as a total
probability multiple fudge factor to get both the historical
success model and the a priori model to be neither too optimistic
nor too pessimistic (as measured by the relative log-loss between
succeeding and failing hops in our sample data).
We then compared the resulting log-loss for the historical success
model and selected the candidate PDF with the lowest log-loss,
skipping a few candidates with similar resulting log-loss but with
more extreme constants (such as a power of 11 with a higher
`min_zero_implies_no_successes` penalty).
In every case, the historical model performed substantially better
than the live-bounds model, so here we simply disable the
live-bounds model by default and use only the historical model.
Further, we use the calculated total probability multiple fudge
factor (0.7886892844179266) to choose the ratio between the
historical model and the per-hop penalty (as multiplying each hop's
probability by 78% is equivalent to adding a per-hop penalty of
log10(0.78) of our probabilistic penalty).
We take this opportunity to bump the penalties up a bit as well, as
anecdotally LDK users are willing to pay more than they do today to
get more successful paths.
Fixes#3040
Utilizing the results of probes sent once a minute to a random node
in the network for a random amount (within a reasonable range), we
were able to analyze the accuracy of our resulting success
probability estimation with various PDFs.
For each candidate PDF (as well as other parameters, to be tuned in
the coming commits), we used the `min_zero_implies_no_successes`
fudge factor in `success_probability` as well as a total
probability multiple fudge factor to get both the historical
success model and the a priori model to be neither too optimistic
nor too pessimistic (as measured by the relative log-loss between
succeeding and failing hops in our sample data).
We then compared the resulting log-loss for the historical success
model and selected the candidate PDF with the lowest log-loss,
skipping a few candidates with similar resulting log-loss but with
more extreme constants (such as a power of 11 with a higher
`min_zero_implies_no_successes` penalty).
This resulted in a PDF of `128 * (1/256 + 9*(x - 0.5)^8)` with a
`min_zero_implies_no_successes` probability multiplier of 64/78.
Thanks to @twood22 for being a sounding board and helping analyze
the resulting PDF.
Previously, we wouldn't set the field as we aren't yet making use of it.
Here, we start setting the field. To this end, we make `best_block` an
`RwLock<Option<BestBlock>>` rather than `Option<RwLock<BestBlock>>`.
When a peer misbehaves/sends bogus data we reply with an error message
and insert it to the ignored list.
Here, we avoid having this list grow unboundedly over time by removing
peers again once they disconnect, allowing them a second chance upon
reconnection.
We should update the return types on the signing methods here as
well, but we should at least start by documenting which methods are
async and which are not.
Once we complete async support for `get_per_commitment_point`, we
can change the return types as most things in the channel signing
traits will be finalized.
Prior to bcaba29f92, the
`chanmon_consistency` fuzzer checked that payments sent either
succeeded or failed as expected by looking at the `APIError` which
we received as a result of calling the send method.
bcaba29f92 removed the legacy send
method during fuzzing so attempted to replicate the old logic by
checking for events which contained the legacy `APIError` value.
While this was plenty thorough, it was somewhat brittle in that it
made expectations about the event state of a `ChannelManager` which
turned out to not be true.
Instead, here, we validate the send correctness by examining the
`RecentPaymentDetails` list from a `ChannelManager` immediately
after sending.
bcaba29f92 started returning
pre-built `Route`s from the router in the `chanmon_consistency`
fuzzer. In doing so, it didn't properly fill in the `route_parms`
field which is expected to match the requested parameters. This
causes a debug assertion when sending.
Here we fix this by setting the correct `route_params`.
bcaba29f92 introduced a deadlock in
the `chanmon_consistency` fuzzer by holding a lock on the route
expectations before sending a payment, which ultimately tries to
lock the route expectations. Here we fix this deadlock.
