Our network graph has to be iterable in a deterministic order and
with the ability to iterate over a specific range. Thus,
historically, we've used a `BTreeMap` to do the iteration. This is
fine, except our map needs to also provide high performance lookups
in order to make route-finding fast. Sadly, `BTreeMap`s are quite
slow due to the branching penalty.
Here we replace the `BTreeMap`s in the scorer with a dummy wrapper.
In the next commit the internals thereof will be replaced with a
`HashMap`-based implementation.
As evidenced by the previous commit, it appears our A* router
does worse than a more naive approach. This isn't super surpsising,
as the A* heuristic calculation requires a map lookup, which is
relatively expensive.
```
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 169,991,943 ns/iter (+/- 30,838,048)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 122,144,987 ns/iter (+/- 61,708,911)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 48,546,068 ns/iter (+/- 10,379,642)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 32,898,557 ns/iter (+/- 14,157,641)
```
Adds two new payment `Method`s for identifying payments with custom
`min_final_cltv_expiry_delta` as payments with LDK or user payment
hashes.
The `min_final_cltv_expiry_delta` value is packed into the first 2
bytes of the expiry timestamp in the payment secret metadata.
All utility functions for invoice construction will now also accept an
Option<>al `min_final_cltv_expiry_delta` which is useful for things like
swaps etc. The `min_final_cltv_expiry_delta` will default back to
`MIN_FINAL_CLTV_EXPIRY_DELTA` if `None` is provided.
This matches the spec and helps avoid any confusion around
naming. We're also then consistent with `cltv_expiry` in an HTLC being
the actual block height value for the CLTV and not a delta.
For std builds, Invoice::created_at can be automatically set upon
construction using SystemTime::now() offset by SystemTime::UNIX_EPOCH.
Change InvoiceRequest::respond_with and Refund::respond_with to only
take a created_at parameter in no-std builds.
Add a builder for creating invoices for a refund and required fields.
Other settings are optional and duplicative settings will override
previous settings. Building produces a semantically valid `invoice`
message for the refund, which then can be signed with the key associated
with the provided signing pubkey.
Add a builder for creating invoices for an offer from a given request
and required fields. Other settings are optional and duplicative
settings will override previous settings. Building produces a
semantically valid `invoice` message for the offer, which then can be
signed with the key associated with the offer's signing pubkey.
Define an interface for BOLT 12 `invoice` messages. The underlying
format consists of the original bytes and the parsed contents.
The bytes are later needed for serialization. This is because it must
mirror all the `offer` and `invoice_request` TLV records, including
unknown ones, which aren't represented in the contents.
Invoices may be created for an Offer (from an InvoiceRequest) or for a
Refund. The primary difference is how the signing pubkey is given -- by
the writer of the offer or the reader of the refund.
Historically we've had various bugs in keeping the
`lowest_inbound_channel_fees` field in `NodeInfo` up-to-date as we
go. This leaves the A* routing less efficient as it can't prune
hops as aggressively.
In order to get accurate benchmarks, this commit updates the
minimum-inbound-fees field on load. This is not the most efficient
way of doing so, but suffices for fetching benchmarks and will be
removed in the coming commits.
Note that this is *slower* than the non-updating version in the
previous commit. While I haven't dug into this incredibly deeply,
the graph snapshot in use has min-fee info for only 9,618 of
20,818 nodes. Thus, it is my guess that with the graph snapshot
as-is the branch predictor is able to largely remove the A*
heuristic lookups, but with this change it is forced to wait for
A* heuristic map lookups to complete, causing a performance
regression.
```
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 182,980,059 ns/iter (+/- 32,662,047)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 151,170,457 ns/iter (+/- 75,351,011)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 58,187,277 ns/iter (+/- 11,606,440)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 41,210,193 ns/iter (+/- 18,103,320)
```
The previous copy was more than one and a half years old, the
lightning network has changed a lot since!
As of this commit, performance on my Xeon W-10885M with a
SK hynix Gold P31 storing a BTRFS volume is as follows:
```
test ln::channelmanager::bench::bench_sends ... bench: 5,896,492 ns/iter (+/- 512,421)
test routing::gossip::benches::read_network_graph ... bench: 1,645,740,604 ns/iter (+/- 47,611,514)
test routing::gossip::benches::write_network_graph ... bench: 234,870,775 ns/iter (+/- 8,301,775)
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 166,155,032 ns/iter (+/- 30,206,162)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 136,843,661 ns/iter (+/- 67,111,218)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 52,954,598 ns/iter (+/- 11,360,547)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 37,598,126 ns/iter (+/- 17,262,519)
test bench::bench_sends ... bench: 37,760,922 ns/iter (+/- 5,179,123)
test bench::bench_reading_full_graph_from_file ... bench: 25,615 ns/iter (+/- 1,149)
```
Secrets should not be exposed in-memory at the interface level as it
would be impossible the implement it against a hardware security
module/secure element.
When using bytes from an InvoiceRequest to constructing bytes for an
Invoice, any signature TLV records in the bytes must be excluded. Define
a wrapper for encoding such pre-serialized bytes in this manner. This
will allow the forthcoming InvoiceBuilder to construct bytes for an
Invoice properly.
Provide a helper for skipping signature TLV records from a TLV stream.
This prevents needing to duplicate the check for signature TLV records
when writing a TLV stream without signatures in an upcoming commit.
We have a number of debug assertions which are expected to never
fire when running in a single thread. This is just fine in tests,
and gives us good coverage of our lockorder requirements, but is
not-irregularly surprising to users, who may run with their own
debug assertions in test environments.
Instead, we gate these checks by the `cfg(test)` setting as well as
the `_test_utils` feature, ensuring they run in our own tests, but
not downstream tests.
The lightning protocol uses u16s for lengths in most cases. As our
serialization framework primarily targets that, we must as well.
However, because we may serialize objects that have more than 65K
entries, we want to be able to store larger values. Thus, we define
a variable length integer here which is backwards-compatible but
treats 0xffff as "read eight more bytes".
This doesn't address any specific known issue, but feels like good
practice just in case.
