Without this, we have hardly any enforcement. This is why the schema
mistake fixed in the previous patches weren't spotted immediately.
The hard work was done by:
```
$ for f in lightning-*.json; do grep -v '^ "additionalProperties": false,' $f | bagto $f; done
$ for f in lightning-*.json; do sed 's/"properties": {/"additionalProperties": false, "properties": {/' $f | bagto $f; done
$ make fmt-schemas
```
Then checking where 'additionalProperties: true' had been turned to
false (we deliberately use it in some places where there are if
statements in the schema, or occasionally where there can be arbitrary
fields).
[Including doc/rpc-schema-draft.json update by Shahana]
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows the next patch (which makes the schemas stricter) to not
break our tests.
We add some missing fields (including dev fields, but they're empty and hidden),
and add a few minor clarifications and a spelling fix. Most of these are new
schemas for this release, so no mention in Changelog.
Here is the difference in the man pages:
--- doc/lightning-askrene-inform-channel.7.md.old 2024-10-29 17:33:07.714521584 +1030
+++ doc/lightning-askrene-inform-channel.7.md 2024-10-29 17:42:37.434280109 +1030
@@ -24,6 +24,8 @@
- **short\_channel\_id\_dir** (short\_channel\_id\_dir): The short channel id and direction
+- **layer** (string): The name of the layer to apply this change to.
+- **timestamp** (u64): The UNIX timestamp when this constraint was created.
- **maximum\_msat** (msat, optional): The maximum value which this channel could pass.
--- doc/lightning-askrene-listlayers.7.md.old 2024-10-29 17:33:07.716521571 +1030
+++ doc/lightning-askrene-listlayers.7.md 2024-10-29 17:42:37.424280316 +1030
@@ -29,13 +29,16 @@
- **channel\_updates** (array of objects):
+ - **short\_channel\_id\_dir** (short\_channel\_id\_dir): The short channel id and direction this update applies to.
+ - **enabled** (boolean, optional): True if this can be used, false otherwise.
- **htlc\_minimum\_msat** (msat, optional): The minimum value allowed in this direction.
- **htlc\_maximum\_msat** (msat, optional): The maximum value allowed in this direction.
- **fee\_base\_msat** (msat, optional): The base fee to apply to use the channel in this direction.
- **fee\_proportional\_millionths** (u32, optional): The proportional fee (in parts per million) to apply to use the channel in this direction.
- - **delay** (u16, optional): The CLTV delay required for this direction.
+ - **cltv\_expiry\_delta** (u16, optional): The CLTV delay required for this direction.
- **constraints** (array of objects):
- **short\_channel\_id\_dir** (short\_channel\_id\_dir): The short channel id and direction
+ - **timestamp** (u64, optional): The UNIX timestamp when this constraint was created.
- **maximum\_msat** (msat, optional): The maximum value which this channel could pass.
--- doc/lightning-askrene-listreservations.7.md.old 2024-10-29 17:33:07.719521550 +1030
+++ doc/lightning-askrene-listreservations.7.md 2024-10-29 17:42:37.428280233 +1030
@@ -16,7 +16,7 @@
-On success, an object containing **layers** is returned. It is an array of objects, where each object contains:
+On success, an object containing **reservations** is returned. It is an array of objects, where each object contains:
--- doc/lightning-autoclean-status.7.md.old 2024-10-29 17:33:07.732521462 +1030
+++ doc/lightning-autoclean-status.7.md 2024-10-29 17:42:37.441279965 +1030
@@ -9,7 +9,7 @@
-The **autoclean-status** RPC command tells you about the status of the autclean plugin, optionally for only one subsystem.
+The **autoclean-status** RPC command tells you about the status of the autoclean plugin, optionally for only one subsystem.
--- doc/lightning-renepay.7.md.old 2024-10-29 17:33:07.927520140 +1030
+++ doc/lightning-renepay.7.md 2024-10-29 17:42:37.996268504 +1030
@@ -58,6 +58,9 @@
- **status** (string) (one of "complete", "pending", "failed"): Status of payment.
+- **bolt11** (string, optional): The bolt11 invoice paid. *(added v23.08)*
+- **bolt12** (string, optional): The bolt12 invoice paid. *(added v23.08)*
+- **groupid** (u64, optional): The groupid used for these payment parts (as can be seen in listsendpays) *(added v23.08)*
- **destination** (pubkey, optional): The final destination of the payment.
--- doc/lightning-sendonion.7.md.old 2024-10-29 17:33:07.937520073 +1030
+++ doc/lightning-sendonion.7.md 2024-10-29 17:42:37.957269309 +1030
@@ -22,7 +22,7 @@
-- **first\_hop** (object): Instructs Core Lightning which peer to send the onion to. It is a JSON dictionary that corresponds to the first element of the route array returned by *getroute*.:
+- **first\_hop** (object): Instructs Core Lightning which peer to send the onion to. It is a JSON dictionary that corresponds to the first element of the route array returned by *getroute* (so fields not mentioned here are ignored).:
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `renepay` return fields documented in schema (`bolt11`, `bolt12` and `groupid`)
This is like `sendonion` but unwraps the onion as the first hop,
avoiding nasty special cases for blinded paths which start with this
node, and also self-pay.
Tests split into multiple ones after Christian's review.
Changelog-Added: JSON-RPC: `injectpaymentonion` for initiating an HTLC like a peer would do.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Don't assume we have an outgoing HTLC at this level.
Note that previously we didn't save the failed onion unless it was
unparsable: we keep that both for space savings and because our
`waitsendpay` logic assumes that when it fetches from the db if
there's a failonion it was unparsable!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This message is supposed to include the msat amount received. But this is
obviously per-HTLC, and we hacked it to use the value for the first one.
And we add logging whenever we fail an HTLC set, since we removed logging
by not calling failmsg_incorrect_or_unknown() (which, now, no longer needs
to log).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Removes the `COMPAT_V070` functionality for `listfowards`.
Changelog-Changed: The `listforwards` command will now return a value
of 0 for `received_time` for very old forward attempts.
Changelog-Fixed: The documentation version was calculated as `pre-v24.08` for point releases like v24.08.1` also because `CLN_NEXT_VERSION` has not been included in the point release branches. Updating the script to build documentation on new tags and change the version to `pre-cln-next-version` for non-tagged commits.
Changelog-Added: JSON-RPC: `decode` now used modern BOLT 4 language for blinded paths, `first_path_key`.
Changelog-Deprecated: JSON-RPC: `decode` `blinding` in blinded path: use `first_path_key`.
Changelog-Added: Plugins: `onion_message_recv` and `onion_message_recv_secret` hooks now used modern BOLT 4 language for blinded paths, `first_path_key`.
Changelog-Deprecated: JSON-RPC: `onion_message_recv` and `onion_message_recv_secret` hooks `blinding` in blinded path: use `first_path_key`.
No code changes, just catching up with the BOLT changes which rework our
blinded path terminology (for the better!).
Another patch will sweep the rest of our internal names, this tries only to
make things compile and fix up the BOLT quotes.
1. Inside payload: current_blinding_point -> current_path_key
2. Inside update_add_htlc TLV: blinding_point -> blinded_path
3. Inside blinded_path: blinding -> first_path_key
4. Inside onion_message: blinding -> path_key.
5. Inside encrypted_data_tlv: next_blinding_override -> next_path_key_override
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is obsolete (since modern onions) and so removed from spec.
We should not set it, and don't need to handle it specially.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we should support it by default.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: `option_quiesce` enabled by default.
Changelog-Deprecated: Config: --experimental-quiesce: it's now the default.
We build with this: it changes the blinded_path field to sciddir_or_pubkey.
But it wasn't committed, so if someone rebuilt the wire files they'd be wrong.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we know the total reservations on each hop, we can more easily
determine probabilities than using flowset_probability() which has to
replicate this collision detection.
We leave both in place for now, to check. The results are not
identical, due to slightly different calculation methods.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were trying to get the max capacity of a flow to see if we could add some
more sats, and hit an assertion:
tests/test_askrene.py:707:
```
DEBUG plugin-cln-askrene: notify msg info: Flow reduced to deliver 88070161msat not 90008000msat, because 107x1x0/1 has remaining capacity 88071042msat
DEBUG plugin-cln-askrene: notify msg info: Flow reduced to deliver 284138158msat not 284787000msat, because 108x1x0/1 has remaining capacity 284141000msat
**BROKEN** plugin-cln-askrene: Flow delivers 129565000msat but max only 56506138msat
INFO plugin-cln-askrene: Killing plugin: exited during normal operation
```
We need to *unreserve* our flow before asking for max capacity. We were
also missing a few less important cases where we altered flows without altering
the reservation, so fix those too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that increasing mu a little bit sometimes made a big difference,
because by completely ignoring fees we were choosing the worst of two channels
in some cases.
Start at 1% fees; this saves a lot on initial fees in this test!
Here's the new stats on mu levels:
96 mu=1
90 mu=10
41 mu=20
30 mu=30
24 mu=40
19 mu=50
22 mu=60
8 mu=70
95 mu=80
19 mu=90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: `askrene` is now better at finding low-fee paths.
While the `k=8` value worked for the current main network tests with the
amounts in those tests, it wasn't robust across a wider range of values
(as demonstrated when other test changes broke tests!).
Time to do this properly: calculate the ratio at the time we combine them,
using median values.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Even after the previous fix, we still occasionally increase fees when my increases.
This is due to the difference between MCF's linear fees, and actual fees, and
is unavoidable, but add a check if it somehow happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed this in the logs:
plugin-cln-askrene: notify msg unusual: The flows had a fee of 151950msat, greater than max of 53697msat, retrying with mu of 10%...
plugin-cln-askrene: notify msg unusual: The flows had a fee of 220126msat, greater than max of 53697msat, retrying with mu of 20%...
We would expect increasing mu to *reduce* the fee!
Turns out that our linear fee is a bad terrible approximation, because I
was using base_fee_penalty of 10.0.
|
| / __ <- real fee, with base: fee = base + propfee * amount.
| / __/
| _//
| __/
| __/_/
|/ _/
| _/ <- linearized fee: fee = linear * amount
|/
+-----------------------------------
These cross over where linear = propfee + base / amount. Assume we split the
payment into 10 parts, this implies that the base_fee_penalty should be 10 / amount
(this gives a slight penalty to the normal case, but that's ok).
This gives better results, too: we get down to 650099 sats in fees, vs 801613
before.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During "test_real_data", then only successes with reduced fees were 92 on "mu=10", and only
1 on "mu=30": the rest went to mu=100 and failed.
I tried numerous approaches, and in the end, opted for the simplest:
The typical range of probability costs looks likes:
min = 0, max = 924196240, mean = 10509.4, stddev = 1.9e+06
The typical range of linear fee costs looks like:
min = 0, max = 101000000, mean = 81894.6, stddev = 2.6e+06
This implies a k factor of 8 makes the two comparable.
This makes the two numbers comparable, and thus makes "mu" much more
effective. Here are the number of different mu values we succeeded at:
87 mu=0
90 mu=10
42 mu=20
24 mu=30
17 mu=40
19 mu=50
19 mu=60
11 mu=70
95 mu=80
19 mu=90
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The current prob_cost_factor setting does not seem to make mu very
effective, in fact, it gives strange results:
plugin-cln-askrene: notify msg unusual: The flows had a fee of 151950msat, greater than max of 53697msat, retrying with mu of 10%...
plugin-cln-askrene: notify msg unusual: The flows had a fee of 220126msat, greater than max of 53697msat, retrying with mu of 20%...
We would expect increasing mu to *reduce* the fee!
As a first step, simplify (it can't be infinite, and the -1 are weird).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We ask it again, but reduce fees by 1msat from the previous answer.
This is really nasty, as it frequently exercises the case where we
only go over fee when we do the refinement step.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
