We can get bad gossip if a node processes a gossip message after we've closed:
```
_________________________________________ ERROR at teardown of test_closing_specified_destination _________________________________________
...
> raise ValueError(str(errors))
E ValueError:
E Node errors:
E - lightningd-1: had warning messages
E - lightningd-4: had bad gossip messages
E Global errors:
...
lightningd-1 2024-02-03T00:29:02.299Z INFO 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 105x1x0
lightningd-1 2024-02-03T00:29:02.300Z DEBUG 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199-connectd: peer_in WIRE_WARNING
lightningd-1 2024-02-03T00:29:02.300Z INFO 0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 103x1x0
lightningd-1 2024-02-03T00:29:02.339Z DEBUG 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: peer_in WIRE_WARNING
lightningd-1 2024-02-03T00:29:02.339Z INFO 035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 103x1x0
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Instead of "new" and "load", we don't really need to "load" anything,
so do everything in gossip_store_new.
Have it do the compaction/rewrite, and collect the dying records
The gossip_store_load is now basically a noop, since gossmap
does that.
gossipd removes a pile of routines dealing with messages,
in favor of just handing them to gossmap_manage.
The stub gossmap_manage constructor is removed entirely.
We simplified behaviour around channel_announcements with
no channel update: we now add them to the store, and go
back to fix the timestamp later. This changes a test,
which explicitly tests for the old behaviour.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We weakened this progressively over time, and gossip v1.5 makes spam
impossible by protocol, so we can wait until then.
Removing this code simplifies things a great deal!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: we no longer ratelimit gossip messages by channel, making our code far simpler.
We never enabled it, because we seemed to be eliminating valid
channels. We discard zombie-marked records on loading.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It was an obscure dev command, as it never worked reliably.
It would be much easier to re-implement once this is done.
This turned out to reveal a tiny leak on
tests/test_gossip.py::test_gossip_store_load_amount_truncated where we
didn't immedately free chan_ann if it was dangling.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
update_count is just the count of the records for a tx. To calculate onchain fees
for an account we must sum all credits vs debits. We don't need to GROUP BY update_count
nor ORDER BY update_count since it is just a running index of updates to this tx.
Remove grouping by update_count which resulted in a crash due to bad arithmetic
caused by fee calculation returned rows not being consolidated.
Remove xfail.
```
2024-02-02T10:30:38.6479143Z __________________ ERROR at teardown of test_multifunding_one __________________
...
2024-02-02T10:30:38.6491895Z > raise ValueError(str(errors))
2024-02-02T10:30:38.6492208Z E ValueError:
2024-02-02T10:30:38.6492465Z E Node errors:
2024-02-02T10:30:38.6492833Z E - lightningd-2: had bad gossip messages
2024-02-02T10:30:38.6493535Z E Global errors:
...
2024-02-02T10:30:38.7458545Z lightningd-2 2024-02-02T10:25:48.410Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Ignoring future channel_announcment for 103x2x1 (current block 102)
2024-02-02T10:30:38.7460140Z lightningd-2 2024-02-02T10:25:48.411Z UNUSUAL lightningd: Bad gossip order: could not find channel 103x2x1 for peer's channel update
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: added a withdraw all to the end of test_onchain_their_unilateral_out to ensure that the unilateral close info is correct with anchors. Tests https://github.com/Blockstream/greenlight/issues/348
This caused a flake in test_gossip_lease_rates, since the peer would ignore
the node_announcement due to duplicate timestamps!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We mine blocks too fast, and l3 discard the channel_announcment as too far in the future:
```
lightningd-3 2023-12-14T06:40:04.744Z DEBUG 0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-gossipd: Ignoring future channel_announcment for 103x1x1 (current block 102)
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This isuseful to find completely dead channels which are worthy of
closing.
Changelog-Added: JSON-RPC: `listpeerchannels` field `last_stable_connection` showing when we last held an established channel for a minute or more.
Changelog-Added: JSON-RPC: `listclosedchannels` field `last_stable_connection` showing when we last held an established channel for a minute or more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is useful to see if we've really made progress, or just bounced
off the peer (e.g. it doesn't know the channel, or we have fallen
behind somehow).
Changelog-Added: JSON-RPC: `listpeerchannels` new field `reestablished` set once we've exchanged `channel_reestablish` messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Rather than take the first two from peers with committed channels, use
the most common address given by at least 2 peers, and accept the majority
from non-committed peers if there are no committed peers.
This works well with the node_announcement rework, which waits until
everyone has a chance to connect before creating the node_announcement.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This commit is a bit messy, but it tries to do the minimal switchover.
Some tests change, so those are included here.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Mining 13 blocks to close the l1->l2 channel also causes the l2->l3
channel to confirm, and I was reliably getting the
channel_announcement to l3 in time to confuse this wait_for test. So
query the channel we want directly to make it more robust.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
After this, the invoice (correctly!) gives the zeroconf channel as a routehint,
so this test fails. Simple workaround: make invoice before creating zeroconf
channel.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was triggering on private updates after gossip changes.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1diff --git a/tests/test_gossip.py b/tests/test_gossip.py
index 677ec4825..285290b71 100644
When we rewrite, we actually use the remote channel_announcement signatures from the db, and notice that l1 here is actually l2:
```
0266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c03518-chan#1: gossipd rejected our channel announcement: WARNING: Bad node_signature_1 304402204ad9f0ca9b5f1f8eec61290b6008bf2c15f011e35543d25c35c5b89d2e1a183402201aaca5883714dbe11d6ae00a146fac7302352e1b9e987eee5ce98e8eb6a82909 hash e713698e14d3d8f973210e8d93c96d1a7c117c58340ccb195785a672fd0b5ef1 on channel_announcement 01004ad9f0ca9b5f1f8eec61290b6008bf2c15f011e35543d25c35c5b89d2e1a18341aaca5883714dbe11d6ae00a146fac7302352e1b9e987eee5ce98e8eb6a829095c05509264fcf72e19180f0285cf0ca0d786f280a75d79e46e5627be20c9dd9a4e9105aa61d7faf3a375454db94376b9b9df29cf078130ed827836b0849f119014ec750ab163baef701e7889f430ee37ea7e38857aa2b2745440600999c85f2903abefde5210dd06ae12f24b8801416800db84f5c73d35594af594f9001800025dee15519d29eb81e9d1ac761deebdad91ba1d08a05c027c16ea5fc08f3a9b663f8433209392395f0107f6b4d266c46b0946bd098f88c0693817fb560e0526a8000006226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006900000100000266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c035180266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c0351802324266de8403b3ab157a09f1f784d587af61831c998c151bcc21bb74c2b2314b0206540493a98599967a61c45172df49681e398a6cb0890a506e74f76c2d2be242
```
Explicitly remove those messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a complete copy of the `db_provider` and associated classes in
teh `tests/` directory, causing them to drift apart. Consolidating on
one version makes this more maintainable.
Changelog-None
For example, lnprototest got the error 'You gave bad parameters: Did not support channel_type ' which doesn't make it clear that it's rejecting the empty channel type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Let's tell the caller what channel_type they got!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel`, `multifundchannel`, `fundchannel_start` and `openchannel_init`: new field `channel_type`.
As suggested in https://github.com/lightning/bolts/pull/1092.
We still support channels opened without it, but you can no longer open new ones without it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `option_gossip_queries` is now required (advertized by all but 16 nodes)
We're about to make static_remotekey compulsory, but we still want to
do tests for pre-existing channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add a request schema for multifundchannel.
Changelog-Added: JSON-RPC: `fundchannel` and `multifundchannel` now take an optional `channel_type` parameter.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And add request schemas for openchannel_init and fundchannel_start.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `fundchannel_start` and `openchannel_init` now take an optional `channel_type` parameter.
As suggested in https://github.com/lightning/bolts/pull/1092.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `option_gossip_queries` is now required (advertized by all but 11 nodes)
As suggested in https://github.com/lightning/bolts/pull/1092.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: `option_data_loss_protect` is now required (advertized by all but 11 nodes)
Min. Cost Flow does not take into account fees when computing a flow
with liquidity constraints.
This is a work-around solution that reduces the amount on every route to
respect the liquidity bound. The deficity in the delivered amount is
solved by running MCF once again.
Changes:
1. the function `flow_complete` allocates amounts to send over the set of routes
computed by the MCF algorithm, but it does not allocate more than liquidity
bound of the route. For this reason `minflow` returns a set of routes that
satisfy the liquidity bounds but it is not guaranteed that the total payment
reaches the destination therefore there could a deficit in the delivery:
`deficit = amount_to_deliver - delivering`.
2. in the function `add_payflows` after `minflow` returns a set of routes we
call `flows_fit_amount` that tries to a allocate the `deficit` in the routes
that the MCF have computed.
3. if the resulting flows pass all payment constraints then we update
`amount_to_deliver = amount_to_deliver - delivering`, and the loop
repeats as long as `amount_to_deliver` is not zero.
In other words, the excess amount, beyond the liquidity bound,
in the routes is removed and then we try to allocate it
into known routes, otherwise we do a whole MCF again just for the
remaining amount.
Fixes issue #6599
We stopped doing empty journal logs, so we no longer need to switch
our log severity based on whether or not an account exists.
Should make bookkeeper less chatty and remove noisy logs
Changelog-None
We were putting out a lot of empty journal entries. Let's
stop doing that.
Now the wallet balance stays uninitialized until/unless you have
funds in it.
Fixes#5672
