Commit Graph

225 Commits

Author SHA1 Message Date
Rusty Russell
063a883303 pytest: make tor-requiring tests clearly distinguishable
This way you can run pytest with '-k no _tor_'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-22 11:08:04 +09:30
Rusty Russell
745d3f6226 pytest: fix race in test_gossip_not_dying
Writing to the gossip_store file is not explicitly synchronized, so it
seems that connectd has not caught up with the dying flags we've set.

Simply wait for a second.  Hacky, but should work.

```
    def test_gossip_not_dying(node_factory, bitcoind):
        l1 = node_factory.get_node()
        l2, l3 = node_factory.line_graph(2, wait_for_announce=True)
    
        l1.rpc.connect(l2.info['id'], 'localhost', l2.port)
        # Wait until it sees all the updates, node announcments.
        wait_for(lambda: len([n for n in l1.rpc.listnodes()['nodes'] if 'alias' in n])
                 + len(l1.rpc.listchannels()['channels']) == 4)
    
        def get_gossip(node):
            out = subprocess.run(['devtools/gossipwith',
                                  '--initial-sync',
                                  '--timeout-after=2',
                                  '{}@localhost:{}'.format(node.info['id'], node.port)],
                                 check=True,
                                 timeout=TIMEOUT, stdout=subprocess.PIPE).stdout
    
            msgs = []
            while len(out):
                l, t = struct.unpack('>HH', out[0:4])
                msg = out[2:2 + l]
                out = out[2 + l:]
    
                # Ignore pings, timestamp_filter
                if t == 265 or t == 18:
                    continue
                # channel_announcement node_announcement or channel_update
                assert t == 256 or t == 257 or t == 258
                msgs.append(msg)
    
            return msgs
    
        assert len(get_gossip(l1)) == 5
    
        # Close l2->l3, mine block.
        l2.rpc.close(l3.info['id'])
        bitcoind.generate_block(1, wait_for_mempool=1)
    
        l1.daemon.wait_for_log("closing soon due to the funding outpoint being spent")
    
        # We won't gossip the dead channel any more (but we still propagate node_announcement)
>       assert len(get_gossip(l1)) == 2
E       assert 4 == 2
E        +  where 4 = len([b'\x01\x01L\xc2\xbe\x08\xbb\xa8~\x8f\x80R\x9e`J\x1cS\x18|\x12\n\xe5_6\xb0\xa6S\x9fU\xae\x19\x9c\x1fXB\xab\x81N\x13\xdc\x8e}\xb9\xb0\xb6\xe6\x14h\xd4:\x90\xce\xc3\xad\x9ezR`~\xba@\xc9\x91e\x89\xab\x00\x07\x88\xa0\x00\n\x02i\xa2d\xb8\xa9`\x02-"6 \xa3Y\xa4\x7f\xf7\xf7\xacD|\x85\xc4l\x92=\xa53\x89"\x1a\x00T\xc1\x1c\x1e<\xa3\x1dY\x02-"SILENTARTIST-27fc801-modded\x00\x00\x00\x00\x00\x00\x00', b'\x01\x01M\x00\x86\x8e4\xc8\x90p\n\x98\xf7\xce4\x1e\xd9\xd6-6\xfb(\xf0\xe4\xb7\x90\x7f\x89\xb9\xfa\x00\x82\x1b\xeb\x1fY\x93\x1e\xe0c\xb2\x0e<\xe6\x06x\xb7\xe54};\xfbd\xa0\x01S\xcf\xe8{\xf8\x8f/\xa7\xc0\xe2h\x00\x07\x88\xa0\x00\n\x02i\xa2d\xb8\xa9`\x03]+\x11\x92\xdf\xba\x13N\x10\xe5@\x87]6n\xbc\x8b\xc3S\xd5\xaavk\x80\xc0\x90\xb3\x9c:]\x88]\x03]+HOPPINGFIRE-27fc801-modded\x00\x00\x00\x00\x00\x00\x00\x00', b'\x01\x02~\xe0\x13\xb4\x84Gz\xcf(\xd4w\xa7\x9bZ\x1a\xe82\xd1\xe1\x1bLm\xc8\n\xcd\xd4\xfb\x88\xf8\xc6\xdbt\\v\x89~\xd1.e\xc8\xa8o\x9c`\xd5\xa8\x97\x11l\xf2g\xcb\xa8\xcf\r\x869\xd3\xb5\xd5\x9a\xa0my\x9f\x87\xebX\x0b\x9e_\x11\xdc!\x1e\x9f\xb6j\xbb6\x99\x99\x90D\xf8\xfe\x14h\x01\x16#\x936B\x86\xc6\x00\x00g\x00\x00\x01\x00\x00d\xb8\xa9d\x01\x02\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\n\x00\x00\x00\x00;\x023\x80', b"\x01\x0284\xf1a\x86z\x8e\xf2\xe5'\xf7\xfe1\x8d\x96R\x0c\xe7\x1fj#\xaf\xbd/\xba\x10e\xd1\xccQ-\xcf/>\xa5g\xc6\xd8\x9cO \xe7~\xb3\xda\xe0\\vg\xfb\x02&T\x93\xa0\xd4\x95\x8e\xd5L\x12\x9a\xf7\xe6\x9f\x87\xebX\x0b\x9e_\x11\xdc!\x1e\x9f\xb6j\xbb6\x99\x99\x90D\xf8\xfe\x14h\x01\x16#\x936B\x86\xc6\x00\x00g\x00\x00\x01\x00\x00d\xb8\xa9d\x01\x03\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\n\x00\x00\x00\x00;\x023\x80"])
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-22 08:52:55 +09:30
Rusty Russell
55d6a13ffc gossipd: aggressively advertize *both* sides of channel_update.
While one side was not produced by us, we have a vested interest in propagating it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: When we send our own gossip when a peer connects, also send any incoming channel_updates.
2023-07-21 07:13:11 +09:30
Rusty Russell
7409a93d17 gossipd: when we dump our own gossip, include our node_announcement.
@endothermicdev and I found this while investigating a "nobody sees my node_announcement" bug report.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #6410
Reported-by: benjaminchodroff on discord
Changelog-Fixed: Protocol: When we send our own gossip when a peer connects, send our node_announcement too (regression in v23.05)
2023-07-21 07:13:11 +09:30
Rusty Russell
ef4db6648d pytest: test for force-broadcasting our own gossip.
We do this as suggested by the spec: send our own gossip even if they didn't ask for it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-21 07:13:11 +09:30
Rusty Russell
656517f643 connectd: don't gossip dying channels.
Fixes: #6368
Changelog-Fixed: Protocol: we no longer gossip about recently-closed channels (Eclair gets upset with this).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-20 11:47:32 +09:30
Rusty Russell
26bb791298 pytest: test for whether we gossip spent channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-07-20 11:47:32 +09:30
Rusty Russell
4b9537730b pytest: fix flake in test_routing_gossip.
If we mine too fast, gossip can reach a node which considers it too far in the guture.  Break it up.

```
    @pytest.mark.developer("Too slow without --dev-fast-gossip")
    def test_routing_gossip(node_factory, bitcoind):
        nodes = node_factory.get_nodes(5)
    
        for i in range(len(nodes) - 1):
            src, dst = nodes[i], nodes[i + 1]
            src.rpc.connect(dst.info['id'], 'localhost', dst.port)
            src.openchannel(dst, CHANNEL_SIZE, confirm=False, wait_for_announce=False)
    
        # openchannel calls fundwallet which mines a block; so first channel
        # is 4 deep, last is unconfirmed.
    
        # Allow announce messages.
        mine_funding_to_announce(bitcoind, nodes, num_blocks=6, wait_for_mempool=1)
    
        # Deep check that all channels are in there
        comb = []
        for i in range(len(nodes) - 1):
            comb.append((nodes[i].info['id'], nodes[i + 1].info['id']))
            comb.append((nodes[i + 1].info['id'], nodes[i].info['id']))
    
        def check_gossip(n):
            seen = []
            channels = n.rpc.listchannels()['channels']
            for c in channels:
                seen.append((c['source'], c['destination']))
            missing = set(comb) - set(seen)
            logging.debug("Node {id} is missing channels {chans}".format(
                id=n.info['id'],
                chans=missing)
            )
            return len(missing) == 0
    
        for n in nodes:
>           wait_for(lambda: check_gossip(n))

tests/test_gossip.py:721: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

success = <function test_routing_gossip.<locals>.<lambda> at 0x7f3200534ef0>
timeout = 180

    def wait_for(success, timeout=TIMEOUT):
        start_time = time.time()
        interval = 0.25
        while not success():
            time_left = start_time + timeout - time.time()
            if time_left <= 0:
>               raise ValueError("Timeout while waiting for {}".format(success))
E               ValueError: Timeout while waiting for <function test_routing_gossip.<locals>.<lambda> at 0x7f3200534ef0>
2023-07-10 13:37:22 +02:00
Rusty Russell
21958879cf lightningd: deprecated --announce-addr-dns.
This obsoletes the use of --announce-addr-dns which I know Michael
didn't really like either.

Changelog-Deprecated: Config: `announce-addr-dns`; use `--bind-addr=dns:ADDR` for finer control.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-06-01 09:28:39 +09:30
Rusty Russell
8ef4b36a1f gossipd: send our own gossip aggressively when a new peer connects.
This was previously the role of connectd, but it's actually more
efficient for us to do it: connectd has to sweep through the entire
gossip_store, but we have datastructures for this already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-13 08:48:50 -07:00
Rusty Russell
ba7901bebd pytest: fix up test_gossip_ratelimit.
We were in fact feeding l1 its own gossip, which it doesn't ratelimit (this was
a bit fuzzy before, but definitely is the case now!).

So make this node actually l3, so we test what we expected to test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-04-13 08:48:50 -07:00
Rusty Russell
4bc5d6a0c5 pytest: remove zombie test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2023-03-06 16:15:22 -06:00
Michael Schmoock
eea4781606 pytest: allow ipv6 in test_announce_dns_suppressed
The test runs fine on CI but can fail locally on IPv6 systems,
as the address descriptor is just checked agains IPv4.

Changelog-None
2023-02-13 16:33:25 -06:00
Greg Sanders
eef0c087fc More accurate elements commitment tx size estimation 2023-02-06 18:50:26 -06:00
niftynei
c9c367d770 dual-fund: remove anchor assumption for all dual-funded channels
Only add the anchor channel_type if it's negotiated separately!
2023-02-04 15:31:16 +10:30
Rusty Russell
9ab488fc41 plugins/topology: add direction field to listchannels.
It's a core concept in the spec which isn't directly exposed.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listchannels` added a `direction` field (0 or 1) as per gossip specification.
2023-01-30 15:15:41 -06:00
Alex Myers
98bfd23fff pytest: test_channel_resurrection now succeeds 2023-01-30 16:33:03 +10:30
Alex Myers
6bff10cd40 gossip_store: add a flag for zombie entries
This will allow gossipd to store and persist gossip for channels rather
than deleting them entirely when the channels are pruned from the
network.
2023-01-30 16:33:03 +10:30
Alex Myers
f2f05117aa pytest: gossipd: test resumption of pruned channels 2023-01-30 16:33:03 +10:30
Vincenzo Palazzo
a2347c7452 tests: use listpeerchannels. 2023-01-13 10:42:42 +10:30
Rusty Russell
ec025344cc lightningd: don't announce names as DNS by default.
This broke BTCPayServer, so revert.  I originally (accidentally!)
implemented this such that it broadcast both DNS and IP entries, but
Michael reported earlier that they still don't propagage well, so
simply suppress them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5795
Changelog-Changeed: Config: `announce-addr-dns` needs to be set to *true* to put DNS names into node announcements, otherwise they are suppressed.
Changelog-Deprecated: Config: `announce-addr-dns` (currently defaults to `false`).  This will default to `true` once enough of the network has upgraded to understand DNS entries.
2022-12-08 11:04:53 +01:00
Alex Myers
8f48406985 pytest: test for gossip store corruption by private channel updates
Adds a simple test covering the gossip_store corruption issue #5656
stemming from failure to delete prior private channel updates in all
cases.
2022-11-03 11:29:11 +01:00
Rusty Russell
d4ef20d54a pytest: fix flake in test_gossip_persistence.
We used to ensure the l3<->l4 channel was private by
simply not mining enough blocks to announce.  Then
we started mining 13 blocks to close the channel, and
it will get announced, causing a failure:

```
        assert non_public(l2) == []
>       wait_for(lambda: non_public(l3) == [scid34, scid34])
...
>               raise ValueError("Timeout while waiting for {}", success)
E               ValueError: ('Timeout while waiting for {}', <function test_gossip_persistence.<locals>.<lambda> at 0x7f6cc69b4170>)
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-27 13:14:20 +02:00
Rusty Russell
a1f62ba0e7 gossipd: don't close non-local channels immediately, add 12 block delay.
This adds a new "chan_dying" message to the gossip_store, but since we
already changed the minor version in this PR, we don't bump it again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: Protocol: We now delay forgetting funding-spent channels for 12 blocks (as per latest BOLTs, to support splicing in future).
2022-09-24 15:22:27 +09:30
Rusty Russell
253b25522b BOLT: update to version which requires option_channel_htlc_max.
We will now simply reject old-style ones as invalid.  Turns out the
only trace we could find is a channel between two nodes unconnected to
the rest of the network.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Protocol: We now require all channel_update messages include htlc_maximum_msat (as per latest BOLTs)
2022-09-24 15:22:27 +09:30
Rusty Russell
daa5269ea2 gossipd: bump gossip_store to indicate all channel_update have htlc_max.
And in the next patch, gossipd will no longer put new ones in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Rusty Russell
fe556d1ed9 gossipd: don't try to upgrade ancient gossip_store.
If they really upgrade directly from 0.9.2, it will simply delete the
store and re-fetch it.

We still update from v9 (which could be v0.11), since it's a noop.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-24 15:22:27 +09:30
Michael Schmoock
e0d6f3ceb1 connectd: DNS Bolt7 #911 no longer EXPERIMENTAL
Changelog-Changed: Bolt7 #911 DNS annoucenent support is no longer EXPERIMENTAL
2022-09-13 06:42:20 +09:30
Alex Myers
0abe2e3af1 pytest: Add debugging to test_gossip_store_compact_on_load
This flake has been difficult to reproduce, so let's dump the gossip
store to aid in debugging. See issue #5410.

Changelog-None
2022-08-26 07:03:10 +09:30
Rusty Russell
8c9fa457ba pytest: fix flake in test_gossip_timestamp_filter
```
        # 0x0100 = channel_announcement
        # 0x0102 = channel_update
        # (Node announcement may have any timestamp)
        types = Counter([m[0:4] for m in msgs])
        assert types['0100'] == 1
>       assert types['0102'] == 2
E       assert 1 == 2

tests/test_gossip.py:324: AssertionError
```

Examining the logs shows that we ask l4 for timestamps
"first_timestamp=1658892115 timestamp_range=13", and the timestamp on
the missing update is exactly `1658892128`.

So round the end time up by 1 for filtering.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 15:08:44 +09:30
Rusty Russell
4cada557ba pytest: don't redirect stderr by default.
Some tests need to inspect it, but most don't, and I suspect I'm missing some
error messages due to this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-21 15:37:05 -05:00
Rusty Russell
2962b93199 pytest: don't assume disconnect finished atomically, and suppress interfering redirects.
In various places, we assumed that when `connected` is false,
everything is finished.  This is not true: we should wait for the
state we expect.

In addition, various places allows reconnections, which interfered
with the logic; suppress them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-18 20:50:04 -05:00
Michael Schmoock
65433de05f options: set DNS port to network default if not specified
- set port for a DNS announcement without port to network default
- remove x-fail

Changelog-Fixed: Port of a DNS announcement can be 0 if unspecified
2022-07-17 23:26:07 +02:00
Michael Schmoock
6d4285d7c4 pytest: add xfail test to show DNS w/o port issue
This adds an X-Fail testcase that demonstrates that currently
the port of a DNS announcement is not set to the corresponding
network port (in this case regtest), but it will be set to 0.

Changelog-None
2022-07-17 23:26:07 +02:00
Rusty Russell
f98df63b75 Revert "pytest: fix test_gossip_no_backtalk flake."
This reverts commit 6cc4858847.

No longer needed now we don't flush gossip rcvd_filter as aggressively.
2022-07-12 21:41:19 +09:30
Rusty Russell
06e1e119aa pytest: fix test_gossip_no_empty_announcements flake.
This is a side-effect of fixing aging: sometimes, we age our
rcvd_filter cache too fast, and thus re-xmit.  This breaks
our test, since it used dev-disconnect on the channel_announce,
but that closes to l3, not l1!

```
>       assert l1.rpc.listchannels()['channels'] == []
E       AssertionError: assert [{'active': T...ags': 1, ...}] == []
E         Left contains 2 more items, first extra item: {'active': True, 'amount_msat': 100000000msat, 'base_fee_millisatoshi': 1, 'channel_flags': 0, ...}
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5403
2022-07-12 21:41:19 +09:30
Rusty Russell
9d1b5a654e pytest: fix flake in test_node_reannounce
Fixes 25699994e5 (pytest: fix flake in test_node_reannounce).

Converting ->set->list does not give deterministic order, so we can
end up with the two lists differing due to order:

```
 May send its own announcement *twice*, since it always spams us.
        msgs2 = list(set(msgs2))
>       assert msgs == msgs2
E       AssertionError: assert ['01012ff5580...000000000000'] == ['01014973d81...000000000000']
E         At index 0 diff: '01012ff55800f5b9492021372d74df4d6547bb0d32aec8d4c932a8c3b044e4bd983c429154e73091b0a2aff1cf9bbf16b37e6e9dd10ce4c2d949217366472acd341b0007800000080269a262bbd1750266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c035180266e453454e494f524245414d000000000000000000000000000000000000000000000000' != '01014973d8160dd8fc28e8fb25c40b9d5c68aed8dfb36af9fc13e4d2040fb3718553051a188ce98239c0bed138e1f8713a64acc7de98c183c9597fa58bf37f0b89bb0007800000080269a262bbd16c022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59022d2253494c454e544152544953542d336333626132392d6d6f6464656400000000000000'
E         Full diff:
E           [
E         +  '01012ff55800f5b9492021372d74df4d6547bb0d32aec8d4c932a8c3b044e4bd983c429154e73091b0a2aff1cf9bbf16b37e6e9dd10ce4c2d949217366472acd341b0007800000080269a262bbd1750266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c035180266e453454e494f524245414d000000000000000000000000000000000000000000000000',
E            '01014973d8160dd8fc28e8fb25c40b9d5c68aed8dfb36af9fc13e4d2040fb3718553051a188ce98239c0bed138e1f8713a64acc7de98c183c9597fa58bf37f0b89bb0007800000080269a262bbd16c022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59022d2253494c454e544152544953542d336333626132392d6d6f6464656400000000000000',
E         -  '01012ff55800f5b9492021372d74df4d6547bb0d32aec8d4c932a8c3b044e4bd983c429154e73091b0a2aff1cf9bbf16b37e6e9dd10ce4c2d949217366472acd341b0007800000080269a262bbd1750266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c035180266e453454e494f524245414d000000000000000000000000000000000000000000000000',
E           ]
``

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 12:27:05 +09:30
Rusty Russell
6cc4858847 pytest: fix test_gossip_no_backtalk flake.
Because we expire cache every flush, in DEVELOPER mode that can happen in
just over a second.  If gossipd takes a while to process the gossip,
this can mean we actually forget we received it from the peer.

Easiest fix is to run this test in non-DEVELOPER mode.

```       # With DEVELOPER, this is long enough for gossip flush.
        time.sleep(2)
>       assert not l3.daemon.is_in_log(r'\[OUT\] 0100')
E       AssertionError: assert not '2022-06-30T06:00:31.031Z 022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-connectd: [OUT] 0100294834e94aa2407ace65373c84c2a8056d67a6778c22bd43f03a9000d8a01f075c3996ab6c88951bd9805adbfde78292022d7514ffcec37d5e466ef5ebd235c87ffa8a8567904502cd65ae913dbb8014509f442a6e37b00fd59fa3a06811a4965f45df1612d3a32b2566001430cfec4346cc021bd0a1bf6a87417dbc29e0715f16c4f9dc5a01bff07368ba8f54cf9813c3fc8f9f8b8c8005da2a18021f322a9a168af8f82d9deba49ced51bb26a9abc001bbad314a15baccc87c685e9302533c2e86a7b9dfe90bd4cb6be5c2da375be8c4a535a63cacff0544957a34ca865fdb61f9198d19dfd990a0fcbf8de39fa2c0fc54435c16e74df2b49fe3b6e905d599000006226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f0000670000010001022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d590266e4598d1d3c415f572a8488830b60f7e744ed9235eb0b1ba93283b315c0351802e3bd38009866c9da8ec4aa99cc4ea9c6c0dd46df15c61ef0ce1f271291714e5702324266de8403b3ab157a09f1f784d587af61831c998c151bcc21bb74c2b2314b'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 09:59:52 +09:30
Rusty Russell
6aec374674 lightningd: make log-prefix actually prepend all log messages as expected.
It actually only sets the prefix for the lightningd core log messages;
the other logs have their own prefix.

Make it a real, process-wide prefix which actually goes in front of the timestamp.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: options: `log-prefix` now correctly prefixes *all* log messages.
2022-07-09 09:59:52 +09:30
Alex Myers
87a66c1802 gossipd: store and index most recent and last non-rate-limited gossip
This grows the routing state in order to index both okay-to-broadcast
and rate-limited gossip. The gossip_store also logs the rate-limited
gossip if useful. This allows the broadcast of the last non-rate-limited
gossip.
2022-07-06 14:31:19 +09:30
Alex Myers
08a2b3b86c pytest: test_gossip_ratelimit checks routing graph and squelch 2022-07-06 14:31:19 +09:30
Alex Myers
cbafc0fa33 gossip_store: add flag for spam gossip, update to v10
This will be used to decouple internal use of gossip from what is
passed to gossip peers. Updates GOSSIP_STORE_VERION to 10.

Changelog-Changed: gossip_store updated to version 10.
2022-07-06 14:31:19 +09:30
Vincenzo Palazzo
2c9d7484a2 test: introduce the default port function and remove hard coded port 2022-06-28 06:09:01 +09:30
Rusty Russell
25699994e5 pytest: fix flake in test_node_reannounce
Again, our new behaviour of sending our own gossip even before
they ask can confuse our gossip query tests.

In this case, simply eliminate duplicates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell
a677ea404b pytest: fix flake in test_query_short_channel_id
Again, our new behaviour of sending our own gossip even before
they ask can confuse our gossip query tests.

Create a new node, attach it, and perform queries on it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell
4970704d24 pytest: fix gossipwith flake harder.
Even though we generally wait until a node has seen the gossip,
that doesn't mean that connectd has processed it!  This means when
we connect it may still send us "old" gossip.

So we set the OPT_GOSSIP_QUERIES bit, which means don't send until we
ask.  But now it sends us WIRE_QUERY_CHANNEL_RANGE, so everyone needs
to filter that out.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell
c1a111b68c pytest: wait for stderr, rather than asserting.
Two almost identical assertion fails under CI.  Try waiting.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell
930860c781 pytest: fix flakes in test_gossip_query_channel_range
Nodes now spray their own gossip at the start; this can race with
our query, and they can give us gossip we didn't ask for.

The fix is to always query an uninvolved node: in this case, we
only connected l3 and l4 to test zlib, which is gone, so now we can
remove that and simply always query l4.

```
2022-05-23T01:32:53.4695809Z         # reply_channel_range == 264
2022-05-23T01:32:53.4696416Z >       assert msgs == ['0108'
2022-05-23T01:32:53.4696835Z                         # blockhash
2022-05-23T01:32:53.4697328Z                         + genesis_blockhash
2022-05-23T01:32:53.4697841Z                         # first_blocknum, number_of_blocks, complete
2022-05-23T01:32:53.4698685Z                         + format(0, '08x') + format(65535, '08x') + '01'
2022-05-23T01:32:53.4699245Z                         # encoded_short_ids
2022-05-23T01:32:53.4699859Z                         + format(len(encoded) // 2, '04x')
2022-05-23T01:32:53.4700255Z                         + encoded]
2022-05-23T01:32:53.4700799Z E       AssertionError: assert ['0100194b419...700000010000'] == ['010806226e4...700000010000']
2022-05-23T01:32:53.4710678Z E         At index 0 diff: '0100194b4190af98beedf231901aeda2dcee0e757dd5327219aa007ea6027e769c98665f49e0f99e03b91e80896ed01cd9c4b157c77e908852097d2ba753c6ca01ae41a90a81c9f492c770d9dd35010c702c7c564045cf85a4bf6efaf25cceb9b81b56d1526429d3c60bf1c7ed6a835720730fd7ad1851a937d18aecf8bc3400dd4e0a30d5c4d65999ac39ae6eb3b98d27e3157973598256d2dd4718cd39010a3612580e7f3bba9eda0fb595976f955e345b6affa62fd556f7257a30d54dd87743dc38585e7c31e1ca80b42cf803bd4c1aaa3127351c95de8a4198c46eb0bd7b4ce175a4b51fcaf9676b01806d237758494fcf3d1558279060a0c3b84806a3330c41000006226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f0000700000010000035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de19903c47b1b2afcfd68c7b86c0976adb9a4f0835bc78242fced745d78433497a867d6021c29746e4136ddff456483df3980c2d0d5f31c93ef5ded564f7294a10d7414aa' != '010806226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f000000000000ffff01001900000068000001000100006900000100000000700000010000'
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell
aefcea44a3 pytest: fix flake in test_gossip_timestamp_filter
We didn't get the second channel_announcement in the filter:
```
        # Now choose range which will only give second update.
        msgs = l4.query_gossip('gossip_timestamp_filter',
                               genesis_blockhash,
                               after_12 - backdate,
                               after_23 - after_12 + 1,
                               filters=['0109', '0012'])
    
        # 0x0100 = channel_announcement
        # 0x0102 = channel_update
        # (Node announcement may have any timestamp)
        types = Counter([m[0:4] for m in msgs])
>       assert types['0100'] == 1
E       assert 0 == 1
```

Looking at the gossip store:

```
1089: t=1653008725 channel_announcement: 0100420c13f233d9b461b035257b77704e0003a0280efdbb8f8964bf4c5bd22b8f8b2e205a3c57d7d5f4dcd812cd41be3d8f84c60046c1fd31abefd0985c2263f8f5074bb7dff548ac106dc10ef4485f5921d9b252d094f9518b44a842c1a2c7f98759fa2f8803febeb59e50b65a8f258877e18f6224262e218de38f21438b2112ef0425d64174092bcd8d757f18982c3fb83c8ae2fb5ee88363248e51bdea247cef055a01a17580b60735b82b88fea87dccb724a743beaf4bf3cf578b0da445afcc786fe3b061429f3288e6970819236e97134898ce900b7b61159148d6785e26932e21f1c337cb6f86a16e967e27396b168a19b0d1478c4be2579a9b96aa0125aa000006226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006e0000010000022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d02d595ae92b3544c3250fb772f214ad8d4c51425033740a5bcc357190add6d7e7a02d6063d022691b2490ab454dee73a57c6ff5d308352b461ece69f3c284f2c2412
1533: channel_amount: 100000sat
1555: t=1653008725 channel_update: 01022b98e7d680ffa8096d0a2bda62c0b8b3fc975a5f91c0cff0a42a76b37285276c0156581f85c0be2ee0ed7e7003cb67f5d476a7e3714960836b882bb3ad78097406226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006e00000100006286e955010000060000000000000000000000010000000a0000000005e69ec0
1705: t=1653008729 node_announcement: 010175f8ed9f48db511d43f1da96aa307ccc53a521dcc2f8fb732c6938650e6cc00f672e772134a29e16b723a1e9cf626cf942323bd3fa294befd25430a9395c1fa40007800000080269a26286e959035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d035d2b484f5050494e47464952452d34646636366661000000000000000000000000000000
1866: t=1653008720 channel_update: 010266385083f1d6e554e1701f2456c24212c5aa937f7b090d32d243678b2fe78f7f52ed3626d6cdd52703bf78013bb9d1a621ae5b7d68fc965d29bb94574e340b5606226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006e00000100006286e950010100060000000000000000000000010000000a00000
```

It's not clear what we times we were querying, but let's make it
clearer by putting a full 10 seconds between them, and making our
queries more exact.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-27 17:21:35 +09:30
Rusty Russell
c3efba16ff JSON: don't print deprecated amount fields any more
A small change in one routine creates a lot of changes!  We actually
recommended moving away from these in v0.7.0 (2019-02-28), but never
deprecated them formally.

Changelog-Deprecated: JSON-RPC: `pay`, `decode`, `decodepay`, `getroute`, `listinvoices`, `listpays` and `listsendpays` `msatoshi` fields (use `amount_msat`).
Changelog-Deprecated: JSON-RPC: `getinfo` `msatoshi_fees_collected` field (use `fees_collected_msat`).
Changelog-Deprecated: JSON-RPC: `listpeers` `channels`: `msatoshi_to_us`, `msatoshi_to_us_min`, `msatoshi_to_us_max`, `msatoshi_total`, `dust_limit_satoshis`, `our_channel_reserve_satoshis`, `their_channel_reserve_satoshis`, `spendable_msatoshi`, `receivable_msatoshi`, `in_msatoshi_offered`, `in_msatoshi_fulfilled`, `out_msatoshi_offered`, `out_msatoshi_fulfilled`, `max_htlc_value_in_flight_msat` and `htlc_minimum_msat` (use `to_us_msat`, `min_to_us_msat`, `max_to_us_msat`, `total_msat`, `dust_limit_msat`, `our_reserve_msat`, `their_reserve_msat`, `spendable_msat`, `receivable_msat`, `in_offered_msat`, `in_fulfilled_msat`, `out_offered_msat`, `out_fulfilled_msat`, `max_total_htlc_in_msat` and `minimum_htlc_in_msat`).
Changelog-Deprecated: JSON-RPC: `listinvoices` and `pay` `msatoshi_received` and `msatoshi_sent` (use `amount_received_msat`, `amount_sent_msat`)
Changelog-Deprecated: JSON-RPC: `listpays` and `listsendpays` `msatoshi_sent` (use `amount_sent_msat`)
Changelog-Deprecated: JSON-RPC: `listforwards` `in_msatoshi`, `out_msatoshi` and `fee` (use `in_msat`, `out_msat` and `fee_msat`)
Changelog-Deprecated: JSON-RPC: `listfunds` `outputs` `value` (use `amount_msat`)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-06-21 06:52:35 +09:30