Commit Graph

11176 Commits

Author SHA1 Message Date
Rusty Russell
9f06a59e3c shutdown: don't allow shutdown to p2pkh or p2sh addresses for anchor outputs.
This doesn't have an effect now (except in experimental mode), but it
will when we support anchors.  So we deprecate the use of those in the
close command too.

For experimental mode we have to avoid using p2pkh; adapt that test.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: JSON-RPC: `shutdown` no longer allows p2pkh or p2sh addresses.
2022-04-02 09:40:18 +10:30
Rusty Russell
7491af5495 doc: update BOLT references part1, including bolt11 test vectors with payment_secret.
The signatures on the new examples are sometimes different from what we produce though?
They're valid, however.

And one example has an unneeded feature 5-bit; it's not *wrong*, but
it's not optimal.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell
3b536400f9 specs: update CSV files.
Regenerate from current BOLTS via `make extract-bolt-csv`

1. The remote_addr field was added manually into peer_wire.csv: this
   needs to be a patch otherwise it vanishes on regen.
2. We never brought into the channel_disabled fields, because it was
   too much hassle (we never actually generate this!).  Do it now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell
906fb6ca86 wire/tlvstream: const.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-02 09:40:18 +10:30
Rusty Russell
9bddfc2048 connectd: take dev-suppress-gossip from gossipd.
Gossipd didn't actually suppress all gossip, resulting in a flake!
Doing it in connectd now makes much more sense.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 19:38:05 +10:30
Rusty Russell
2f7f7ec255 lightningd: add "style" to listforwards.
Suggested-by: @t-bast
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `listforwards` has new entry `style`, currently "legacy" or "tlv".
2022-03-31 19:38:05 +10:30
Rusty Russell
116a77f1be lightningd: Restore forwarding of legacy onions.
Partial revert of 43a833e405
"lightningd: remove support for legacy onion format."; we restore the
ability to decode legacy onions for forwarding, but not to generate them.
(We don't accept them properly since making payment_secret compulsory
anyway, so no real change there!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: Protocol: ... but we still forward legacy HTLC onions for now.
2022-03-31 19:38:05 +10:30
Rusty Russell
141d4ef675 pytest: test legacy onion acceptance.
Using a canned sendonion call from an old version (v0.10.2), where I
forced it to use legacy onion for l2.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 19:38:05 +10:30
Rusty Russell
910d594214 doc: add partid to sendonion schema.
We never called this except from plugins, which didn't test schema.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 19:38:05 +10:30
Rusty Russell
cd9ce92d28 onchaind: fix htlc_stub organization.
We get sent three corresponding arrays:
1. htlc stubs
2. whether we want to know if they're missing,
3. whether to wait 3 blocks or tell us immediately

We then sorted the htlc stubs by CLTV, *but didn't sort the corresponding arrays*.

This fixes that the simplest way possible, and probably also:

Fixes: #4649

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: onchaind: we sometimes failed to close upstream htlcs if more than one HTLC is in flight during unilateral close.
2022-03-31 13:40:27 +10:30
Rusty Russell
861922fb1c channeld: log more information about restoring HTLCs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell
0b7f789291 lightningd: extra sanity checks and rescue attempts for missing HTLCs.
These trip when anything weird happens; turns out that we tell
onchaind about old htlcs (e.g. for penalties), so in that case we can
actually have it tell us about missing HTLCs which we no longer have
in memory.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell
e616b4fff7 lightningd: add extra debugging for weird onchain htlc interactions.
This doesn't actually fix anything, but may shed more clues if it
happens again.

The broken() logs are overzealous, see next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell
5033e22835 pytest: make test_onchain_close_upstream more accurately reflect report.
1. Don't use dust HTLCs.
2. Make l3 unresponsive, like report.
3. Make l2-l3 fail because we time out on successive HTLC.

We use sendpay rather than pay, because pay can do multiple attempts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell
1e08c3b882 pytest: simple "does onchaind on missing HTLC cause upstream failure" test.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Rusty Russell
ea7120a313 lightningd: add --dev-no-ping-timer to avoid ping response timeouts.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 13:40:27 +10:30
Vincenzo Palazzo
a35e1b23e6 docker-ci: fixed script path inside the entry point
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-31 13:01:04 +10:30
Vincenzo Palazzo
154a391cf7 ci: adding pytest parameter to tracing lnprotetest
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-31 13:01:04 +10:30
Vincenzo Palazzo
d9d30fb8e0 docker-ci: upgrade pip version in the bash script
The CI fails with the error

```
EnvCommandError

  Command ['/usr/bin/python3', '-m', 'pip', 'install', '--no-deps', '-U', '/root/.cache/pypoetry/artifacts/07/6f/ab/ca33bde7c6751a5ad8d13495b766891cd70e61786112885733ce9b0562/cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl'] errored with the following return code 1, and output: 
  ERROR: cryptography-36.0.2-cp36-abi3-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.
  

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env.py:1195 in _run
      1191│                 output = subprocess.check_output(
      1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1193│                 )
      1194│         except CalledProcessError as e:
    → 1195│             raise EnvCommandError(e, input=input_)
      1196│ 
      1197│         return decode(output)
      1198│ 
      1199│     def execute(self, bin, *args, **kwargs):
```

The solution is to upgrade the pip version as suggested in https://github.com/python-poetry/poetry/issues/2688#issuecomment-937837619

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>x
2022-03-31 13:01:04 +10:30
Vincenzo Palazzo
20523f1b03 ci: migrate the protocol test on docker image
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-31 13:01:04 +10:30
Rusty Russell
5cdb16a93c plugins/pay: don't crash on malformed time.
See: https://github.com/ElementsProject/lightning/issues/4991

We seem to correctly set end_time everywhere, so this looks like
a use-after-free somehow?  But this will fix the crash right here :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-31 09:05:53 +10:30
Rusty Russell
5cb4705eb4 unit tests: don't crash if !HAVE_SQLITE3.
Fixes: #4928
Reported-by: @whitslack
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-30 14:20:56 +10:30
Rusty Russell
0c3aa355db tests: add test that pay gets update correct.
This is the part which works: we have another old PR (or simply
uncomment the FIXME section) for the routehint case.

Closes: #4781
See-also: #4808
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-30 14:18:39 +10:30
Christian Decker
0cbf918af1 cln-grpc: Set cln-grpc version to 0.0.1 until it's complete
Suggested-by: Vincenzo Palazzo <@vincenzopalazzo>
2022-03-30 12:15:55 +10:30
Rusty Russell
ae5b98a727 pytest: fix flake due to cln-grpc starting before "public key" message.
e.g.
```
lightningd-1: 2022-03-28T11:02:12.476Z DEBUG   plugin-cln-grpc: add_pem_file processed 1 valid and 0 invalid certs
lightningd-1: 2022-03-28T11:02:12.478Z DEBUG   plugin-cln-grpc: Connecting to \"lightning-rpc\" and serving grpc on 0.0.0.0:36331
lightningd-1: 2022-03-28T11:02:12.478Z DEBUG   connectd: REPLY WIRE_CONNECTD_ACTIVATE_REPLY with 0 fds
lightningd-1: 2022-03-28T11:02:12.478Z INFO    lightningd: --------------------------------------------------
lightningd-1: 2022-03-28T11:02:12.478Z INFO    lightningd: Server started with public key
```

Which means we don't see it, since start() swallows it:

```
>               raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E               TimeoutError: Unable to find "[re.compile('serving grpc on 0.0.0.0:')]" in logs.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-30 12:15:55 +10:30
Christian Decker
09ee28cb51 cln-grpc: Do not start unless a grpc-port is specified
For now we don't want to autostart.

Suggested-by: Rusty Russell <@rustyrussell>
2022-03-30 12:15:55 +10:30
Christian Decker
bf7ad86ef2 cln-grpc: Rename the grpc plugin and binary
Suggested-by: Rusty Russell <@rustyrussell>
Changelog-Added: plugins: `cln-grpc` first class GRPC interface for remotely controlling nodes over mTLS authentication
2022-03-30 12:15:55 +10:30
Christian Decker
461a65ba16 py: Add grpcio-tools for the cln-grpc-plugin 2022-03-30 12:15:55 +10:30
Christian Decker
dd66c85fcb grpc-plugin: Make the grpc port to listen on configurable
Changelog-Added: cln-grpc-plugin: The plugin can be configured to listen on a specific port using the `grpc-port` option
2022-03-30 12:15:55 +10:30
Christian Decker
647ed6a8c8 cln: Add grpc port as configurable option 2022-03-30 12:15:55 +10:30
Christian Decker
6098386182 cln-grpc: Add the connect method 2022-03-30 12:15:55 +10:30
Christian Decker
f5147bbf1e pytest: Add a test for incorrect credentials
If we aren't using the correct certificates we should reject the
connections during the mTLS connection setup. This test tries to
connect with the wrong client cert to the node, and the server will
reject it.
2022-03-30 12:15:55 +10:30
Christian Decker
27e468d2ae grpc-plugin: Generate mTLS certificates and use them in grpc 2022-03-30 12:15:55 +10:30
Christian Decker
d221c9b491 pytest: Add a test for the grpc plugin
Currently still unencrypted, but will get its mTLS authentication in
the next commits.
2022-03-30 12:15:55 +10:30
Christian Decker
a17edeb839 cln-grpc-plugin: Add basic grpc-plugin 2022-03-30 12:15:55 +10:30
Christian Decker
f3d95530f4 make: Generate grpc bindings if we want to test with rust enabled 2022-03-30 12:15:55 +10:30
Christian Decker
8bc4f0137a make: Add a hook for us to depend on generated files for tests
We are about to generate the python grpc bindings, but only when we
have Rust enabled.
2022-03-30 12:15:55 +10:30
Christian Decker
b4c7ba2d8f cln-grpc-plugin: Add scaffolding for the cln-grpc-plugin 2022-03-30 12:15:55 +10:30
Rusty Russell
5c949e3116 subd: make channel/peer own the subd.
We get some memleak reports because ld owns the subd, but once
the peer/channel is freed, there's no reference for the brief time
until the subd exits.

This happens for both opening and closingd.  For openingd, the
peer owns it, for others (including dualopend) the channel owns it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-30 06:27:52 +10:30
Christian Decker
c673c092dc py: Update the PyPI job to use poetry and correct versions
Let's use poetry when we already use it for everything else.

Changelog-None
2022-03-30 06:12:40 +10:30
Rusty Russell
20392ae526 connectd: restore obs2 onion support.
I removed these prematurely: we *haven't* had a release since
introducing them!

This consists of reverting d15d629b8b
"plugins/fetchinvoice: remove obsolete string-based API." and
plugins/fetchinvoice: remove obsolete string-based
API. "onion_messages: remove obs2 support."

Some minor changes due to updated fromwire_tlv API since they
were removed, but not much.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: REVERT: Removed backwards compat with onion messages from v0.10.1.
2022-03-29 10:55:12 +10:30
Rusty Russell
9e11ae1a0b plugins/topology: don't get upset if we hit deleted channel.
There's a race under CI, where a channel is deleted then we see the
channel_update in the gossip store.  We assumed this wouldn't happen,
but it can!

```
[gw1] [ 95%] FAILED tests/test_connection.py::test_multichan
[gw1] [ 95%] ERROR tests/test_connection.py::test_multichan
...
>           raise ValueError(str(errors))
E           ValueError:
E           Node errors:
E            - lightningd-3: had BROKEN messages
E            - lightningd-3: Node exited with return code 1
E           Global errors:
...
lightningd-3: 2022-03-28T00:11:42.160Z DEBUG   wallet: Owning output 0 100000sat (SEGWIT) txid 30616903feba1839a3834e2b3b6123759ce1fe0d76414ca77e2dbc17414772e0 CONFIRMED
lightningd-3: 2022-03-28T00:11:42.392Z DEBUG   hsmd: Client: Received message 5 from client
lightningd-3: 2022-03-28T00:11:42.393Z DEBUG   hsmd: new_client: 2
lightningd-3: 2022-03-28T00:11:42.398Z INFO    plugin-topology: Killing plugin: exited during normal operation
lightningd-3: 2022-03-28T00:11:42.400Z **BROKEN** plugin-topology: Plugin marked as important, shutting down lightningd!
...
----------------------------- Captured stderr call -----------------------------
topology: update for channel 105x1x1 not found!
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-29 10:50:01 +10:30
Vincenzo Palazzo
7e6893af9e rust: fixed compiler warning in the example
Changelog-None: rust: fixed compiler warning in the example

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-29 10:07:25 +10:30
Rusty Russell
aad4495f56 delinvoice: allow desconly arg to only remove the description.
Means that field is now optional in JSON output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON-RPC: `delinvoice` has a new parameter `desconly` to remove description.
2022-03-29 10:04:16 +10:30
Rusty Russell
ccaf04d268 invoice: add deschashonly parameter.
LNURL wants this so they can include images etc in descriptions.

Replaces: #4892
Changelog-Added: JSON-RPC: `invoice` has a new parameter `deschashonly` to put hash of description in bolt11.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-29 10:04:16 +10:30
Rusty Russell
290dfd2b81 doc/schemas: handle delinvoice of paid invoice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-29 10:04:16 +10:30
Rusty Russell
7abc491f4c pay: fix crash on invalid onion in reply in some cases.
fail can be NULL here.

Reported-by: https://github.com/shafemtol
Fixes: #5053
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:59:46 +10:30
Rusty Russell
5c20b8b1a3 devtools/decodemsg: add --list-tlvs
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
b19f3a5e7f devtools/decodemsg: don't require --onion for onion tlvs, fail if unknown tlvname.
Generate a table, let decodemsg sort it out.  Do more up-front work in
argparsing too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
7147fea2ff common: add const to accepted_extra_tlvs arg.
It was tlv_fields_valid that wanted a non-const: now that's gone, we
can make this correctly const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30