Commit Graph

9000 Commits

Author SHA1 Message Date
Rusty Russell
5c167d16ab tools/generate-wire.py: use helpers.
This was terrible cut & paste.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
dc83e64003 tools/generate-wire: don't use void * pointers for tlv fromwire.
And fix up the one place which got it wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
c5a2e64fd4 plugins/libplugin-pay: expose gossmap for plugin special effects.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
695a8bd868 lightningd: split onion_message hook.
Only way to be sure that plugins don't accidentally respond to onion_message
sent via reply path from another message (which would potentially leak our
identity!).

To quote BOLT #7 (Onion Messages) in the offers PR:

```markdown
The reader:
- MUST ignore any message which contains a `blinding` which it did not expect, or does not contain
  a `blinding` when one is expected.
...
`blinding` is critical to the use of blinded paths: there are various
means by which a blinded path is passed to a node.  The receipt of an
expected `blinding` indicates that blinded path has been used: it is
important that a node not accept unblinded messages when it is expecting
a blinded message, as this implies the sender is probing to detect if
the recipient is the terminus of the blinded path.

Similarly, since blinded paths don't expire, a node could try to use
a blinded path to send an unexpected message hoping for a response.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
61422193d9 channeld: hand input blinding to lightningd.
Required to determine if this msg used expected reply path.

Also remove FIXME (om->enctlv is handled above).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
54c57e7495 libplugin-pay: don't expose bolt11 details.
When we support bolt12, this won't exist.  We only need min_final_cltv_expiry,
routes and features, so put them into struct payment explicitly.

We move the default final ctlv out to the caller, too, which is clearer.
e.g. keysend was using this value, but it was hard to tell.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Christian Decker
76124eb800 pyln: Replace undecodeable symbols when tailing logs
Logs may contain non-ASCII and non-UTF8 symbols, which crashes the
tailer. It's better to replace them with a glyph representing
undecodeable symbols instead, and handle the issue further up the
call-chain.
2020-12-04 18:34:12 -06:00
Christian Decker
bb0910999e pytest: Use datetime.strptime instead of datetime.fromisoformat
The latter is available only starting with Python 3.7, and it's the
only test that fails with <3.7. This allows us to keep on testing with
older versions.
2020-12-04 18:34:12 -06:00
Christian Decker
c9d24b1de2 pyln: Set the prefix in TailableProc
We rely on it when logging, but didn't set it in the class, rather we
did in the sub-classes. This proved to be rather annoying to search
since it'd just fail silently and not log any output.
2020-12-04 18:34:12 -06:00
João Paulo
7514d7fb48 plugins: add txprepare to .gitignore
Changelog-None
2020-12-04 11:30:58 +10:30
Rusty Russell
dc745cdf2f peer_htlcs.c: close taken() leak.
Found in tests/test_connection.py::test_restart_many_payments:

`lightningd: outstanding taken(): lightningd/peer_htlcs.c:532:towire_temporary_channel_failure(((void *)0), ((void *)0))`

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-03 11:35:30 +01:00
Rusty Russell
e4950db9a3 pytest: recreate wallet on bitcoind restart.
Doesn't seem to stick in master.  Andy Chow suggested we
simply turn off wallet on older versions, and always create/load.

```
[gw8] [ 40%] FAILED tests/test_misc.py::test_bitcoind_goes_backwards

============================================================= FAILURES ==============================================================
___________________________________________________ test_bitcoind_goes_backwards ____________________________________________________
[gw8] linux -- Python 3.8.5 /usr/bin/python3

node_factory = <pyln.testing.utils.NodeFactory object at 0x7f931859a760>
bitcoind = <pyln.testing.utils.BitcoinD object at 0x7f931865eee0>

    def test_bitcoind_goes_backwards(node_factory, bitcoind):
        """Check that we refuse to acknowledge bitcoind giving a shorter chain without explicit rescan"""
        l1 = node_factory.get_node(may_fail=True, allow_broken_log=True)

        bitcoind.generate_block(10)
        sync_blockheight(bitcoind, [l1])
        l1.stop()

        # Now shrink chain (invalidateblock leaves 'headers' field until restart)
        bitcoind.rpc.invalidateblock(bitcoind.rpc.getblockhash(105))
        # Restart without killing proxies
        bitcoind.rpc.stop()
        TailableProc.stop(bitcoind)
        bitcoind.start()

        # Will simply refuse to start.
        with pytest.raises(ValueError):
            l1.start()

        # Nor will it start with if we ask for a reindex of fewer blocks.
        l1.daemon.opts['rescan'] = 3

        with pytest.raises(ValueError):
            l1.start()

        # This will force it, however.
        l1.daemon.opts['rescan'] = -100
        l1.start()

        # Now mess with bitcoind at runtime.
>       bitcoind.generate_block(6)

tests/test_misc.py:1307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
contrib/pyln-testing/pyln/testing/utils.py:399: in generate_block
    return self.rpc.generatetoaddress(numblocks, self.rpc.getnewaddress())
contrib/pyln-testing/pyln/testing/utils.py:322: in f
    return proxy._call(name, *args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bitcoin.rpc.RawProxy object at 0x7f93184f6a30>, service_name = 'getnewaddress', args = ()
postdata = '{"version": "1.1", "method": "getnewaddress", "params": [], "id": 1}'
headers = {'Authorization': b'Basic cnBjdXNlcjpycGNwYXNz', 'Content-type': 'application/json', 'Host': 'localhost', 'User-Agent': 'AuthServiceProxy/0.1'}
response = {'error': {'code': -18, 'message': 'No wallet is loaded. Load a wallet using loadwallet or create a new one with createwallet. (Note: A default wallet is no longer automatically created)'}, 'id': 1, 'result': None}
```
2020-12-03 11:35:30 +01:00
Rusty Russell
4424b41c65 channeld: fix gcc warning on gcc 8.3.0 (Debian 8.3.0-6)
```
channeld/channeld.c:237:2: error: ‘shutdown_status’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
```

Reported-by: az0re on IRC
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-03 11:33:40 +01:00
Rusty Russell
1a3c0a0c0d invoice: fix potential race where invoice is paid/expired while we're calling hook.
There's actually a (very unlikely) race here: we would previously have
crashed with an assertion in invoices_resolve.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 16:50:12 +10:30
ZmnSCPxj jxPCSnmZ
a437936c78 lightningd/log.c: Fix up handling of SIGHUP.
Fixes: #4240

ChangeLog-Fixed: log: Do not terminate on the second received SIGHUP.
2020-12-02 16:48:48 +10:30
ZmnSCPxj jxPCSnmZ
67d8fdcc75 tests/test_misc.py: Add a test for issue #4240. 2020-12-02 16:48:48 +10:30
niftynei
4185a86d90 df-mfc: set the minimum input weight to 110 iif v2
We only need a bumped up input weight if we're talking to nodes that
require v2.
2020-12-02 14:19:08 +10:30
niftynei
b01734087d df-mfc: set protocol type when we know it
We'll use it for figuring out whether or not to set a utxo witness
minimum, which comes much before we were setting this field.

Now we set the protocol as soon as we can reasonably deduce it.
2020-12-02 14:19:08 +10:30
niftynei
e81d78ec4c fundpsbt/utxopsbt: new min_witness_weight param
Needed for v2 of channel opens, where the minimum weight is 110; a
'simple utxo' (sig + key) weighs in at 107, so we a need a way to
establish a floor for this case.

Changelog-Added: JSON-RPC: fundpsbt/utxopsbt have new param, `min_witness_utxo`, which sets a floor for the weight calculation of an added input
2020-12-02 14:19:08 +10:30
niftynei
b4aebc17a1 bitcoin: break out method to calculate input weight
We have a required minimum witness weight for dual funded opens.
2020-12-02 14:19:08 +10:30
niftynei
b23c6c50a0 df-tests: use multifundchannel pathways instead of custom plugin
Since fundchannel/multifundchannel now handles v2 opens, we use that
instead of this hacked together plugin.
2020-12-02 14:19:08 +10:30
niftynei
9f74dd9fb3 df: rename the dual_open wires to match other daemon wires 2020-12-02 14:19:08 +10:30
Rusty Russell
0ad269f5b6 invoice: make invoice_payment hook a multi-user hook.
We register on it for offers, and without this nobody else can.

Changelog-Changed: plugins: more than one plugin can now register invoice_payment hook.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 12:56:15 +10:30
Rusty Russell
ca2bd98082 unittest: use common_setup / common_shutdown almost everywhere.
Avoids much cut & paste.  Some tests don't need any of it, but most
want at least some of this infrastructure.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 12:55:09 +10:30
Rusty Russell
39f4ca98b5 common: check for outstanding taken() pointers in common_shutdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 12:55:09 +10:30
Rusty Russell
ae1a130ab5 ccan: update ccan/utf-8 to reject NULs embedded in strings.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
c29e290ee6 common/json: json_add_stringn helper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
9361b62e3a libplugin: add command_hook_success helper.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
d9586bbde1 json: add json_tok_endswith and json_tok_startswith helpers.
I wanted this for offers, and it's generally useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
3408f8f998 gossipd: fix memleak.
Not really a leak, since we eventually process pending, but still:

```
    **BROKEN** gossipd: MEMLEAK: 0x5562fa63bab8
    **BROKEN** gossipd:   label=wire/peer_exp_wiregen.c:3858:u8[]
    DEBUG   gossipd:   backtrace:
    DEBUG   gossipd:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
    DEBUG   gossipd:     ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)
    DEBUG   gossipd:     wire/peer_exp_wiregen.c:3858 (fromwire_channel_announcement)
    DEBUG   gossipd:     gossipd/routing.c:1706 (handle_channel_announcement)
    DEBUG   gossipd:     gossipd/gossipd.c:238 (handle_channel_announcement_msg)
    DEBUG   gossipd:     gossipd/gossipd.c:444 (peer_msg_in)
    DEBUG   gossipd:     common/daemon_conn.c:31 (handle_read)
    DEBUG   gossipd:     ccan/ccan/io/io.c:59 (next_plan)
    DEBUG   gossipd:     ccan/ccan/io/io.c:407 (do_plan)
    DEBUG   gossipd:     ccan/ccan/io/io.c:417 (io_ready)
    DEBUG   gossipd:     ccan/ccan/io/poll.c:445 (io_loop)
    DEBUG   gossipd:     gossipd/gossipd.c:1730 (main)
    **BROKEN** gossipd:   parents:
    **BROKEN** gossipd:     gossipd/routing.c:1698:struct pending_cannouncement
    **BROKEN** gossipd:     gossipd/gossipd.c:1700:struct daemon
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
25fcc4ddbf Makefile: separate check-python targets.
There's a 60 second delay in one of the contrib tests, and I just want
to run flake8 on my alterations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
f0621cec0d JSON-RPC: don't allow any strings which aren't valid UTF-8.
We already do some sanity checks, add this one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: invalid UTF-8 strings now rejected.
2020-12-02 10:38:04 +10:30
Rusty Russell
5bdd282c2b common/bolt11: reject bad UTF-8 strings.
We don't have a problem with them, but callers may; easier to reject bad
UTF8 here than let the caller fail when it tries to parse output.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Rusty Russell
f1bea50e1d common/utils: simple utf8 helpers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-02 10:38:04 +10:30
Michael Schmoock
7bfb5f10c7 pyln: failing test msat from float str
We were not able to create pyln Millisatoshi from floats, e.g.:
 - "0.01btc"
 - "0.1sat"
 - ...

This adds a test that makes sure this won't happen again.
2020-12-02 10:37:31 +10:30
Michael Schmoock
83a21138b8 pyln: fix msat from float str
Changelog-fixed: pyln: parsing msat from a float string
2020-12-02 10:37:31 +10:30
Michael Schmoock
899022e452 pytest: use default fundamount on two tests 2020-12-01 11:15:30 +10:30
Michael Schmoock
bdad9fcdd9 pyln: def amount 10**6 for fund- and openchannel
Changelog-None
2020-12-01 11:15:30 +10:30
Rusty Russell
68c6ae67b3 external/Makefile: fix parallel build with missing secp headers.
Fixes: #4229
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-30 16:10:26 +01:00
ZmnSCPxj jxPCSnmZ
32de621886 lightningd/plugin_hook.c: Make db_write a chained hook.
Fixes: #4219

Changelog-Changed: Plugins: Multiple plugins can now register `db_write` hooks.
2020-11-30 10:40:11 +10:30
ZmnSCPxj jxPCSnmZ
904e110554 lightningd/plugin.c: Make plugin-exclusive loop support multiple plugins. 2020-11-30 10:40:11 +10:30
Jon Griffiths
5c2fc4c1cb wally: remove wally casts that are no longer needed
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-11-30 10:28:04 +10:30
Jon Griffiths
5bdad04976 tx: Remove wally workaround that is no longer needed
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
2020-11-30 10:28:04 +10:30
Rusty Russell
70977a4fbd libwally: update to latest which uses libsecp-zkp submodule
And get rid of the now-obsolete PYTHON_VERSION hack.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-25 11:59:22 -06:00
niftynei
5e71de7819 df-spec: fail the open if they send us too many tx-collab msgs 2020-11-23 12:41:05 -06:00
niftynei
29c3532856 mfc-psbt: mark all of our inputs as "ours", then only sign those
we only want to sign the inputs that we've reserved via utxopsbt or
fundpsbt. we mark them with a flag (reusing the now defunct max-len
flag is fine), then look for inputs with that flag to pass to signonly
2020-11-23 12:41:05 -06:00
niftynei
6077eca660 df: pass back 'close_to' for completed/commitment secured channels
When commitments are secured, also return the 'close_to' script if we've
got a local_upfront_shutdown_script set.
2020-11-23 12:41:05 -06:00
niftynei
405453859c mfc-df: add 'happy path' tests for the v1+v2 things
We can't test disconnects et.al. quite yet because the 'cancel' flow for
openchannelv2 still needs to be resolved
2020-11-23 12:41:05 -06:00
niftynei
da98a9d0af df-accepter plugin: temporarily dont pass in signpsbt 2020-11-23 12:41:05 -06:00
niftynei
c6b45e052b mfc-df: after openchannel_signed is finished, we call finished
We done!?
2020-11-23 12:41:05 -06:00