Commit Graph

7944 Commits

Author SHA1 Message Date
Rusty Russell
93d04d08d0 wire: update to latest version of the spec.
The main change here is that the previously-optional open/accept
fields and reestablish fields are now compulsory (everyone was
including them anyway).  In fact, the open/accept is a TLV
because it was actually the same format.

For more details, see lightning-rfc/f068dd0d8dfa5ae75feedd99f269e23be4777381

Changelog-Removed: protocol: support for optioned form of reestablish messages now compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-23 18:49:15 +02:00
niftynei
a66415a43e plugins-test: use pyln.client, not lightning to import Plugin 2020-06-23 14:49:32 +02:00
niftynei
0f8157a02e libwally: update to latest commit on master, which contains psbt fixes
fixes for p2sh-p2wpkh sigs in libwally etc
2020-06-23 14:49:32 +02:00
niftynei
0bd0de54fa psbt: have withdraw_tx use psbt's to create signed txs
this will allow us to add inputs that aren't ours to a tx that we sign
and finalize
2020-06-23 14:49:32 +02:00
niftynei
0a7550473b psbt: add redeemscript info to psbt for utxos that have it 2020-06-23 14:49:32 +02:00
niftynei
85dca84327 psbt-json: remove reliance on bitcoin_tx, use straight wally_psbt struct 2020-06-23 14:49:32 +02:00
niftynei
175fcf381a psbt: have wally_tx serialization methods be legible for gen'd code
our code generators expect the serialization name to match the struct
type
2020-06-23 14:49:32 +02:00
niftynei
b63abef542 psbt: add method to finalize + extract a psbt
will either use a temporary psbt (and not munge the passed in psbt)
or will finalize in place -- finalization erases most of the signature
metadata from the psbt struct
2020-06-23 14:49:32 +02:00
niftynei
0d1f1bc66e psbt: add helper method for setting a psbt input's redeemscript 2020-06-23 14:49:32 +02:00
niftynei
c3ae44e296 psbt: don't crash if we can't add a partial sig
instead return a boolean indicating the success/failure of a sig set
2020-06-23 14:49:32 +02:00
niftynei
80072b389e psbt: remove script sig info from inputs before adding them to global
PSBT's dont' serialize / unserialize if there's any sig info set on the
global transaction
2020-06-23 14:49:32 +02:00
lisa neigut
16656a85cf withdraw: refactor change output handling
We're not using the change_outnum for withdraw tx's (and the way
we were calculating it was broken as of the addition of 'multiple
outputs'). This removes the change output knowhow from withdraw_tx
entirely, and pushes the responsibility up to the caller to
include the change output in the output set if desired.

Consequently, we also remove the change output knowhow from hsmd.
2020-06-23 14:49:32 +02:00
niftynei
09815c7e7f psbt: return NULL instead of aborting on wally-lib problems
This lets us parse invalid/bad psbt data from user input without
crashing
2020-06-23 14:49:32 +02:00
niftynei
5ecacf3dd0 psbt: add type-to-string that prints b64 string
Re-uses code from what was the bitcoin_tx_to_psbt_b64
2020-06-23 14:49:32 +02:00
ZmnSCPxj jxPCSnmZ
c100de6d93 common/utils.c: Correctly handle NULL taken pointer in tal_dup_talarr.
Fixes: #3757

Reported-by: @sumBTC

Changelog-None
2020-06-23 12:48:20 +09:30
ZmnSCPxj jxPCSnmZ
a25af262e9 tests/test_pay.py: Add test to replicate #3757. 2020-06-23 12:48:20 +09:30
ZmnSCPxj jxPCSnmZ
5db69f1b41 lightningd/opening_control.c: Remove 'Try fundchannel_cancel again' error.
Changelog-Changed: `fundchannel_cancel` will now succeed even when executed while a `fundchannel_complete` is ongoing; in that case, it will be considered as cancelling the funding *after* the `fundchannel_complete` succeeds.

Let me introduce the concept of "Sequential Consistency":
All operations on parallel processes form a single total order agreed upon by all processes.

So for example, suppose we have parallel invocations of `fundchannel_complete` and `fundchannel_cancel`:

                          +--[fundchannel_complete]-->
                          |
    --[fundchannel_start]-+
                          |
                          +--[fundchannel_cancel]---->

What "Sequential Consistency" means is that the above parallel operations can be serialized as a single total order as:

    --[fundchannel_start]--[fundchannel_complete]--[fundchannel_cancel]-->

Or:

    --[fundchannel_start]--[fundchannel_cancel]--[fundchannel_complete]-->

In the first case, `fundchannel_complete` succeeds, and the `fundchannel_cancel` invocation also succeeds, sending an `error` to the peer to make them forget the chanel.

In the second case, `fundchannel_cancel` succeeds, and the succeeding `fundchannel_complete` invocation fails, since the funding is already cancelled and there is nothing to complete.

Note that in both cases, `fundchannel_cancel` **always** succeeds.

Unfortunately, prior to this commit, `fundchannel_cancel` could fail with a `Try fundchannel_cancel again` error if the `fundchannel_complete` is ongoing when the `fundchannel_cancel` is initiated.
This violates Sequential Consistency, as there is no single total order that would have caused `fundchannel_cancel` to fail.

This commit is a minimal patch which just reschedules `fundchannel_cancel` to occur after any `fundchannel_complete` that is ongoing.
2020-06-22 03:15:55 +00:00
ZmnSCPxj jxPCSnmZ
e8936f9d23 common/json.c: Check that JSMN result is well-formed.
xref: https://lists.ozlabs.org/pipermail/c-lightning/2020-June/000188.html

Changelog-Fixed: Reject some bad JSON at parsing.
2020-06-19 11:46:18 +09:30
Antoine Poinsot
4302afd9a5 rpc: don't go below feerate_floor when converting vbytes
We passed below the floor when the user specified `1000perkb`.
Matt Whitlock says :

    I was withdrawing with feerate=1000perkb, which should be the minimum-allowed fee rate. Indeed, bitcoin-cli getmempoolinfo reports:

    {
      "loaded": true,
      "size": 15097,
      "bytes": 9207924,
      "usage": 32831760,
      "maxmempool": 64000000,
      "mempoolminfee": 0.00001000,
      "minrelaytxfee": 0.00001000
    }

Changelog-fixed: rpc: The `feerate` parameters now correctly handle the standardness minimum when passed as `perkb`.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Reported-by: Matt Whitlock
2020-06-18 14:07:00 +02:00
Antoine Poinsot
0b99e2cc42 pytest: reproduce issue #3591
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-06-18 14:07:00 +02:00
Antoine Poinsot
3e9fcc43f9 lightningd/jsonrpc: don't assume the jcon to be alive at command execution
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-06-18 14:07:00 +02:00
niftynei
2900da6112 migrations: ignore channels that don't have a peer_id
We erase peer data after the last channel close transaction for that
peer is 100 blocks deep. We were failing to finish the migration because
the peer_id lookup on these was failing.

Now we ignore any channel with a null peer_id.

Fixes #3768
2020-06-17 14:30:41 +02:00
Rusty Russell
02338a6b25 pyln.proto.message: add to_py() operation.
This delivers the message contents in a much friendlier form for
manipulation: in particular, it makes it easy to compare two
messages without having to know all the message type internals.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-16 16:17:52 +02:00
Rusty Russell
ee76504e53 pyln.proto.message: fix handling of ... with subtypes.
This time, with a test!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-16 16:17:52 +02:00
Rusty Russell
2309a7a561 pyln.proto.message: don't leave 'None' in dict for missing fields.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-16 16:17:52 +02:00
niftynei
431463b57a listfunds: also list reserved outputs
Currently 'listfunds' lies, a teensy eeinsy bit, in that it doesn't list
all of the funds in a wallet (it omits reserved wallet UTXOs). This
change makes the reserved outputs visible by listing them in the
'outputs' section along with a new field, 'reserved', which denotes the
UTXO's state

Changelog-Changed: JSON-RPC: `listfunds` 'outputs' now includes reserved outputs, designated as 'reserved' = true
2020-06-16 15:58:58 +02:00
Christian Decker
5bb1fd4205 pyln-proto: Add Makefile targets for releases 2020-06-15 16:14:53 +02:00
Christian Decker
48cee3c3d2 pyln: Bump version to 0.8.2 in order to match with binary distro
Keeping them out of sync seems like a great way to create confusion. Let's
sync them up whenever possible.
2020-06-15 16:14:53 +02:00
Christian Decker
1933cc2523 docs: Fix typo in the pay command docs 2020-06-15 16:14:53 +02:00
Christian Decker
cbc959ef30 pytest: Mark test_funding_push as network-specific
It checks for both exact values, that may differ due to fees, as well as the
unit which varies from network to network.
2020-06-15 16:14:53 +02:00
Rusty Russell
aaefbe2e9e pyln.proto.message: fix handling of missing optional fields.
If they don't exist, that's OK. These will eventually be going away
from the spec, but there are still some in gossip messages for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
fd3ea91b44 pyln.proto.message: expose array types, add set_field for Message class.
Exposing the array types is required for our dummyrunner in the lnprototest suite, since
it wants to be able to generate fake fields.

The set_field is similarly useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
f52065201b pyln.proto.message.*: add type annotations.
Other changes along the way:

1. In a couple of places we passed None as a dummy for for
   `otherfields` where {} is just as good.
2. Turned bytes into hex for errors.
3. Remove nonsensical (unused) get_tlv_by_number() function from MessageNamespace
4. Renamed unrelated-but-overlapping `field_from_csv` and
   `type_from_csv` static methods, since mypy thought they should have
   the same type.
5. Unknown tlv fields are placed in dict as strings, not ints, for
   type simplicity.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
da070e73b2 pyln.proto.message.*: Add Makefile to do mypy checks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
acfeaebb62 pyln.proto.message: allow fields with options to be missing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
ee6c58cbd5 pyln.proto.message: export more.
FieldType lets you make new field types, and split_field helps with
parsing.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
b33dc9847c pyln.proto.message: python-fluency feedback from @darosior
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
8f38bc7230 pyln.proto.message: support adding two namespaces.
They must not have duplicate names!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
85bb93618b new modules: pyln.proto.message.{bolt1,bolt2,bolt4,bolt7}
These are autogenerated, but now they export their own
MessageNamespace, as well as the raw csv.

They also expose their SubtypeTypes, MessageTypes and TlvStreamTypes,
though in theory these could clash (they don't for now, and it'd be
kinda awkward if they did).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
59bb6b90ac pyln: new module pyln.proto.message.bolts
This contains the CSVs for the current bolts (autogenerated).  It's a
separate module because I expect it to be updated alongside the spec.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message.bolts
2020-06-12 15:35:35 +02:00
Rusty Russell
daa707d213 pyln.proto.message: separate fundamental types from other subtypes.
This will be useful for the next patch, which introduces per-bolt
modules.  This makes it easier for them generate variables for each
field type they parse (they don't want to export u16, for example)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
e4b5679f5d message: support option fields.
These are (probably) going away soon, but just tag them for now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
9992a577b4 pyln: add (undocumented) u8 fundamental type.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
42aab89b9f pyln.proto.message: expose fundamental MessageTypes as variables.
Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
47631cc23c pyln.proto.message: use BufferedIOBase instead of bytes for binary ops.
Instead of val_to_bin/val_from_bin which deal with bytes, we implement
read and write which use streams.  This simplifies the API. 

Suggested-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
Rusty Russell
ed4eadc8f3 patch message-export-types.patch 2020-06-12 15:35:35 +02:00
Rusty Russell
eb73a0dd8f pyln: add pyln.proto.message.
This supports infrasructure for creating messages.  In particular, it
can be fed CSV from the spec's `tools/extract-formats.py` and then convert
them all to and from strings and binary formats.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: pyln: new module pyln.proto.message
2020-06-12 15:35:35 +02:00
Rusty Russell
5b4be02ed3 pyln: add Makefile
This runs flake8 and the python tests.  Helps me, at least!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-06-12 15:35:35 +02:00
niftynei
9e0ef45048 psbt: handle 'unsetting' final witness stack
Prior to this commit, passing a NULL stack to `bitcoin_tx_input_set_witness`
unsets the witness stack on the bitcoin_tx's wally_tx but leaves the
final witness on the PSBT unchanged.

at the moment, libwally's `wally_psbt_input_set_final_witness` will blow
up if you attempt to set a NULL witness -- instead we manually remove it
if the passed in stack is NULL. previously we would leave the PSBT's
witness unchanged.
2020-06-11 13:13:13 +02:00
niftynei
58282819a9 psbt: if a transaction has witnesses/scriptSig set, add it to psbt
For any transaction that's got 'finalized' signature data for an input,
we should add this information to the psbt also
2020-06-11 13:13:13 +02:00