Commit Graph

336 Commits

Author SHA1 Message Date
Rusty Russell
06a54606a3 check-includes: allow redundant "config.h"
We should actually be including this (as it may define _GNU_SOURCE
etc) before any system headers.  But where we include <assert.h> we
often didn't, because check-includes would complain that the headers
included it too.

Weaken that check, and include config.h in C files before assert.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:36 +10:30
Christian Decker
46cd5d8923 tools: Allow versions to either have a v prefix or not
Sometimes it's v0.9.3 and sometimes it's 0.9.3, and I always get it
wrong. This is painful when you have to create a signed tag after
committing to one of the two only to find out you chose wrong and have
to redo the tagging.
2021-01-26 15:45:13 +01:00
Antoine Poinsot
48595674fa hsmtool: don't streq() on NULL
This would cause a segfault on the default network parameter for
`dumponchaindescriptors`.

Introduced in 1513a2d07e

Changelog-Fixed: hsmtool: fix a segfault on `dumponchaindescriptors` without network parameter
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-20 10:22:31 +01:00
Rusty Russell
6dab04ebd5 tools/generate-wire.py: include digits in #ifndef idempotent header.
Otherwise check complains that it's '#ifndef LIGHTNING_WIRE_BOLT_WIREGEN_H':

wire/bolt12_wiregen.h seems to be missing the expected include guard:
  #ifndef LIGHTNING_WIRE_BOLT12_WIREGEN_H
  #define LIGHTNING_WIRE_BOLT12_WIREGEN_H
  ...
  #endif /* LIGHTNING_WIRE_BOLT12_WIREGEN_H */

make: *** [Makefile:458: check-includes] Error 1

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-13 14:45:36 +01:00
Antoine Poinsot
45bb1bfa3c hsmd: cleanup encrypted hsm_secret detection
This makes use of the constant defined in the previous commits to more
accurately detect plaintext, encrypted, and invalid seeds. We now error
on invalid seeds.

Changelog-changed: hsmd: we now error at startup on invalid hsm_secret
Changelog-changed: hsmtool: all commands now error on invalid hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
d2a903992c hsmd: regroup hsm_secret decryption logic
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
c6bc22b0f5 hsmd: group hsm_secret encryption
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
a441485a35 lightningd: regroup hsm_secret password input logic
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
917f78a4f8 lightningd: group hsm_secret encryption key derivation
This avoids duplication of both logic and error-prone values, such as
the salt. Grouping all hsm encryption logic into a public API will also
allow us to fuzz it.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
56c223886c lightning: confirm password on hsm_secret encryption
Changelog-changed: lightningd: the `--encrypted-hsm` now asks you to confirm your password when first set
Changelog-changed: hsmtool: the `encrypt` now asks you to confirm your password
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
38204f9b6d hsmtool: use errx() instead of err() everywhere
errx() was printing the confusing errno as well ("Error could not [...] :Success")

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
19afe15968 hsmtool: only write 32 bytes to hsm_secret on generatehsm
Fixes #4302
Changelog-fixed: hsmtool: the `generatehsm` command now generates an appropriately-sized hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot
aa2586a45a pytest: reproduce issue #4302
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot
dd1def335c hsmtool: reuse read_stdin_pass for generatehsm passphrase
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 16:15:40 +01:00
Antoine Poinsot
1513a2d07e hsmtool: input encryption password from stdin
This slightly breaks the API, but still accept the input: we just don't
take it into account anymore.

For `dumponchaindescriptors`, we have to still take the old place of the
`network` parameter into account to not entirely break the API.

Changelog-Added: hsmtool: password must now be entered on stdin. Password passed on the command line are discarded.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 16:15:40 +01:00
niftynei
1ca2c66c54 make: have git submodule update be recursive
libwally added libsecp256k1 as a submodule, so we need to recursively
update the submodules now
2020-12-11 13:20:54 +01:00
Rusty Russell
a11edebb7c utf8: handle UTF-8 arrays.
BOLT 12 introduces this as a new fundamental type, which lets us easily
validate them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-09 16:40:13 +10:30
Rusty Russell
ff8830876d wire/tlvstream: add tlv_make_fields helper to populate ->fields array.
This is vital for calculating merkle trees; I previously used
towire+fromwire to get this!

Requires generation change so we can magic the ARRAY_SIZE var (the C
pre-processor can't uppercase things).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
5c167d16ab tools/generate-wire.py: use helpers.
This was terrible cut & paste.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
dc83e64003 tools/generate-wire: don't use void * pointers for tlv fromwire.
And fix up the one place which got it wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
68c6ae67b3 external/Makefile: fix parallel build with missing secp headers.
Fixes: #4229
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-30 16:10:26 +01:00
Antoine Poinsot
09fb34c6f3 hsmtool: add a 'testnet' option to 'dumponchaindescriptors'
Actually, it's more complex to translate the xpub descriptor to
testnet because of the descriptor checksum.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Antoine Poinsot
9f8e2e0502 tools: add a new 'dumponchaindescriptors' command to hsmtool
This adds a command which outputs the two output descriptors
corresponding to our onchain wallet.

This can be useful for an external service to monitor / send fund to our
wallet.

Further, an "xpriv" version of such descriptors could be used to import
onchain funds on a new wallet.

Changelog-Added: lightning-hsmtool: a new command was added to hsmtool for dumping descriptors of the onchain wallet
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
positiveblue
fa1483a00d hsm_secret generation from a seed-phrase
tools: Add `generatehsm` method to hsmtool to derivate BIP32 seeds from a
mnemonic using the BIP39 standard.

The new method uses libwally for the BIP39 to BIP32 derivation. It also
fails if an hsm_secret file already exists, so we do not overwrite
someone else's wallet without noticing.

It allows the use of passphrases, the ECHO mode in the terminal is
disable for higher security.

It currently supports "en", "es", "fr", "it", "jp", "zhs", "zht".

Changelog-Added: hsmtool: `hsm_secret` generation from a seed-phrase following BIP39.
2020-11-09 17:49:24 -06:00
Rusty Russell
a8177e9013 Makefile: make check-includes check all the non-generated files.
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.

We also include the plugin headers in ALL_C_HEADERS so they get
checked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-22 12:14:34 +10:30
Rusty Russell
5e865ce42b Makefile: unify generated files definition.
We change gen_ to _gen, because filtering on gen_% doesn't work if they're
in subdirectories :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-22 12:14:34 +10:30
niftynei
82c0b48215 wires: towire/fromwire for wally_tx
We're eventually moving away from 'bitcoin_tx
2020-10-20 12:50:31 +10:30
Christian Decker
274540f556 repro: Allow dashes in the version number 2020-09-16 06:27:12 +09:30
Antoine Poinsot
2a9e847055 tools: remove headerversions.o on clean
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-10 10:23:35 +09:30
Christian Decker
fb4f4bff46 repro: Add config for ubuntu:20.04 2020-09-09 20:22:42 +09:30
Rusty Russell
f7de4ac919 Makefile: fix missing dependencies for tools tests.
This means they are included in `make check-source`, which they now
pass (since previous commit).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 16:23:58 +09:30
Rusty Russell
0e805427dc tools/generate-wire.py: strip trailing whitespace on lines, fix bolt quotes.
There's a lot of it, and it means we can't `make check-source` on
these files.

Also bring bolt quotes up-to-date.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 16:23:58 +09:30
Rusty Russell
c34c055d82 Makefile: use completely separate spec-derived files for EXPERIMENTAL_FEATURES
This avoids overwriting the ones in git, and generally makes things neater.

We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 09:42:00 +09:30
Rusty Russell
aa441e3b27 tools/generate-wire.py: fix loop logic for towire_xxx_array
We have to handle singletons which are arrays of variable-length entries:
this needs to be a ptr-to-ptr.

```C
struct blinded_payinfo {
        u32 fee_base_msat;
        u32 fee_proportional_millionths;
        u16 cltv_expiry_delta;
        u8 *features;
};
```

Before:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo *blindedpay;
```

After:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo **blindedpay;
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
897c53ce1c tools/generate-wire.py: fix loop logic for towire_xxx_array
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong.  This happens in the
the bolt13 patch:

```C
struct tlv_offer_tlvs_blindedpath {
        struct pubkey blinding;
        struct onionmsg_path **path;
};
```

Before:
wire/gen_bolt13_tlv.c:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
76813d6f90 tools/generate-wire.py: fix varsize assignment.
Code like this is suspicious:

	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);

In fact, it is a memory leak since we copy and don't free the fromwire
result.  Really, field_0 should be a pointer.

We don't hit this case (yet!) in spec-generated code, but I did for
bolt13.

Here's the difference in gen_test output:

```patch
  diff -ur /tmp/before/test/gen_test.c /tmp/after/test/gen_test.c
  --- /tmp/before/test/gen_test.c	2020-05-07 16:23:31.651611235 +0930
  +++ /tmp/after/test/gen_test.c	2020-05-07 16:20:54.232574482 +0930
  @@ -214,12 +214,12 @@
   static void towire_subtype_varsize_struct(u8 **p, const struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  -	towire_test_features(p, &subtype_varsize_struct->field_0);
  +	towire_test_features(p, subtype_varsize_struct->field_0);
   }
   static void fromwire_subtype_varsize_struct(const u8 **cursor, size_t *plen, struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  - 	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);
  + 	subtype_varsize_struct->field_0 = fromwire_test_features(subtype_varsize_struct, cursor, plen);
   }
   
   /* SUBTYPE: SUBTYPE_VAR_LEN */
  @@ -373,7 +373,7 @@
   
   	ptr = tal_arr(ctx, u8, 0);
   
  -	towire_test_features(&ptr, &r->tlv3->features);
  +	towire_test_features(&ptr, r->tlv3->features);
   
   	towire_amount_msat(&ptr, r->tlv3->amount_msat_1);
   
  @@ -385,7 +385,7 @@
   	struct tlv_test_n1 *r = vrecord;
   
   	r->tlv3 = tal(r, struct tlv_test_n1_tlv3);
  -	r->tlv3->features = *fromwire_test_features(r->tlv3, cursor, plen);
  +	r->tlv3->features = fromwire_test_features(r->tlv3, cursor, plen);
   	r->tlv3->amount_msat_1 = fromwire_amount_msat(cursor, plen);
   	r->tlv3->amount_msat_2 = fromwire_amount_msat(cursor, plen);
   }
  @@ -824,11 +824,11 @@
   
   	towire_test_short_id(&ptr, &r->tlv3->subtype);
   
  -	towire_subtype_var_len(&ptr, &r->tlv3->varlen_subtype);
  +	towire_subtype_var_len(&ptr, r->tlv3->varlen_subtype);
   
  -	towire_subtype_var_assign(&ptr, &r->tlv3->varlen_assigned);
  +	towire_subtype_var_assign(&ptr, r->tlv3->varlen_assigned);
   
  -	towire_subtype_varlen_varsize(&ptr, &r->tlv3->test_sbt_varlen_varsize);
  +	towire_subtype_varlen_varsize(&ptr, r->tlv3->test_sbt_varlen_varsize);
   
   		for (size_t i = 0; i < 2; i++)
   		towire_u32(&ptr, r->tlv3->arr_assign[i]);
  @@ -868,9 +868,9 @@
   
   	r->tlv3 = tal(r, struct tlv_test_n3_tlv3);
   	fromwire_test_short_id(cursor, plen, &r->tlv3->subtype);
  -	r->tlv3->varlen_subtype = *fromwire_subtype_var_len(r->tlv3, cursor, plen);
  -	r->tlv3->varlen_assigned = *fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  -	r->tlv3->test_sbt_varlen_varsize = *fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_subtype = fromwire_subtype_var_len(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_assigned = fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  +	r->tlv3->test_sbt_varlen_varsize = fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
   		for (size_t i = 0; i < 2; i++) {
   		u32 tmp;
   		tmp = fromwire_u32(cursor, plen);
  diff -ur /tmp/before/test/gen_test.h /tmp/after/test/gen_test.h
  --- /tmp/before/test/gen_test.h	2020-05-07 16:23:30.399617108 +0930
  +++ /tmp/after/test/gen_test.h	2020-05-07 16:20:52.912584680 +0930
  @@ -45,7 +45,7 @@
           struct test_short_id field_1;
   };
   struct subtype_varsize_struct {
  -        struct test_features field_0;
  +        struct test_features *field_0;
   };
   struct subtype_var_len {
           struct test_short_id *field_2;
  @@ -60,15 +60,15 @@
           struct test_short_id field3[2];
   };
   struct tlv_test_n1_tlv3 {
  -        struct test_features features;
  +        struct test_features *features;
           struct amount_msat amount_msat_1;
           struct amount_msat amount_msat_2;
   };
   struct tlv_test_n3_tlv3 {
           struct test_short_id subtype;
  -        struct subtype_var_len varlen_subtype;
  -        struct subtype_var_assign varlen_assigned;
  -        struct subtype_varlen_varsize test_sbt_varlen_varsize;
  +        struct subtype_var_len *varlen_subtype;
  +        struct subtype_var_assign *varlen_assigned;
  +        struct subtype_varlen_varsize *test_sbt_varlen_varsize;
           /*  array assigtest_nable */
           u32 arr_assign[2];
           /*  array structs */
  Binary files /tmp/before/test/gen_test.o and /tmp/after/test/gen_test.o differ
  Binary files /tmp/before/test/run-test-wire and /tmp/after/test/run-test-wire differ
  Binary files /tmp/before/test/run-test-wire.o and /tmp/after/test/run-test-wire.o differ
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
390c6470ad configure: we don't need mako any more.
And guard the mako-specific tests with a check instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
b3d5220da5 tools/refresh-submodules.sh: don't exit, but wait if called in parallel.
Otherwise make thinks we're done, and we can get errors.  Include
primitive code if we abort build halfway and leave .refresh-submodules dir.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
1746406e41 Makefile: normalize all the Makefiles
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS.  Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
2835cf8a43 tools/check-bolt: don't trigger ourselves.
We're going to make check-source cover every C file soon.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
8150d28575 Makefile: use generic rules to make spec-derived sources.
Now we use the same Makefile rules for all CSV->C generation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Christian Decker
b71f6e8eaa repro: Skip confirmation of package installation in repro-build.sh 2020-08-30 20:03:42 +02:00
Matt Whitlock
abbc712afb allow building without sqlite3
Changelog-Changed: build: SQLite3 is no longer a hard build requirement. C-Lightning can now be built to support only the PostgreSQL back-end.
2020-08-30 12:44:56 +02:00
Rusty Russell
dc8458d1e2 tools: fix comment in template.
It's (usually) a .csv not an _csv file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
ZmnSCPxj jxPCSnmZ
7f24646139 tools/hsmtool.c: Make password optional for guesstoremote and dumpcommitments commands.
Changelog-Fixed: Actually make the password argument optional for `guesstoremote` and `dumpcommitments` sub-commands, as shown in our documentation and help text.

Reported by `Barno` on #c-lightning IRC.
2020-07-07 13:34:11 +00:00
joe.miyamoto
0b1f8fdbf0 Take LIGHTNINGD_NETWORK env variable in Dockerfile.
Before this, docker image will never detects that
`lightning-rpc` was created if it is running in regtest
or testnet, because the file will be created under
subfolder for each network name, and entrypoint does not
check "lightning-rpc" file in those folders.
By specifying `LIGHTNINGD_NETWORK` environment var
in dockerfile, we can now check correct path.

Changelog-Added: Docker build now includes `LIGHTNINGD_NETWORK` ENV variable which defaults to "bitcoin". An user can override this (e.g. by `-e` option in `docker run`) to run docker container in regtest or testnet or any valid argument to `--network`.
2020-07-07 12:14:42 +02:00
Christian Decker
c0b30ac907 wiregen: Add enums for TLV types so we can call them by their name
Suggested-by: Lisa Neigut <@niftynei>
Signed-off-by: Christian Decker <@cdecker>
2020-07-01 12:19:02 +02:00
ZmnSCPxj jxPCSnmZ
5d720536e2 Makefile: Install tools/hsmtool as lightning-hsmtool.
Changelog-Added: We now install `lightning-hsmtool` for your `hsm_secret` needs.

See: https://github.com/ElementsProject/lightning/issues/3717#issuecomment-644844594

It seems reasonable to add this to the standard install, and to document it properly as well, hopefully we can fill in the documentation better later on.
2020-07-01 09:24:03 +09:30
niftynei
175fcf381a psbt: have wally_tx serialization methods be legible for gen'd code
our code generators expect the serialization name to match the struct
type
2020-06-23 14:49:32 +02:00
niftynei
185fe722be update-mocks: make it a bit easier to tell what step is happening
Adding a small explainer before printing the filename makes it a bit
clearer what's going on when parsing make logs
2020-05-30 15:36:56 +02:00