Commit Graph

512 Commits

Author SHA1 Message Date
Alex Myers
f3934cda50 reckless: use argparse subparsers
A more pythonic approach which should also enable additional help context
for subcommands.
2022-11-08 13:19:36 +01:00
Alex Myers
651c5b6de0 reckless: use config that was explicitly passed to lightningd
Regtest environments commonly use explicit definition of the config
file for lightningd.  This can be queried and utilized by default,
saving redundant definitions between lightning and reckless.
2022-11-08 13:19:36 +01:00
Alex Myers
f18c5e320d reckless: detect pip3 or pip 2022-11-08 13:19:36 +01:00
Alex Myers
83dd431cdc reckless: use the lightning path when invoking lightning-cli
This enables compatibility with startup_regtest.sh among other uses.
The lightning-cli network flag is also set in case there is no config file.
2022-11-08 13:19:36 +01:00
Alex Myers
7e8a889d89 reckless: it turns out the warning is a bit much.
The user should be informed that their config now has a new source, but
but any config files created downstream should be automatically populated.
2022-11-08 13:19:36 +01:00
Alex Myers
b1b280d10b reckless: new tool to manage lightningd plugins
A simple standalone python executable to track plugin repositories, clone
to /tmp, install requirements, test plugin runs, then install and enable in
lightningd and in the config.
Changelog-Added: Reckless - a Core Lightning plugin manager
2022-11-08 13:19:36 +01:00
Rusty Russell
5cf86a1a2e common: update to latest onion message spec.
Mainly, field name changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: Protocol: Support for forwarding blinded payments (as per latest draft)
2022-10-26 11:29:06 +10:30
niftynei
910116c9e6 build-release: configure before submodcheck
We added a submod dep (lowdown) that requires config to run first.
2022-09-22 11:36:10 +02:00
Rusty Russell
2da5244e83 jsonrpc: make error codes an enum.
This allows GDB to print values, but also allows us to use them in
'case' statements.  This wasn't allowed before because they're not
constant terms.

This also made it clear there's a clash between two error codes,
so move one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: JSON-RPC: Error code from bcli plugin changed from 400 to 500.
2022-09-19 10:18:55 +09:30
Rusty Russell
fb433a70f8 doc: escape output types (esp short_channel_id).
We can also remove the listpeers closer hack, which was removed from
the schema already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 14:00:41 +02:00
Rusty Russell
2c3d4e46fc tools/test: fix very confused code.
This broke with COPTFLAGS="-flto -O3", and so I took a look (it
complains more than normal because main isn't there).  We should never
be running update-mocks except on programs expected to compile: in
this case, that's tools/test/run-test-wire.c.

Remove the code which tries to run this, which also means
non-developers won't be running update-mocks!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 14:00:41 +02:00
Rusty Russell
1a0f7ddb0d hsmtool: remove hsm_secret passwords on cmdline support in dumponchaindescriptors.
Changelog-Removed: `hsmtool`: hsm_secret (ignored) on cmdline for dumponchaindescriptors (deprecated in v0.9.3)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-12 09:34:52 +09:30
Rusty Russell
50d1043a91 external: build lowdown if not already found.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-08 00:36:51 +03:00
Rusty Russell
04b59d991a doc: always escape underscores in property names
If there's only a single underscore, lowdown ignores it, but if there are multiple
(see min_final_cltv_expiry) it decides we're trying to highlight part of the word.

Reported-by: @wtogami
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-08 00:36:51 +03:00
Rusty Russell
8f1164365e doc: generate correct markdown from schemas.
You can't start a list without a paragraph separator.

```diff
 --- /tmp/before	2022-07-20 22:02:23.485372596 +0930
 +++ /tmp/after	2022-07-20 22:02:33.745528456 +0930
 @@ -21,12 +21,16 @@
         On startup of the daemon, no autoclean is set up.
  
  RETURN VALUE
 -       On success, an object is returned,  containing:  -  enabled  (boolean):
 -       whether invoice autocleaning is active
 +       On success, an object is returned, containing:
  
 -       If enabled is true: - expired_by (u64): how long an invoice must be ex‐
 -       pired (seconds) before we delete it - cycle_seconds (u64): how long  an
 -       invoice must be expired (seconds) before we delete it
 +       • enabled (boolean): whether invoice autocleaning is active
 +
 +       If enabled is true:
 +
 +       • expired_by (u64): how long an invoice must be expired  (seconds)  be‐
 +         fore we delete it
 +       • cycle_seconds  (u64):  how  long an invoice must be expired (seconds)
 +         before we delete it
  
  AUTHOR
         ZmnSCPxj <ZmnSCPxj@protonmail.com> is mainly responsible.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-08 00:36:51 +03:00
Rusty Russell
3c3f4731bd doc: format markdown correctly.
There are no definition lists in Markdown, and lists get mangled if
they follow immediately: they need a line between them.

So use bullets for options, and use an indent so the text gets in the
line below.

Here's a before-and-after example:

```diff
 --- /tmp/after	2022-07-20 21:55:54.355487769 +0930
 +++ /tmp/after2	2022-07-20 21:58:17.305642576 +0930
 @@ -10,38 +10,71 @@
         lightning-cli sends commands to the lightning daemon.

  OPTIONS
 -       --lightning-dir=DIR Set the directory for the lightning daemon we're talking to; defaults to $HOME/.lightning.
 +       • --lightning-dir=DIR

 -       --conf=PATH Sets configuration file (default: lightning-dir/config ).
 +         Set the directory for the lightning daemon we're talking to; defaults to $HOME/.lightning.

 -       --network=network --mainnet --testnet --signet Sets network explicitly.
 +       • --conf=PATH

 -       --rpc-file=FILE Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
 +         Sets configuration file (default: lightning-dir/config ).

 -       --keywords/-k Use format key=value for parameters in any order
 +       • --network=network

 -       --order/-o Follow strictly the order of parameters for the command
 +       • --mainnet

 -       --json/-J Return result in JSON format (default unless help command, or result contains a format-hint field).
 +       • --testnet

 -       --raw/-R Return raw JSON directly as lightningd replies; this can be faster for large requests.
 +       • --signet

 -       --human-readable/-H Return result in human-readable output.
 +         Sets network explicitly.

 -       --flat/-F  Return  JSON result in flattened one-per-line output, e.g. { "help": [ { "command": "check" } ] } would become
 +       • --rpc-file=FILE
 +
 +         Named pipe to use to talk to lightning daemon: default is lightning-rpc in the lightning directory.
 +
 +       • --keywords/-k
 +
 +         Use format key=value for parameters in any order
 +
 +       • --order/-o
 +
 +         Follow strictly the order of parameters for the command
 +
 +       • --json/-J
 +
 +         Return result in JSON format (default unless help command, or result contains a format-hint field).
 +
 +       • --raw/-R
 +
 +         Return raw JSON directly as lightningd replies; this can be faster for large requests.
 +
 +       • --human-readable/-H
 +
 +         Return result in human-readable output.
 +
 +       • --flat/-F
 +
 +         Return  JSON  result  in  flattened  one-per-line  output,  e.g.  {  "help":  [ { "command": "check" } ] } would become
         help[0].command=check.  This is useful for simple scripts which want to find a  specific  output  field  without  parsing
         JSON.

 -       --notifications/-N=LEVEL  If  LEVEL is 'none', then never print out notifications.  Otherwise, print out notifications of
 -       LEVEL or above (one of io, debug, info (the default), unusual or broken: they are prefixed with # .
 +       • --notifications/-N=LEVEL
 +
 +         If  LEVEL  is 'none', then never print out notifications.  Otherwise, print out notifications of LEVEL or above (one of
 +         io, debug, info (the default), unusual or broken: they are prefixed with # .
 +
 +       • --help/-h
 +
 +         Pretty-print summary of options to standard output and exit.  The format can be changed using -F, -R, -J, -H etc.
 +
 +       • --version/-V

 -       --help/-h Pretty-print summary of options to standard output and exit.  The format can be changed using -F,  -R,  -J,  -H
 -       etc.
 +         Print version number to standard output and exit.

 -       --version/-V Print version number to standard output and exit.
 +       • allow-deprecated-apis=BOOL

 -       allow-deprecated-apis=BOOL Enable deprecated options. It defaults to true, but you should set it to false when testing to
 -       ensure that an upgrade won't break your configuration.
 +         Enable deprecated options. It defaults to true, but you should set it to false when testing to ensure that  an  upgrade
 +         won't break your configuration.

  COMMANDS
         lightning-cli simply uses the JSON RPC interface to talk to lightningd, and prints the results. Thus the commands  avail‐
 @@ -60,13 +93,13 @@
         this is not encouraged.

  EXAMPLES
 -       1.     List commands
 +       1.     List commands:

 -       lightning-cli help
 +              • lightning-cli help

 -       2.     Fund a 10k sat channel using uncomfirmed outputs
 +       2.     Fund a 10k sat channel using uncomfirmed outputs:

 -       lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0
 +              • lightning-cli --keywords fundchannel id=028f...ae7d amount=10000sat minconf=0

  BUGS
         This manpage documents how it should work, not how it does work. The pretty printing of results isn't pretty.
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-08 00:36:51 +03:00
Rusty Russell
8e57bf3796 tools: add md2man.sh tool, using lowdown.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-09-08 00:36:51 +03:00
niftynei
6dfba2468a json-schema: allow 'required' to not be present in if switches 2022-07-28 12:08:18 +09:30
Rusty Russell
8c38302ab8 hsmtool: implement checkhsm.
This gives a nice way to ensure your secret is the correct one.

Also, we don't need to suppress VALGRIND for this test, now the output
races are fixed.

Changelog-Added: `hsmtool`: new command `checkhsm` to check BIP39 passphrase against hsm_secret.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-22 16:57:27 +02:00
Christian Decker
b48ae58b56 repro: Update ubuntu jammy reprobuild 2022-07-21 15:21:12 -05:00
Christian Decker
b8bcc7d13f pytest: Add a new RPC interface to talk to grpc
This allows us to re-use existing tests (assuming the call and fields
are covered by `cln-rpc` and `cln-grpc`) to test the full roundtrip
from test over the grpc interface to the json-rpc interface and back
again.

You can switch to the grpc interface by setting the `CLN_TEST_GRPC`
environment variable to 1, but for now only very few shims are
implemented (due to the non-generated nature of LightningRpc).
2022-07-21 14:19:06 +09:30
Rusty Russell
5979a7778f lightningd: expand exit codes for various failures.
Most unexpected ones are still 1, but there are a few recognizable error codes
worth documenting.

Rename the HSM ones to put ERRCODE_ at the front, since we have non-HSM ones
too now.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-20 19:28:33 +09:30
Rusty Russell
3e672b784d Makefile: use a library archive for CCAN
The linker discards whole files in an archive if it doesn't need them,
so saves a bit of space (and time).  Also allows us to add more niche
things to CCAN (e.g. runes support!) without bloating all the binaries.

We also had many places which depended on $(CCAN_FILES), but that was
already a dependent of $(ALL_PROGRAMS) and $(ALL_TEST_PROGRAMS).

Before:

```
$ size lightningd/lightning*d
   text	   data	    bss	    dec	    hex	filename
2247683	   8696	  39008	2295387	 23065b	lightningd/lightning_channeld
2086607	   7432	  38880	2132919	 208bb7	lightningd/lightning_closingd
2227916	   8056	  39200	2275172	 22b764	lightningd/lightning_connectd
3369236	 119288	  39240	3527764	 35d454	lightningd/lightningd
2183551	   8352	  38880	2230783	 2209ff	lightningd/lightning_dualopend
2196389	   8024	  39136	2243549	 223bdd	lightningd/lightning_gossipd
2086216	   7488	  39264	2132968	 208be8	lightningd/lightning_hsmd
2134396	   8136	  39424	2181956	 214b44	lightningd/lightning_onchaind
2133391	   8352	  38880	2180623	 21460f	lightningd/lightning_openingd
1512168	   2136	  34384	1548688	 17a190	lightningd/lightning_websocketd
```

After:
```
text	   data	    bss	    dec	    hex	filename
2192065	   8488	  38912	2239465	 222be9	lightningd/lightning_channeld
2030957	   7224	  38816	2076997	 1fb145	lightningd/lightning_closingd
2179571	   7968	  39104	2226643	 21f9d3	lightningd/lightning_connectd
3354296	 119288	  39208	3512792	 3599d8	lightningd/lightningd
2127933	   8144	  38816	2174893	 212fad	lightningd/lightning_dualopend
2141699	   7856	  39072	2188627	 216553	lightningd/lightning_gossipd
2024482	   7288	   5240	2037010	 1f1512	lightningd/lightning_hsmd
2072074	   7920	   5400	2085394	 1fd212	lightningd/lightning_onchaind
2077773	   8144	  38816	2124733	 206bbd	lightningd/lightning_openingd
1408958	   1752	    344	1411054	 1587ee	lightningd/lightning_websocketd
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-17 08:51:02 +09:30
adi2011
829fe09c13 doc: Add documentation for new RPCs and a FIXME: in fromschema.py 2022-07-14 12:24:48 -05:00
adi2011
286d6c3165 tools/gen: Always return bool! 2022-07-14 12:24:48 -05:00
adi2011
e42ba8366b common: Add scb_wire for serializing the static_chan_backup 2022-07-14 12:24:48 -05:00
Rusty Russell
32af92145b update-mocks: handle missing deprecated_apis.
This expands update-mocks to be able to handle (simple!) missing
symbols which are not functions.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-09 09:59:52 +09:30
Christian Decker
9039c9c46e docker: Update name from c-lightning to Core-Lightning 2022-05-14 11:43:46 +09:30
Christian Decker
182c900cea repro: Add reprobuild support for ubuntu:22.04 2022-04-28 12:28:48 +09:30
Rusty Russell
f5df69020d v0.11.0.1: final fixes for release build
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-04-25 20:49:44 +09:30
Christian Decker
09ee28cb51 cln-grpc: Do not start unless a grpc-port is specified
For now we don't want to autostart.

Suggested-by: Rusty Russell <@rustyrussell>
2022-03-30 12:15:55 +10:30
Rusty Russell
b19f3a5e7f devtools/decodemsg: don't require --onion for onion tlvs, fail if unknown tlvname.
Generate a table, let decodemsg sort it out.  Do more up-front work in
argparsing too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
83ee68ab06 common/tlvstream: put TLV checking back in the generic function.
Callers were supposed to call "tlv_fields_valid" after fromwire_tlv,
but few did.  Make this the default, and call the underlying function
directly where we want to be more flexible (one place).

This loses the ability to allow misordered fields, or to pass through
*any* even fields.  We restore that for special cases in the next
patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
a770f51d0e tools/generate_wire.py: make functions allocate the TLV.
Requiring the caller to allocate them is ugly, and differs from
other types.

This means we need a context arg if we don't have one already.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
04a152fbcf tools/generate-wire.py: make (and expose) individual TLV print functions.
When we actually put bolt12 fields (.e.g tlv_invoice) in onion messages,
that code will try to call printwire_tlv_invoice(), so expose it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
8f39002715 tools/generate-wire.py: have printwire routines do demarshal.
This works better in general: let printwire_x do the work of figuring
out how to demarshal x.  This is particularly important for TLVs, which
require a call to tlv_x_new() first.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
88de64a580 tools/generate-wire.py: simplify printwire routines, fix ... handling.
We make them return bool, and always use names `cursor` and `plen` in
callers, for simplicity.

Also, `...` means "loop until finished" not "loop this many bytes".

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
12c7b156c5 tools/generate_wire.py: allow generated C files to have includes added.
We previously ignored --include= for these, but onion is about to start
needing bolt12.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Rusty Russell
fa0c29f959 tools/generate_wire.py: tlvs should start with tlv_
No more "towire_offer", but "towire_tlv_offer".

This means we double-up on the unfortunately-named `tlv_payload` inside
the onion, but we should rename that in the spec when we remove
old payloads.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-25 13:55:44 +10:30
Vincenzo Palazzo
53806d1abd cli: make the command line more user friendly.
Also has to fix up tests.

Changelog-Fixed: cli doesn't required anymore to confirm the password if the `hsm_secret` is already encrypted.

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2022-03-24 09:53:38 +10:30
Ken Sedgwick
36466af3eb hsmd: Add fields to hsmd_sign_{,remote_}commitment_tx for validating signers 2022-03-16 12:06:21 +10:30
Christian Decker
f4fda04fb6 misc: Make the spell checker just a tiny bit less trigger happy 2022-03-08 05:00:36 +10:30
Christian Decker
1ef77504b1 misc: Add build targets for the tarball and debian packages
The tarball needs to materialze all submodules, and git needs to be
removed as a build dependency.
2022-02-22 09:43:56 +10:30
Michael Schmoock
7c8b12db0b wiregen: adds note about what <count> can be 2022-02-22 05:45:47 +10:30
Vincenzo Palazzo
43ff949ea7 lightningd: support hsm error code
Suggested-by: Rusty Russell
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Changelog-Changed: Support hsm specific error error code in lightning-cli
2021-12-15 12:24:54 +10:30
Rusty Russell
186b895440 tools/check-includes.sh: make shellcheck happy.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell
4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Vincenzo Palazzo
b933b2fa4d tools: add a tool to formatting schema, and fixed (hack) the Makefile code checking.
This include the following commits:

- review 1/2: move from tab to space, and remove the exp. prop from doc;
- review 2/2: remove experimental features;


Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2021-11-28 10:55:48 +10:30
Christian Decker
39022028e9 db: Ensure sqlite3 library minimum version and major version match
We were really strict with the version check, requiring an exact
match. We now check that we are running at least with the version we
were compiled with (distro upgrades continue to work, and repro builds
are built off of an unupdated installation matching this minimum
requirement), and a major version match (since major versions can and
will introduce breaking changes).

Changelog-Fixed: sqlite3: Relaxed the version match requirements to be at least a minimum version and a major version match
2021-10-12 09:15:48 +10:30
W. J. van der Laan
dd93f5dd29 Add missing headers for compiling on FreeBSD 13
- `netinet/in.h`: struct sockaddr_in, struct sockaddr_in6
- `unistd.h`: close()
2021-09-24 12:40:09 +09:30
Rusty Russell
24536c5561 common/autodata: use instead of ccan/autodata
This means it needs to be linked ~everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-21 18:04:43 +02:00
Rusty Russell
7401b26824 cleanup: remove unneeded includes in C files.
Before:
 Ten builds, laptop -j5, no ccache:

```
real	0m36.686000-38.956000(38.608+/-0.65)s
user	2m32.864000-42.253000(40.7545+/-2.7)s
sys	0m16.618000-18.316000(17.8531+/-0.48)s
```

 Ten builds, laptop -j5, ccache (warm):

```
real	0m8.212000-8.577000(8.39989+/-0.13)s
user	0m12.731000-13.212000(12.9751+/-0.17)s
sys	0m3.697000-3.902000(3.83722+/-0.064)s
```

After:
 Ten builds, laptop -j5, no ccache: 8% faster

```
real	0m33.802000-35.773000(35.468+/-0.54)s
user	2m19.073000-27.754000(26.2542+/-2.3)s
sys	0m15.784000-17.173000(16.7165+/-0.37)s
```

 Ten builds, laptop -j5, ccache (warm): 1% faster

```
real	0m8.200000-8.485000(8.30138+/-0.097)s
user	0m12.485000-13.100000(12.7344+/-0.19)s
sys	0m3.702000-3.889000(3.78787+/-0.056)s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
ea30c34d82 cleanup: remove unneeded includes in header files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
00a0d09340 tools/check-includes.sh: test that c files include their .h files.
This is best-practice (to ensure prototypes match up), but there were a
few places we didn't (at least, directly).  Make it a requirement,
either of form "foo.h" or <dir/foo.h>.

The noise is the change to our print templates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
183fe107e8 lightningd: use channel_type, pass to-and-from channeld.
Instead of explicit option_static_remotekey and option_anchor_outputs flags.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell
cb22015b2a common/channel_type: wrapper for generated 'struct channel_type'.
We make it a first-class citizen internally, even though we won't use
it over the wire (at least, non-experimental builds).  This scheme
follows the latest draft, in which features are flagged compulsory.

We also add several helper functions.

Since uses the *even* bits (as per latest spec), not the *odd* bits,
we have some other fixups.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell
1b8551923d tools: handle generating print templates for CSV without tlvs, messages
We want to use this to handle the simple description for channel_type.

It also needs to handle variable-size types (just like subtypes).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Dustin Dettmer
6f2e27c768 Shellcheck cleanup
Updates from output of latest version of shellcheck as per SC2268

“Avoid x-prefix in comparisons as it no longer serves a purpose”
https://www.shellcheck.net/wiki/SC2268
2021-09-05 18:55:53 +02:00
Rusty Russell
6d1897b796 doc: fix GH/readthedocs rendering of manual pages.
They render the comment as if it's in the list.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-05 15:02:17 +02:00
Rusty Russell
225fec065a Build race fixes: create ccan/config.h atomically, make tools/test/test_gen.c depend on it
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-08-14 12:43:54 +09:30
Rusty Russell
ceb40dea38 lightningd: don't turn zero-length tlv fields into NULL.
Fixes: #4667
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
niftynei
265f960cfe liquidity-ads: persist channel blockheight states to disk
Adds new tables to database, backfills, basically copies the fee_rates
state machine for channeld.
2021-07-20 13:28:38 -04:00
LightningHelper
dbb7f31d39 added minimum word list needed
Not every user know the minimum word list that is needed.
2021-07-19 12:47:11 +09:30
Christian Decker
b4ead97517 tlv: Allow passing some extra types to accept when parsing the stream 2021-06-26 10:55:13 +09:30
Nalin Bhardwaj
0fccfc0a9a tools/check-spelling: Fix grep error on macOS
Changelog-Fixed: Fixes `make full-check` errors on macOS
2021-06-25 10:16:41 +09:30
Nalin Bhardwaj
bbaca4bb5c tools/mockup: Fix macOS bash 3 syntax error
Changelog-Fixed: Fixes `make` with `--enable-developer` option on macOS.
2021-06-25 10:16:41 +09:30
Rusty Russell
6e636a835f tools/fromschema.py: handle deprecated null field, don't create empty lists.
1. listpeers has a deprecated `"closer": null`, which we need
   to handle in the schema, while trying not to damage our
   documentation too much.

2. Don't print a condition if there are no fields to print.

3. Allow a special "untyped" marker for multifundchannel which returns
   arbitrary JSON in a field.

4. Allow a single field return (for 'stop').

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-25 09:49:33 +09:30
Antoine Poinsot
526e6026f5 hsmtool: this is 'success', not 'succes'
I tried to wait_for_log() on "successfully" without success :/

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-06-18 10:40:07 +09:30
Rusty Russell
38fad0f3e4 ccan: update to get RETURNS_NONNULL macro.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-16 10:29:17 +09:30
Rusty Russell
4cc3eaf79c common/bech32: update to bech32m reference.
I did this by copying the updated bech32 code, and then re-patching in
our minor changes:

1. Headers modded (we need size_t)
2. Explicit length for bech32_encode/decode (not 90).
3. Exposing and bech32_ prefix for convert_bits, charset, charset_rev.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-11 13:54:20 +09:30
Rusty Russell
2fea448498 gen/impl_template: fix generation of singleton varsize elements.
And as Lisa requested, add testcases.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-06-04 16:13:08 +09:30
Rusty Russell
2c9eaed294 tools/fromschema.py: tool to replace start/end markers in markdown with schema.
It can also be run standalone for debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-27 20:28:49 +09:30
Rusty Russell
2e3e4a50ec update-mocks: prefer to find function defs from local includes.
This matters for:
	- common/wallet.h vs wallet/wallet.h
	- common/gossip_store.h vs gossipd/gossip_store.h
	- common/json.h vs lightningd/json.h
	- common/ping.h vs lightningd/ping.h

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-22 17:53:04 +09:30
Rusty Russell
25b5e1e099 update-mocks: make sure we cover all test programs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-22 17:53:04 +09:30
Rusty Russell
cc6f1fd120 update-mocks: don't try to regenerate files.
If you do update-mocks in a dirty tree, the recursive make that it
uses will try to rebuild things!  Suppress that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-05-22 17:53:04 +09:30
fiatjaf
0d68febd8d reorganize .gitignore entries across subdirs. 2021-05-18 09:43:50 +09:30
Rusty Russell
06a54606a3 check-includes: allow redundant "config.h"
We should actually be including this (as it may define _GNU_SOURCE
etc) before any system headers.  But where we include <assert.h> we
often didn't, because check-includes would complain that the headers
included it too.

Weaken that check, and include config.h in C files before assert.h.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-02-04 12:02:36 +10:30
Christian Decker
46cd5d8923 tools: Allow versions to either have a v prefix or not
Sometimes it's v0.9.3 and sometimes it's 0.9.3, and I always get it
wrong. This is painful when you have to create a signed tag after
committing to one of the two only to find out you chose wrong and have
to redo the tagging.
2021-01-26 15:45:13 +01:00
Antoine Poinsot
48595674fa hsmtool: don't streq() on NULL
This would cause a segfault on the default network parameter for
`dumponchaindescriptors`.

Introduced in 1513a2d07e

Changelog-Fixed: hsmtool: fix a segfault on `dumponchaindescriptors` without network parameter
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-20 10:22:31 +01:00
Rusty Russell
6dab04ebd5 tools/generate-wire.py: include digits in #ifndef idempotent header.
Otherwise check complains that it's '#ifndef LIGHTNING_WIRE_BOLT_WIREGEN_H':

wire/bolt12_wiregen.h seems to be missing the expected include guard:
  #ifndef LIGHTNING_WIRE_BOLT12_WIREGEN_H
  #define LIGHTNING_WIRE_BOLT12_WIREGEN_H
  ...
  #endif /* LIGHTNING_WIRE_BOLT12_WIREGEN_H */

make: *** [Makefile:458: check-includes] Error 1

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-01-13 14:45:36 +01:00
Antoine Poinsot
45bb1bfa3c hsmd: cleanup encrypted hsm_secret detection
This makes use of the constant defined in the previous commits to more
accurately detect plaintext, encrypted, and invalid seeds. We now error
on invalid seeds.

Changelog-changed: hsmd: we now error at startup on invalid hsm_secret
Changelog-changed: hsmtool: all commands now error on invalid hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
d2a903992c hsmd: regroup hsm_secret decryption logic
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
c6bc22b0f5 hsmd: group hsm_secret encryption
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
a441485a35 lightningd: regroup hsm_secret password input logic
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
917f78a4f8 lightningd: group hsm_secret encryption key derivation
This avoids duplication of both logic and error-prone values, such as
the salt. Grouping all hsm encryption logic into a public API will also
allow us to fuzz it.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
56c223886c lightning: confirm password on hsm_secret encryption
Changelog-changed: lightningd: the `--encrypted-hsm` now asks you to confirm your password when first set
Changelog-changed: hsmtool: the `encrypt` now asks you to confirm your password
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
38204f9b6d hsmtool: use errx() instead of err() everywhere
errx() was printing the confusing errno as well ("Error could not [...] :Success")

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-06 13:50:01 +01:00
Antoine Poinsot
19afe15968 hsmtool: only write 32 bytes to hsm_secret on generatehsm
Fixes #4302
Changelog-fixed: hsmtool: the `generatehsm` command now generates an appropriately-sized hsm_secret
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot
aa2586a45a pytest: reproduce issue #4302
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 19:02:28 +01:00
Antoine Poinsot
dd1def335c hsmtool: reuse read_stdin_pass for generatehsm passphrase
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 16:15:40 +01:00
Antoine Poinsot
1513a2d07e hsmtool: input encryption password from stdin
This slightly breaks the API, but still accept the input: we just don't
take it into account anymore.

For `dumponchaindescriptors`, we have to still take the old place of the
`network` parameter into account to not entirely break the API.

Changelog-Added: hsmtool: password must now be entered on stdin. Password passed on the command line are discarded.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2021-01-04 16:15:40 +01:00
niftynei
1ca2c66c54 make: have git submodule update be recursive
libwally added libsecp256k1 as a submodule, so we need to recursively
update the submodules now
2020-12-11 13:20:54 +01:00
Rusty Russell
a11edebb7c utf8: handle UTF-8 arrays.
BOLT 12 introduces this as a new fundamental type, which lets us easily
validate them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-09 16:40:13 +10:30
Rusty Russell
ff8830876d wire/tlvstream: add tlv_make_fields helper to populate ->fields array.
This is vital for calculating merkle trees; I previously used
towire+fromwire to get this!

Requires generation change so we can magic the ARRAY_SIZE var (the C
pre-processor can't uppercase things).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
5c167d16ab tools/generate-wire.py: use helpers.
This was terrible cut & paste.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
dc83e64003 tools/generate-wire: don't use void * pointers for tlv fromwire.
And fix up the one place which got it wrong.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-12-04 20:16:54 -06:00
Rusty Russell
68c6ae67b3 external/Makefile: fix parallel build with missing secp headers.
Fixes: #4229
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-11-30 16:10:26 +01:00
Antoine Poinsot
09fb34c6f3 hsmtool: add a 'testnet' option to 'dumponchaindescriptors'
Actually, it's more complex to translate the xpub descriptor to
testnet because of the descriptor checksum.

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
Antoine Poinsot
9f8e2e0502 tools: add a new 'dumponchaindescriptors' command to hsmtool
This adds a command which outputs the two output descriptors
corresponding to our onchain wallet.

This can be useful for an external service to monitor / send fund to our
wallet.

Further, an "xpriv" version of such descriptors could be used to import
onchain funds on a new wallet.

Changelog-Added: lightning-hsmtool: a new command was added to hsmtool for dumping descriptors of the onchain wallet
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-11-10 10:34:13 -06:00
positiveblue
fa1483a00d hsm_secret generation from a seed-phrase
tools: Add `generatehsm` method to hsmtool to derivate BIP32 seeds from a
mnemonic using the BIP39 standard.

The new method uses libwally for the BIP39 to BIP32 derivation. It also
fails if an hsm_secret file already exists, so we do not overwrite
someone else's wallet without noticing.

It allows the use of passphrases, the ECHO mode in the terminal is
disable for higher security.

It currently supports "en", "es", "fr", "it", "jp", "zhs", "zht".

Changelog-Added: hsmtool: `hsm_secret` generation from a seed-phrase following BIP39.
2020-11-09 17:49:24 -06:00
Rusty Russell
a8177e9013 Makefile: make check-includes check all the non-generated files.
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.

We also include the plugin headers in ALL_C_HEADERS so they get
checked.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-22 12:14:34 +10:30
Rusty Russell
5e865ce42b Makefile: unify generated files definition.
We change gen_ to _gen, because filtering on gen_% doesn't work if they're
in subdirectories :(

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-10-22 12:14:34 +10:30
niftynei
82c0b48215 wires: towire/fromwire for wally_tx
We're eventually moving away from 'bitcoin_tx
2020-10-20 12:50:31 +10:30
Christian Decker
274540f556 repro: Allow dashes in the version number 2020-09-16 06:27:12 +09:30
Antoine Poinsot
2a9e847055 tools: remove headerversions.o on clean
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
2020-09-10 10:23:35 +09:30
Christian Decker
fb4f4bff46 repro: Add config for ubuntu:20.04 2020-09-09 20:22:42 +09:30
Rusty Russell
f7de4ac919 Makefile: fix missing dependencies for tools tests.
This means they are included in `make check-source`, which they now
pass (since previous commit).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 16:23:58 +09:30
Rusty Russell
0e805427dc tools/generate-wire.py: strip trailing whitespace on lines, fix bolt quotes.
There's a lot of it, and it means we can't `make check-source` on
these files.

Also bring bolt quotes up-to-date.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-09 16:23:58 +09:30
Rusty Russell
c34c055d82 Makefile: use completely separate spec-derived files for EXPERIMENTAL_FEATURES
This avoids overwriting the ones in git, and generally makes things neater.

We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 09:42:00 +09:30
Rusty Russell
aa441e3b27 tools/generate-wire.py: fix loop logic for towire_xxx_array
We have to handle singletons which are arrays of variable-length entries:
this needs to be a ptr-to-ptr.

```C
struct blinded_payinfo {
        u32 fee_base_msat;
        u32 fee_proportional_millionths;
        u16 cltv_expiry_delta;
        u8 *features;
};
```

Before:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo *blindedpay;
```

After:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo **blindedpay;
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
897c53ce1c tools/generate-wire.py: fix loop logic for towire_xxx_array
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong.  This happens in the
the bolt13 patch:

```C
struct tlv_offer_tlvs_blindedpath {
        struct pubkey blinding;
        struct onionmsg_path **path;
};
```

Before:
wire/gen_bolt13_tlv.c:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
76813d6f90 tools/generate-wire.py: fix varsize assignment.
Code like this is suspicious:

	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);

In fact, it is a memory leak since we copy and don't free the fromwire
result.  Really, field_0 should be a pointer.

We don't hit this case (yet!) in spec-generated code, but I did for
bolt13.

Here's the difference in gen_test output:

```patch
  diff -ur /tmp/before/test/gen_test.c /tmp/after/test/gen_test.c
  --- /tmp/before/test/gen_test.c	2020-05-07 16:23:31.651611235 +0930
  +++ /tmp/after/test/gen_test.c	2020-05-07 16:20:54.232574482 +0930
  @@ -214,12 +214,12 @@
   static void towire_subtype_varsize_struct(u8 **p, const struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  -	towire_test_features(p, &subtype_varsize_struct->field_0);
  +	towire_test_features(p, subtype_varsize_struct->field_0);
   }
   static void fromwire_subtype_varsize_struct(const u8 **cursor, size_t *plen, struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  - 	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);
  + 	subtype_varsize_struct->field_0 = fromwire_test_features(subtype_varsize_struct, cursor, plen);
   }
   
   /* SUBTYPE: SUBTYPE_VAR_LEN */
  @@ -373,7 +373,7 @@
   
   	ptr = tal_arr(ctx, u8, 0);
   
  -	towire_test_features(&ptr, &r->tlv3->features);
  +	towire_test_features(&ptr, r->tlv3->features);
   
   	towire_amount_msat(&ptr, r->tlv3->amount_msat_1);
   
  @@ -385,7 +385,7 @@
   	struct tlv_test_n1 *r = vrecord;
   
   	r->tlv3 = tal(r, struct tlv_test_n1_tlv3);
  -	r->tlv3->features = *fromwire_test_features(r->tlv3, cursor, plen);
  +	r->tlv3->features = fromwire_test_features(r->tlv3, cursor, plen);
   	r->tlv3->amount_msat_1 = fromwire_amount_msat(cursor, plen);
   	r->tlv3->amount_msat_2 = fromwire_amount_msat(cursor, plen);
   }
  @@ -824,11 +824,11 @@
   
   	towire_test_short_id(&ptr, &r->tlv3->subtype);
   
  -	towire_subtype_var_len(&ptr, &r->tlv3->varlen_subtype);
  +	towire_subtype_var_len(&ptr, r->tlv3->varlen_subtype);
   
  -	towire_subtype_var_assign(&ptr, &r->tlv3->varlen_assigned);
  +	towire_subtype_var_assign(&ptr, r->tlv3->varlen_assigned);
   
  -	towire_subtype_varlen_varsize(&ptr, &r->tlv3->test_sbt_varlen_varsize);
  +	towire_subtype_varlen_varsize(&ptr, r->tlv3->test_sbt_varlen_varsize);
   
   		for (size_t i = 0; i < 2; i++)
   		towire_u32(&ptr, r->tlv3->arr_assign[i]);
  @@ -868,9 +868,9 @@
   
   	r->tlv3 = tal(r, struct tlv_test_n3_tlv3);
   	fromwire_test_short_id(cursor, plen, &r->tlv3->subtype);
  -	r->tlv3->varlen_subtype = *fromwire_subtype_var_len(r->tlv3, cursor, plen);
  -	r->tlv3->varlen_assigned = *fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  -	r->tlv3->test_sbt_varlen_varsize = *fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_subtype = fromwire_subtype_var_len(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_assigned = fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  +	r->tlv3->test_sbt_varlen_varsize = fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
   		for (size_t i = 0; i < 2; i++) {
   		u32 tmp;
   		tmp = fromwire_u32(cursor, plen);
  diff -ur /tmp/before/test/gen_test.h /tmp/after/test/gen_test.h
  --- /tmp/before/test/gen_test.h	2020-05-07 16:23:30.399617108 +0930
  +++ /tmp/after/test/gen_test.h	2020-05-07 16:20:52.912584680 +0930
  @@ -45,7 +45,7 @@
           struct test_short_id field_1;
   };
   struct subtype_varsize_struct {
  -        struct test_features field_0;
  +        struct test_features *field_0;
   };
   struct subtype_var_len {
           struct test_short_id *field_2;
  @@ -60,15 +60,15 @@
           struct test_short_id field3[2];
   };
   struct tlv_test_n1_tlv3 {
  -        struct test_features features;
  +        struct test_features *features;
           struct amount_msat amount_msat_1;
           struct amount_msat amount_msat_2;
   };
   struct tlv_test_n3_tlv3 {
           struct test_short_id subtype;
  -        struct subtype_var_len varlen_subtype;
  -        struct subtype_var_assign varlen_assigned;
  -        struct subtype_varlen_varsize test_sbt_varlen_varsize;
  +        struct subtype_var_len *varlen_subtype;
  +        struct subtype_var_assign *varlen_assigned;
  +        struct subtype_varlen_varsize *test_sbt_varlen_varsize;
           /*  array assigtest_nable */
           u32 arr_assign[2];
           /*  array structs */
  Binary files /tmp/before/test/gen_test.o and /tmp/after/test/gen_test.o differ
  Binary files /tmp/before/test/run-test-wire and /tmp/after/test/run-test-wire differ
  Binary files /tmp/before/test/run-test-wire.o and /tmp/after/test/run-test-wire.o differ
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
390c6470ad configure: we don't need mako any more.
And guard the mako-specific tests with a check instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
b3d5220da5 tools/refresh-submodules.sh: don't exit, but wait if called in parallel.
Otherwise make thinks we're done, and we can get errors.  Include
primitive code if we abort build halfway and leave .refresh-submodules dir.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
1746406e41 Makefile: normalize all the Makefiles
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS.  Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
2835cf8a43 tools/check-bolt: don't trigger ourselves.
We're going to make check-source cover every C file soon.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
8150d28575 Makefile: use generic rules to make spec-derived sources.
Now we use the same Makefile rules for all CSV->C generation.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Christian Decker
b71f6e8eaa repro: Skip confirmation of package installation in repro-build.sh 2020-08-30 20:03:42 +02:00
Matt Whitlock
abbc712afb allow building without sqlite3
Changelog-Changed: build: SQLite3 is no longer a hard build requirement. C-Lightning can now be built to support only the PostgreSQL back-end.
2020-08-30 12:44:56 +02:00
Rusty Russell
dc8458d1e2 tools: fix comment in template.
It's (usually) a .csv not an _csv file.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-25 12:53:13 +09:30
ZmnSCPxj jxPCSnmZ
7f24646139 tools/hsmtool.c: Make password optional for guesstoremote and dumpcommitments commands.
Changelog-Fixed: Actually make the password argument optional for `guesstoremote` and `dumpcommitments` sub-commands, as shown in our documentation and help text.

Reported by `Barno` on #c-lightning IRC.
2020-07-07 13:34:11 +00:00
joe.miyamoto
0b1f8fdbf0 Take LIGHTNINGD_NETWORK env variable in Dockerfile.
Before this, docker image will never detects that
`lightning-rpc` was created if it is running in regtest
or testnet, because the file will be created under
subfolder for each network name, and entrypoint does not
check "lightning-rpc" file in those folders.
By specifying `LIGHTNINGD_NETWORK` environment var
in dockerfile, we can now check correct path.

Changelog-Added: Docker build now includes `LIGHTNINGD_NETWORK` ENV variable which defaults to "bitcoin". An user can override this (e.g. by `-e` option in `docker run`) to run docker container in regtest or testnet or any valid argument to `--network`.
2020-07-07 12:14:42 +02:00
Christian Decker
c0b30ac907 wiregen: Add enums for TLV types so we can call them by their name
Suggested-by: Lisa Neigut <@niftynei>
Signed-off-by: Christian Decker <@cdecker>
2020-07-01 12:19:02 +02:00
ZmnSCPxj jxPCSnmZ
5d720536e2 Makefile: Install tools/hsmtool as lightning-hsmtool.
Changelog-Added: We now install `lightning-hsmtool` for your `hsm_secret` needs.

See: https://github.com/ElementsProject/lightning/issues/3717#issuecomment-644844594

It seems reasonable to add this to the standard install, and to document it properly as well, hopefully we can fill in the documentation better later on.
2020-07-01 09:24:03 +09:30
niftynei
175fcf381a psbt: have wally_tx serialization methods be legible for gen'd code
our code generators expect the serialization name to match the struct
type
2020-06-23 14:49:32 +02:00
niftynei
185fe722be update-mocks: make it a bit easier to tell what step is happening
Adding a small explainer before printing the filename makes it a bit
clearer what's going on when parsing make logs
2020-05-30 15:36:56 +02:00
niftynei
76c57595c3 tools-make: add mock parser for clang ld output
According to #3226, it looks like clang's LD error format has changed.
This patch adds the new format so we can parse the mocks successfully.

Apple clang version 11.0.3 (clang-1103.0.32.29)

```
checking for ANSI C header files... Undefined symbols for architecture x86_64:
  "_fromwire_amount_msat", referenced from:
      _fromwire_tlv_test_n1_tlv3 in ccj4zKdV.o
  "_fromwire_bool", referenced from:
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_test_enum", referenced from:
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_tlvs", referenced from:
      _fromwire_test_tlv1 in ccj4zKdV.o
      _fromwire_test_tlv2 in ccj4zKdV.o
      _fromwire_test_tlv3 in ccj4zKdV.o
  "_fromwire_tu32", referenced from:
      _fromwire_tlv_test_n2_tlv2 in ccj4zKdV.o
  "_fromwire_tu64", referenced from:
      _fromwire_tlv_test_n1_tlv1 in ccj4zKdV.o
      _fromwire_tlv_test_n2_tlv1 in ccj4zKdV.o
  "_fromwire_u16", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_arrays in ccj4zKdV.o
      _fromwire_tlv_test_n1_tlv4 in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_tlv1 in ccj4zKdV.o
      ...
  "_fromwire_u32", referenced from:
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_u64", referenced from:
      _fromwire_test_short_id in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_fromwire_u8", referenced from:
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_var_len in ccj4zKdV.o
      _fromwire_subtype_varlen_varsize in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_fromwire_u8_array", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_arrays in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
      ...
```

Changelog-Fixed: Build for macOS Catalina / Apple clang v11.0.3 fixed
2020-05-30 15:36:56 +02:00
niftynei
3640befc10 tools-make: also be quiet if --quiet is flagged
Missed a update-mocks.sh call
2020-05-30 15:36:56 +02:00
Rusty Russell
dafaf854c5 bitcoin/tx_parts: infrastructure for partial bitcoin txs.
`struct tx_parts` is just a txid and a bunch of inputs and outputs,
some of which may be NULL.

This is both a nod towards a future where we (or our peer) can combine
HTLCs or (in an eltoo world) commitments, although for the moment all
our tx_parts will be complete.

It also matches our plan to split `bitcoin_tx` into two types: this
`struct tx_parts` where we don't know input amounts etc, and `psbt`
where we do.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-27 10:12:03 +09:30
niftynei
bf4cac7fb8 tx: strip out witscript
now that witness script data is saved into the tx/psbt which is
serialized across the wire, there's no reason to use witscript to do
this. good bye witscript!
2020-05-21 18:45:07 +09:30
niftynei
66f59659a7 setup: add setup to make checks
- we've moved tmpctx management to setup.c from daemon.c, so we update
the `check-tmpctx`
- `common_setup(char *)` is now a valid analog for `setup_locale`, so we
check for either in check-setup_locale
2020-05-19 13:35:42 +09:30
Rusty Russell
cfb320c972 wire: move remaining bitcoin functions out to bitcoin/ files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-18 14:51:12 +02:00
Rusty Russell
197d1bcef2 wire: move towire/fromwire_short_channel_id out to bitcoin/short_channel_id.c
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-18 14:51:12 +02:00
Rusty Russell
ebb7daed49 tools/generate-wire.py: don't prettify headers.
The formatting makes it harder for update-mocks, eg:

    /* Generated stub for fmt_wireaddr_without_port */
    char *fmt_wireaddr_without_port(const tal_t *ctx UNNEEDED, const struct wireaddr *a UNNEEDED)
    { fprintf(stderr, "fmt_wireaddr_without_port called!\n"); abort(); }
    /* Could not find declaration for fromwire_onionmsg_path */
    /* Generated stub for json_add_member */

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Rusty Russell
0512e1a33e tools/generate-wire.py: add --include argument for putting #includes in spec-generated files.
We need this for bolt13.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Rusty Russell
ae7485d2ac tools/generate-wire.py: don't define empty enums.
For bolt 13, we have no message types, just a TLV.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-07 08:44:58 +09:30
Rusty Russell
b0c9059602 tools/generate-wire: no more lonely messages!
When we have only a single member in a TLV (e.g. an optional u64),
wrapping it in a struct is awkward.  This changes it to directly
access those fields.

This is not only more elegant (60 fewer lines), it would also be
more cache friendly.  That's right: cache hot singles!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-05-06 14:56:09 -05:00
lisa neigut
251bae55d7 build: Update build-release.sh to remove i386 build
fixup! docker: Remove Dockerfile for i386 builder
2020-04-19 08:07:47 +09:30
lisa neigut
9663d110d1 build: make non-parallel
There's a race condition with loading the submodules that's causes a
build failure on my machine, since the libwally 'includes' aren't on
disk yet when the gcc build step starts.
2020-04-19 08:07:47 +09:30
Rusty Russell
490a819402 lightningd: add blinding and enctlv field to struct route_hop.
This will be used when we want to specify these in a route.  But for now, they
only alter gossipd, which always sets them to NULL.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-14 12:51:18 +09:30
Rusty Russell
11c21f97a6 devtools/decodemsg: don't crash if we have multiple tlv options.
We call tal_count(msg) after we've moved msg, and that causes an abort.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-07 12:48:13 -05:00
Vasil Dimov
3ce0552dd4 build: use locale-independent sort for mocks
Use `LC_ALL=C sort` instead of `sort` so that mocks get sorted in
the same way on all developers' environments.

Re-record the result of `make update-mocks`.

Changelog-None
2020-04-07 13:52:48 +09:30
Rusty Russell
d9fc99ea39 channeld: simplify loading of pre-existing HTLCs.
We currently abuse the added_htlc and failed_htlc messages to tell channeld
about existing htlcs when it restarts.  It's clearer to have an explicit
'existing_htlc' type which contains all the information for this case.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-04 16:08:49 +10:30
Rusty Russell
dd690553b8 channeld: handle onion messages.
We do most of the decoding here, and just hand the results to lightningd.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-02 14:32:38 +10:30
Rusty Russell
f76ab93df6 EXPERMENTAL_FEATURES: Import onion message types.
This tracks https://github.com/lightningnetwork/lightning-rfc/pull/759

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-04-02 14:32:38 +10:30
Rusty Russell
c95e58ad4b subdaemons: initialize feature routines with explicit feature_set.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-03-31 13:36:02 +02:00
Vasil Dimov
89ceb273f5 wire: remove towire_double()
Before this patch we used to send `double`s over the wire by just
copying them. This is not portable because the internal represenation
of a `double` is implementation specific.

Instead of this, multiply any floating-point numbers that come from
the outside (e.g. JSONs) by 1 million and round them to integers when
handling them.

* Introduce a new param_millionths() that expects a floating-point
  number and returns it multipled by 1000000 as an integer.

* Replace param_double() and param_percent() with param_millionths()

* Previously the riskfactor would be allowed to be negative, which must
  have been unintentional. This patch changes that to require a
  non-negative number.

Changelog-None
2020-02-27 09:07:04 +10:30
Ken Sedgwick
5c8f881a75
hsmd: Added fields to hsm_sign_remote_commitment_tx to allow complete validation.
Changelog-Added: hsmd: Added fields to hsm_sign_remote_commitment_tx to allow complete validation by signing daemon.
2020-02-04 10:40:43 +10:30
Vasil Dimov
0e3fb5e590 build: use [[:space:]] instead of \s in regex
The former is defined by IEEE Std 1003.2 ("POSIX.2"), the latter does
not work on FreeBSD.

Changelog-None
2020-02-03 15:38:11 +00:00
Vasil Dimov
55173a56b7 Use dedicated type for error codes
Before this patch we used `int` for error codes. The problem with
`int` is that we try to pass it to/from wire and the size of `int` is
not defined by the standard. So a sender with 4-byte `int` would write
4 bytes to the wire and a receiver with 2-byte `int` (for example) would
read just 2 bytes from the wire.

To resolve this:

* Introduce an error code type with a known size:
  `typedef s32 errcode_t`.

* Change all error code macros to constants of type `errcode_t`.
  Constants also play better with gdb - it would visualize the name of
  the constant instead of the numeric value.

* Change all functions that take error codes to take the new type
  `errcode_t` instead of `int`.

* Introduce towire / fromwire functions to send / receive the newly added
  type `errcode_t` and use it instead of `towire_int()`.

In addition:

* Remove the now unneeded `towire_int()`.

* Replace a hardcoded error code `-2` with a new constant
  `INVOICE_EXPIRED_DURING_WAIT` (903).

Changelog-Changed: The waitinvoice command would now return error code 903 to designate that the invoice expired during wait, instead of the previous -2
2020-01-31 06:02:47 +00:00