This slightly breaks the API, but still accept the input: we just don't
take it into account anymore.
For `dumponchaindescriptors`, we have to still take the old place of the
`network` parameter into account to not entirely break the API.
Changelog-Added: hsmtool: password must now be entered on stdin. Password passed on the command line are discarded.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This is vital for calculating merkle trees; I previously used
towire+fromwire to get this!
Requires generation change so we can magic the ARRAY_SIZE var (the C
pre-processor can't uppercase things).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Actually, it's more complex to translate the xpub descriptor to
testnet because of the descriptor checksum.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
This adds a command which outputs the two output descriptors
corresponding to our onchain wallet.
This can be useful for an external service to monitor / send fund to our
wallet.
Further, an "xpriv" version of such descriptors could be used to import
onchain funds on a new wallet.
Changelog-Added: lightning-hsmtool: a new command was added to hsmtool for dumping descriptors of the onchain wallet
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
tools: Add `generatehsm` method to hsmtool to derivate BIP32 seeds from a
mnemonic using the BIP39 standard.
The new method uses libwally for the BIP39 to BIP32 derivation. It also
fails if an hsm_secret file already exists, so we do not overwrite
someone else's wallet without noticing.
It allows the use of passphrases, the ECHO mode in the terminal is
disable for higher security.
It currently supports "en", "es", "fr", "it", "jp", "zhs", "zht".
Changelog-Added: hsmtool: `hsm_secret` generation from a seed-phrase following BIP39.
Note that check-whitespace and check-bolt already do this, so we
can eliminate redundant lines in common/Makefile and bitcoin/Makefile.
We also include the plugin headers in ALL_C_HEADERS so they get
checked.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's a lot of it, and it means we can't `make check-source` on
these files.
Also bring bolt quotes up-to-date.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This avoids overwriting the ones in git, and generally makes things neater.
We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have to handle singletons which are arrays of variable-length entries:
this needs to be a ptr-to-ptr.
```C
struct blinded_payinfo {
u32 fee_base_msat;
u32 fee_proportional_millionths;
u16 cltv_expiry_delta;
u8 *features;
};
```
Before:
```C
struct tlv_invoice_tlvs {
...
struct blinded_payinfo *blindedpay;
```
After:
```C
struct tlv_invoice_tlvs {
...
struct blinded_payinfo **blindedpay;
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong. This happens in the
the bolt13 patch:
```C
struct tlv_offer_tlvs_blindedpath {
struct pubkey blinding;
struct onionmsg_path **path;
};
```
Before:
wire/gen_bolt13_tlv.c:
```C
for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise make thinks we're done, and we can get errors. Include
primitive code if we abort build halfway and leave .refresh-submodules dir.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We create ALL_PROGRAMS, ALL_TEST_PROGRAMS, ALL_C_SOURCES and
ALL_C_HEADERS. Then the toplevel Makefile knows which are
autogenerated (by wildcard), so it can have all the rules to clean
them or check the source as necessary.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Actually make the password argument optional for `guesstoremote` and `dumpcommitments` sub-commands, as shown in our documentation and help text.
Reported by `Barno` on #c-lightning IRC.
Before this, docker image will never detects that
`lightning-rpc` was created if it is running in regtest
or testnet, because the file will be created under
subfolder for each network name, and entrypoint does not
check "lightning-rpc" file in those folders.
By specifying `LIGHTNINGD_NETWORK` environment var
in dockerfile, we can now check correct path.
Changelog-Added: Docker build now includes `LIGHTNINGD_NETWORK` ENV variable which defaults to "bitcoin". An user can override this (e.g. by `-e` option in `docker run`) to run docker container in regtest or testnet or any valid argument to `--network`.
Changelog-Added: We now install `lightning-hsmtool` for your `hsm_secret` needs.
See: https://github.com/ElementsProject/lightning/issues/3717#issuecomment-644844594
It seems reasonable to add this to the standard install, and to document it properly as well, hopefully we can fill in the documentation better later on.
According to #3226, it looks like clang's LD error format has changed.
This patch adds the new format so we can parse the mocks successfully.
Apple clang version 11.0.3 (clang-1103.0.32.29)
```
checking for ANSI C header files... Undefined symbols for architecture x86_64:
"_fromwire_amount_msat", referenced from:
_fromwire_tlv_test_n1_tlv3 in ccj4zKdV.o
"_fromwire_bool", referenced from:
_fromwire_test_msg in ccj4zKdV.o
_fromwire_test_msg_option_short_id in ccj4zKdV.o
_fromwire_test_msg_option_one in ccj4zKdV.o
_fromwire_test_msg_option_two in ccj4zKdV.o
"_fromwire_test_enum", referenced from:
_fromwire_test_msg in ccj4zKdV.o
_fromwire_test_msg_option_short_id in ccj4zKdV.o
_fromwire_test_msg_option_one in ccj4zKdV.o
_fromwire_test_msg_option_two in ccj4zKdV.o
"_fromwire_tlvs", referenced from:
_fromwire_test_tlv1 in ccj4zKdV.o
_fromwire_test_tlv2 in ccj4zKdV.o
_fromwire_test_tlv3 in ccj4zKdV.o
"_fromwire_tu32", referenced from:
_fromwire_tlv_test_n2_tlv2 in ccj4zKdV.o
"_fromwire_tu64", referenced from:
_fromwire_tlv_test_n1_tlv1 in ccj4zKdV.o
_fromwire_tlv_test_n2_tlv1 in ccj4zKdV.o
"_fromwire_u16", referenced from:
_fromwire_test_features in ccj4zKdV.o
_fromwire_subtype_var_assign in ccj4zKdV.o
_fromwire_subtype_arrays in ccj4zKdV.o
_fromwire_tlv_test_n1_tlv4 in ccj4zKdV.o
_fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
_fromwire_test_msg in ccj4zKdV.o
_fromwire_test_tlv1 in ccj4zKdV.o
...
"_fromwire_u32", referenced from:
_fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
_fromwire_test_msg in ccj4zKdV.o
_fromwire_test_msg_option_short_id in ccj4zKdV.o
_fromwire_test_msg_option_one in ccj4zKdV.o
_fromwire_test_msg_option_two in ccj4zKdV.o
"_fromwire_u64", referenced from:
_fromwire_test_short_id in ccj4zKdV.o
_fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
"_fromwire_u8", referenced from:
_fromwire_subtype_var_assign in ccj4zKdV.o
_fromwire_subtype_var_len in ccj4zKdV.o
_fromwire_subtype_varlen_varsize in ccj4zKdV.o
_fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
"_fromwire_u8_array", referenced from:
_fromwire_test_features in ccj4zKdV.o
_fromwire_subtype_arrays in ccj4zKdV.o
_fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
_fromwire_test_msg in ccj4zKdV.o
_fromwire_test_msg_option_short_id in ccj4zKdV.o
_fromwire_test_msg_option_one in ccj4zKdV.o
_fromwire_test_msg_option_two in ccj4zKdV.o
...
```
Changelog-Fixed: Build for macOS Catalina / Apple clang v11.0.3 fixed
`struct tx_parts` is just a txid and a bunch of inputs and outputs,
some of which may be NULL.
This is both a nod towards a future where we (or our peer) can combine
HTLCs or (in an eltoo world) commitments, although for the moment all
our tx_parts will be complete.
It also matches our plan to split `bitcoin_tx` into two types: this
`struct tx_parts` where we don't know input amounts etc, and `psbt`
where we do.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
now that witness script data is saved into the tx/psbt which is
serialized across the wire, there's no reason to use witscript to do
this. good bye witscript!
- we've moved tmpctx management to setup.c from daemon.c, so we update
the `check-tmpctx`
- `common_setup(char *)` is now a valid analog for `setup_locale`, so we
check for either in check-setup_locale
The formatting makes it harder for update-mocks, eg:
/* Generated stub for fmt_wireaddr_without_port */
char *fmt_wireaddr_without_port(const tal_t *ctx UNNEEDED, const struct wireaddr *a UNNEEDED)
{ fprintf(stderr, "fmt_wireaddr_without_port called!\n"); abort(); }
/* Could not find declaration for fromwire_onionmsg_path */
/* Generated stub for json_add_member */
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When we have only a single member in a TLV (e.g. an optional u64),
wrapping it in a struct is awkward. This changes it to directly
access those fields.
This is not only more elegant (60 fewer lines), it would also be
more cache friendly. That's right: cache hot singles!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There's a race condition with loading the submodules that's causes a
build failure on my machine, since the libwally 'includes' aren't on
disk yet when the gcc build step starts.
This will be used when we want to specify these in a route. But for now, they
only alter gossipd, which always sets them to NULL.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Use `LC_ALL=C sort` instead of `sort` so that mocks get sorted in
the same way on all developers' environments.
Re-record the result of `make update-mocks`.
Changelog-None
We currently abuse the added_htlc and failed_htlc messages to tell channeld
about existing htlcs when it restarts. It's clearer to have an explicit
'existing_htlc' type which contains all the information for this case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Before this patch we used to send `double`s over the wire by just
copying them. This is not portable because the internal represenation
of a `double` is implementation specific.
Instead of this, multiply any floating-point numbers that come from
the outside (e.g. JSONs) by 1 million and round them to integers when
handling them.
* Introduce a new param_millionths() that expects a floating-point
number and returns it multipled by 1000000 as an integer.
* Replace param_double() and param_percent() with param_millionths()
* Previously the riskfactor would be allowed to be negative, which must
have been unintentional. This patch changes that to require a
non-negative number.
Changelog-None
Before this patch we used `int` for error codes. The problem with
`int` is that we try to pass it to/from wire and the size of `int` is
not defined by the standard. So a sender with 4-byte `int` would write
4 bytes to the wire and a receiver with 2-byte `int` (for example) would
read just 2 bytes from the wire.
To resolve this:
* Introduce an error code type with a known size:
`typedef s32 errcode_t`.
* Change all error code macros to constants of type `errcode_t`.
Constants also play better with gdb - it would visualize the name of
the constant instead of the numeric value.
* Change all functions that take error codes to take the new type
`errcode_t` instead of `int`.
* Introduce towire / fromwire functions to send / receive the newly added
type `errcode_t` and use it instead of `towire_int()`.
In addition:
* Remove the now unneeded `towire_int()`.
* Replace a hardcoded error code `-2` with a new constant
`INVOICE_EXPIRED_DURING_WAIT` (903).
Changelog-Changed: The waitinvoice command would now return error code 903 to designate that the invoice expired during wait, instead of the previous -2
We cannot let users use `sendcustommsg` to inject messages that are handled
internally since it could result in our internal state tracking being borked.
I really want a type which means "I am a wrapped onion reply" as separate
from "I am a normal wire msg". Currently both user u8 *, and I got very
confused trying to figure out where each one was an unwrapped error msg,
or where it still needed (un)wrapping.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Add towire_int() and fromwire_int() functions to "(de)serialize"
"int". This will only work as long as both the caller of towire_int()
and the caller of fromwire_int() use the same in-memory representation
of signed integers and have the same sizeof(int).
Changelog-None
TIL: `rename` doesn't like its source and target to be on different
partitions. This was causing the `hsmtool` tests to fail whenever we ran them
on a different partition than the lightning-dir (e.g., `/dev/shm` for faster
testing), because we made the backup copy in the current working directory.
This changes this and creates the backup next to the original file, which has
a reasonable chance to be on the same partition.
Changelog-Changed: hsmtool: The `hsmtool` now creates its backup copy in the same directory as the original `hsm_secret` file.
Recent FreeBSD versions use LLVM's lld as a linker, not the GNU ld.
Their behavior slightly differs, so adapt the build system to handle
either one.
* The LLVM's linker prints "undefined symbol:" instead of
"undefined reference to". Tweak tools/mockup.sh to also look for that
message.
* The LLVM's linker may only print the first dozen errors (omitting
the rest to avoid flooding the screen). tools/update-mocks.sh relies
on getting all errors as it extracts the missing symbols' names from
the error output and creates mocks for them. Detect if errors were
omitted and re-run, telling the linker to not omit any. The GNU linker
does not support -error-limit=0, so unfortunately we can't just run
with that option unconditionally from the first attempt.
* Nit: FreeBSD's sed(1) prints "t" for "\t" instead of a horizontal tab.
Use a verbatim tab in the command, instead of "\t" which works on
both.
Changelog-Fixed: Developer mode compilation on FreeBSD.
This is the final step: we pass the complete fee_states to and from
channeld.
Changelog-Fixed: "Bad commitment signature" closing channels when we sent back-to-back update_fee messages across multiple reconnects.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This, in the case of data loss on a channel with `option_static_remotekey`
negotiated, allows to likely (if the dbid is not unreasonable) recover
the funds from a remote unilateral close just with the hsm_secret.
Changelog-added: A new command, 'guesstoremote', is added to the hsmtool. It is meant to be used to recover funds after an unilateral close of a channel with `option_static_remotekey` enabled.
Since the parser itself just parses and doesn't include validation anymore we
need to put that functionality somewhere. The validation consists of enforcing
that the types are in monotonically increasing order without duplicates and
that for the even types we know how to handle it.
We were weaving in and out of generic code through `fromwire_tlvs` with custom
parameters defining the types in that namespace. This hard-wires the parser
with the namespace's types. Slowly trying to deprecate `fromwire_tlvs` in
favor of this typesafe variant.
This takes a dbid, a "depth" (how many points to dump), the hsm_secret
path, and a potential password to dump informations about all
commitments until the depth.
Co-Authored-By: Sjors Provoost <sjors@sprovoost.nl>
We now have a pointer to chainparams, that fails valgrind if we do anything
chain-specific before setting it.
Suggested-by: Rusty Russell <@rustyrussell>
These are special in extended regexs, and so we fail to match once we fix
the BOLT comment in common/test/run-bigsize.c
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Somehow this change got lost, but it's needed for option_static_remotekey,
to quote gen_peer_wire_csv:
msgtype,channel_reestablish,136
msgdata,channel_reestablish,channel_id,channel_id,
msgdata,channel_reestablish,next_commitment_number,u64,
msgdata,channel_reestablish,next_revocation_number,u64,
msgdata,channel_reestablish,your_last_per_commitment_secret,byte,32,option_data_loss_protect,option_static_remotekey
msgdata,channel_reestablish,my_current_per_commitment_point,point,,option_data_loss_protect
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Makefile:424: warning: overriding recipe for target 'clean'
tools/Makefile:12: warning: ignoring old recipe for target 'clean'
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Because it required update-mocks, which is a hack which relies on the
format of linker errors (!) I'd prefer to make this --enable-developer
only.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to tell it that the tmp file is an intermediate, so doesn't need
remaking if it doesn't exist.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our TLV serializer relies on TLV outputs to be ordered by type
number. Prior to this commit we relied on 1) the ordering in the
RFC to be correct and 2) users to be using a version of Python that
respects stable ordering of dicts (i.e. Python 3.7+)
Instead of relying on these implicitly, we now explicitly sort messages
by type number when the TLV sets.
Resolves#2956.
Thanks-To: @ScottTre for the sort function
Reported-By: @ZmnSCPxj
1. These days we delete the [Unreleased] tag during rcs.
2. Make sure we test the release build process during rc1, since I
screwed that up last release.
3. Add a section on rc2, etc.
4. Do final release via a github PR, since I screwed that up on the
prior release.
5. Update `tools/build-release.sh` and instructions to show that we now
make a reproducible build for Ubuntu 18.04 x86-64.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
