Until it gains miniscript support, wally doesn't know how to
finalize P2WSH. This happens with `option_anchor_outputs` for
to-remote, which requires a 1 block delay to force use of
anchor outputs for CPFP.
So we finalize this ourselves.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This also means we subtract 660 satoshis more everywhere we subtract
the base fee (except for mutual close, where the base fee is still
used).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Includes:
psbt: Use renamed functions for new wally version
psbt: Set the transaction directly to avoid script workarounds
psbt: Use low-S grinding when computing signatures
tx: Use wally_tx_clone from libwally now that its exported
Signed-off-by: Jon Griffiths <jon_p_griffiths@yahoo.com>
Technically, they could do this themselves, but it's much nicer to have one
place to do it (and it makes sure we get the required information into the
PSBT, which is actually not entirely accessible through listfunds, as that
doesn't want to consult with the HSM for close outputs).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Added: JSON RPC: new low-level coin selection `fundpsbt` routine.
PSETs have a bit different requirements. The witness_utxo needs
the asset tag + values, and these should also be added to the PSET
struct separately as well. To do this, we create a new 'init' method for
elements inputs, which takes care of the elements specific things.
We don't preserve detailed asset information at the moment, so provide a
way to convert from a sat to an amount_asset struct.
We also need a way to convert from an 'amount_asset' to a 'value' for
elements, which for explicit (i.e. non-blinded) asssets is a 0x01 prefix
plus the big-endian encoded value.
These are pulled from wallet/wallet.c, with the fix now that we grind sigs.
This reduces the fees we pay slightly, as you can see in the coinmoves changes.
I now print out all the coin moves in suitable format before we match:
you only see this if the test fails, but it's really helpful.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
most likely unused since the switch to libwally for internal blockchain
things.
these method names were clashing with ones that are to be introduced
with some libwally cleanups, so getting rid of them pre-emptively keeps
us libwally compatible
Our existing coin_moves tracking logic assumed that any tx we had an
input in belonged to *all* of our wallet (not a bad assumption as long
as there was no way to update a tx that spends our wallets)
Now that we've got `signpsbt` implemented, however, we need to be
careful about how we account for withdrawals. For now we do a best guess
at what the feerate is, and lump all of our spent outputs as a
'withdrawal' when it's impossible to disambiguate
the bitcoin_tx version is basically a wrapper for the wally_tx script
extraction -- here we pull it apart so we can easily get a tal'd script
for a wally_tx_output
The main change here is that the previously-optional open/accept
fields and reestablish fields are now compulsory (everyone was
including them anyway). In fact, the open/accept is a TLV
because it was actually the same format.
For more details, see lightning-rfc/f068dd0d8dfa5ae75feedd99f269e23be4777381
Changelog-Removed: protocol: support for optioned form of reestablish messages now compulsory.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
will either use a temporary psbt (and not munge the passed in psbt)
or will finalize in place -- finalization erases most of the signature
metadata from the psbt struct
We're not using the change_outnum for withdraw tx's (and the way
we were calculating it was broken as of the addition of 'multiple
outputs'). This removes the change output knowhow from withdraw_tx
entirely, and pushes the responsibility up to the caller to
include the change output in the output set if desired.
Consequently, we also remove the change output knowhow from hsmd.
Prior to this commit, passing a NULL stack to `bitcoin_tx_input_set_witness`
unsets the witness stack on the bitcoin_tx's wally_tx but leaves the
final witness on the PSBT unchanged.
at the moment, libwally's `wally_psbt_input_set_final_witness` will blow
up if you attempt to set a NULL witness -- instead we manually remove it
if the passed in stack is NULL. previously we would leave the PSBT's
witness unchanged.
calling `wally_psbt_finalize` doesn't return a status indicator; instead
you must call `psbt_is_finalized` to check that it's eligible for
'extraction' -- extraction will fail if the psbt is not in a finalized
state.
Update the `bitcoin_tx_add_input` interface to accept a witness script
and or scriptPubkey.
We save the amount + witness script + witness program (if known) to
the PSBT object for a transaction when creating an input.
`struct tx_parts` is just a txid and a bunch of inputs and outputs,
some of which may be NULL.
This is both a nod towards a future where we (or our peer) can combine
HTLCs or (in an eltoo world) commitments, although for the moment all
our tx_parts will be complete.
It also matches our plan to split `bitcoin_tx` into two types: this
`struct tx_parts` where we don't know input amounts etc, and `psbt`
where we do.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are useful for tx_parts:
1. wally_txid.
2. linearize_wtx.
3. wally_tx_input_spends
4. wally_tx_output_get_amount
5. wally_tx_input_get_txid
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
now that witness script data is saved into the tx/psbt which is
serialized across the wire, there's no reason to use witscript to do
this. good bye witscript!
Move all psbt creation into single method, new_psbt
note that if a psbt is init'd for a transaction that's
deserialized with scripts etc already attached, then set_global_tx
will fail. instead, we empty all of this out first.
if the tx is being re-init'd from a tx source that had a psbt attached
(e.g. fromwire_) then the script/witness data will get populated
appropriatel from there.
If we fail to unmarshal the tx (shouldn't happen, but...) we must
not dereference NULL.
Also tighten the check: towire_ must send 0 or all inputs.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It returns NULL, so you can simply `return fromwire_fail(...)`
if you want to return NULL in this case. Use that more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we now over-write the wally malloc/free functions, we need to do
so for tests as well. Here we pull up all of the common setup/teardown
logic into a separate place, and update the tests that use libwally to
use the new common_setup core
Changelog-None
We did this originally because these types are referred to in the bolts, and we
had no way of injecting the correct include lines into those. Now we do, so
there's less excuse for this.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Transactions that we 'get' from bitciond don't have the input values
available on the transaction; for these cases we'll sum up the inputs
amounts using a different data source than the transaction's
`input_amounts`. So we need to expose it here.
We roll the `elements_add_fee_output` function and the cropping of
overallocated arrays into the `bitcoin_tx_finalize` function. This is supposed
to be the final cleanup and compaction step before a tx can be sent to bitcoin
or passed off to other daemons.
This is the cleanup promised in #3491
This is a common thing to do, so create a macro.
Unfortunately, it still needs the type arg, because the paramter may
be const, and the return cannot be, and C doesn't have a general
"(-const)" cast.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This sets the nLockTime to the tip (and accordingly each input's nSequence to
0xfffffffe) for withdrawal transactions.
Even if the anti fee-sniping argument might not be valid until some time yet,
this makes our regular wallet transactions far less distinguishable from
bitcoind's ones since it now defaults to using native Segwit transactions
(like us). Moreover other wallets are likely to implement this (if they
haven't already).
Changelog-Added: wallet: withdrawal transactions now sets nlocktime to the current tip.
Since we now compute the hash while deserializing the block header we can now
just use it, no reason to serialize the header just to hash it again. This
also allows us to throw away all the added dynafed fields in the next commit
instead of having to carry them around.
This avoids having to re-serialize the block header just to compute the
hash. It also frees us from having to carry around all the details in the
header and we can hand around a minimal version.
This avoids having to re-serialize the block header just to compute the
hash. It also frees us from having to carry around all the details in the
header and we can hand around a minimal version.
We updated the protocol spec tests to verify a sig from a hash
and a private key; this updates mkcommit + mkgossip utilities
to print out the procotol compatible SIG() notation for all signatures.
--verbose will print a computed signature and more data as well.
Also adds --verbose flag to mkgossip.
Changelog-None
this is unnecessary, and actually severely limits the functionality
of `wally_tx_add_input`, which will expand the allocated input
length if there's not enough room for the additional input
```external/libwally-core/src/transaction.c
if (tx->num_inputs >= tx->inputs_allocation_len) {
/* Expand the inputs array */
struct wally_tx_input *p;
p = realloc_array(tx->inputs, tx->inputs_allocation_len,
tx->num_inputs + 1, sizeof(*tx->inputs));
...
tx->inputs = p;
tx->inputs_allocation_len += 1;
```
Currently the only source for amount_asset is the value getter on a tx output,
and we don't hand it too far around (mainly ignoring it if it isn't the
chain's main currency). Eventually we could bubble them up to the wallet, use
them to select outputs or actually support assets in the channels.
Since we don't hand them around too widely I thought it was ok for them to be
pass-by-value rather than having to allocate them and pass them around by
reference. They're just 41 bytes currently so the overhead should be ok.
Signed-off-by: Christian Decker <@cdecker>
We now have a pointer to chainparams, that fails valgrind if we do anything
chain-specific before setting it.
Suggested-by: Rusty Russell <@rustyrussell>
We used to match specifically on `is_elements && coinbase`, but we can just
hand off responsibility to libwally and then make sure we handle it correctly.
This is the main reason we started weaving the chainparams everywhere: being
able to compare the asset type with the fee paying asset tag, thus determining
the value of the asset correctly (we still treat any non-fee-paying assets as
having value 0).
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is required since liquid-regtest and liquidv1 have different asset tags
for L-BTC which is the fee-paying asset.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Especially when we grind fees we may end up setting the fees several times, so
instead of always adding a new fee output look for an existing one and set its
value.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We are checking against chain-dependent constants, so let's make sure we are
using the ones for the correct chain.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We use to use the non-elements ones and then patch them manually. By using the
correct ones right from the start we have less work on our side.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Depending on the network we end up with different signature hash algorithms,
so we just collect that decision in one place.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
If we are handling an elements transaction the value is not stored in the
satoshi field, rather it is stored in the `value` field which is prefixed with
a version (0x01) and is counted in `asset` units.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Skipping coinbase transactions and ensuring that the transaction is serialized
correctly when sending it onwards.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The header is not a contiguous section of memory in elements, and it is of
variable length, so the simple trick of hashing in-memory data won't work
anymore. Some of the datafields would have been wrong on big-endian machines
anyway, so this is better anyway.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was a bit of trial and error due to libwally not liking hints when it
comes to length measurements, also the parsing bumps against a masking issue
in libwally that I'd following up on their issue tracker.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Since the difference between non-elements and elements block headers is just
the middle 2 fields, I split the old parsing code so I could add the middle
part.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Using a global variable is a bit lazy, but weaving the network type through
the entire stack is a daunting task. Maybe we can make that happen at a later
stage.
Most of the changes in `chainparams.c` are just formatting the
`genesis_blockhash` a bit nicer (`clang-format` to the rescue).
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The math is a bit tricky, so encapsulate it.
Includes the extra 'e' in 'announcable' as noted by @cdecker :)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will never be reliable under high load, without making it unable
to detect real errors. But the test is useful because if we don't
have this test we'll never notice if we break the const-timedness of
our implementation.
So, move the calloc out of the test loop (which seems to make it more
reliable), and then after we've run it, check the 1-minute load
average. Too high, we don't complain about results. It's not
perfect, but it's better.
Running 100 times (-O3) serially gave 100 successes with the following results:
Constant: Within 5% 562-926(832.89+/-73)/1000 times
Non-constant: More than 5% slower 860-990(956.35+/-26)/1000 times
More importantly, if we swap the const and non-const tests, we get
the expected 100 failures:
Non-constant: Within 5% 14-79(41.17+/-14)/1000 times
Constant: More than 5% slower 44-231(111.89+/-33)/1000 times
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the normal convention for this type; it makes using converters
a little easier. See next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is more reliable under load now: shorten the times so it is
likely to run in a single timeslice, and add a nanosleep so it's
likely to be at the start of the timeslice.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the other origin, besides `bitcoin_tx`, where we create `bitcoin_tx`
instances, so add the context as soon as possible. Sadly I can't weave the
chainparams into the deserialization code since that'd need to change all the
generated wire code as well.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The way we build transactions, serialize them, and compute fees depends on the
chain we are working on, so let's add some context to the transactions.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Simplifying some operations, erroring in some cases and moving to global
defines for constants.
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <decker.christian@gmail.com>
'bip70_name' is corresponding to the 'chain' field of
the API 'getblockchaininfo'.
At the beginning of lightningd, we use the 'chain' field of 'getblockchaininfo' to check if we are on right blockchain.
We check that memcmp *isn't* constant time, but that's only true under
-O2 or above: __OPTIMIZE__ doesn't distinguish.
So we need a finer-grained test. Also reduce verbosity by default.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The chainparams are needed to know the prefixes, so instead of passing down
the testnet, we pass the entire params struct.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We were deciding whether an address is a testnet address or not in the parser,
and then checking whether it matches our expectation outside as well. This
just returns the address version instead, and still checks it against our
expectation, but without having the parser need to know about address types.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is an intermediate step since the only difference between p2pkh and p2sh
is the argument that the parsing functions take, and parsing twice for that
reason alone is quite useless.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
* remove libbase58, use base58 from libwally
This removes libbase58 and uses libwally instead.
It allocates and then frees some memory, we may want to
add a function in wally that doesn't or override
wally_operations to use tal.
Signed-off-by: Lawrence Nahum lawrence@greenaddress.it
This fixes block parsing on testnet; specifically, non-standard tx versions.
We hit a type bug in libwally (wallt_get_secp_context()) which I had to
work around for the moment, and the updated libsecp adds an optional hash
function arg to the ECDH function.
Fixes: #2563
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Pubkeys are not not actually DER encoding, but Pieter Wuille corrected
me: it's SEC 1 documented encoding.
Results from 5 runs, min-max(mean +/- stddev):
store_load_msec,vsz_kb,store_rewrite_sec,listnodes_sec,listchannels_sec,routing_sec,peer_write_all_sec
38922-39297(39180.6+/-1.3e+02),2880728,41.040000-41.160000(41.106+/-0.05),2.270000-2.530000(2.338+/-0.097),44.570000-53.980000(49.696+/-3),32.840000-33.080000(32.95+/-0.095),43.060000-44.950000(43.696+/-0.72)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is what all of this has been working towards: ripping out the handwoven
transaction handling. By removing the custom parsing we can finally switch
over to using `wally_tx` as sole representation of transactions in
memory. The commit is a bit larger but it's mostly removing setters and old
references to the input and output fields.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
These are handled internally in the `wally_tx` and do not conform to our usual
tallocated strings that can by inspected using `tal_bytelen`, and we don't
really want to litter our code with whitelisting comments for the
`amount_sat.satoshis` access, so these just do read-only on the fly conversions.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The `wally_tx_input`s do not keep track of their input value, which means we
need to track them ourselves if we try to sign these transactions at a later
point in time.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
These are used when grinding the feerate and signing. These are just simple
facades that keep both wally and old style transactions in sync.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
During the migration to `libwally` we want to make absolutely sure that both
transactions are generated identical, and can eventually be switched over.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We are slowly migrating towards a wally-transactions only world, but to make
this reviewable we start building both old and new style transactions in
parallel. In a second pass we'll then start removing the old ones and use
libwally only.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We set the version BIP32_VER_TEST_PRIVATE for testnet/regtest
BIP32 privkey generation with libwally-core, and set
BIP32_VER_MAIN_PRIVATE for mainnet.
For litecoin, we also set it like bitcoin else.
We need to do it in various places, but we shouldn't do it lightly:
the primitives are there to help us get overflow handling correct.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to still accept it when parsing the database, but this flag
should allow upgrade testing for devs building on top
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a bug 0ba547ee10 caused by
short_channel_id overflow. If we'd caught this, we'd have terminated
the peer instead of crashing, so add appropriate checks.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I upgraded my node with --disable-compat, and a heap of channels closed like:
CHANNELD_NORMAL:We disagree on short_channel_ids: I have 557653x0x1351, you say 557653x2373x1",
This is because the scids are strings in the databases, and it failed to parse
them properly.
Now we'll not start if that happens.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Currently only used by gossipd for channel elimination.
Also print them in canonical form (/[01]), so tests need to be
changed.
Suggested-by: @cdecker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is prep work for when we sign htlc txs with
SIGHASH_SINGLE|SIGHASH_ANYONECANPAY.
We still deal with raw signatures for the htlc txs at the moment, since
we send them like that across the wire, and changing that was simply too
painful (for the moment?).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently make sure that all the bitcoin_tx input scripts are NULL
and set the input script of the input we're signing, so we can easily
reuse the tx hashing code for signature checks. This means that we
sometimes jump through hoops to make sure input scripts are NULL, and
also means that the tx can't be const.
Put more logic inside bitcoin/tx so it can simply ignore things we
don't want to hash.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
