the original version of the subtype generator emitted '$'
to designate that a field was a subtype; now it's got a different
format:
funding_type,8,num_inputs,2
funding_type,10,input_info,num_inputs*input_info
this patch updates our generator to understand this new format
This is needed so that some csv's can expose their subtype parsing
functions in their header. This gets used in a later PR where
we start replacing manually created 'subtype' definitions with
generated ones.
We can save significant space by combining both sides: so much that we
can reduce the WIRE_LEN_LIMIT to something sane again.
MCP results from 5 runs, min-max(mean +/- stddev):
store_load_msec:34467-36764(35517.8+/-7.7e+02)
vsz_kb:2637488
store_rewrite_sec:35.310000-36.580000(35.816+/-0.44)
listnodes_sec:1.140000-2.780000(1.596+/-0.6)
listchannels_sec:55.390000-58.110000(56.998+/-0.99)
routing_sec:30.330000-30.920000(30.642+/-0.19)
peer_write_all_sec:50.640000-53.360000(51.822+/-0.91)
MCP notable changes from previous patch (>1 stddev):
-store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
+store_rewrite_sec:35.310000-36.580000(35.816+/-0.44)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Don't turn them to/from pubkeys implicitly. This means nodeids in the store
don't get converted, but bitcoin keys still do.
MCP results from 5 runs, min-max(mean +/- stddev):
store_load_msec:33934-35251(34531.4+/-5e+02)
vsz_kb:2637488
store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
listnodes_sec:1.020000-1.290000(1.146+/-0.086)
listchannels_sec:51.110000-58.240000(54.826+/-2.5)
routing_sec:30.000000-33.320000(30.726+/-1.3)
peer_write_all_sec:50.370000-52.970000(51.646+/-1.1)
MCP notable changes from previous patch (>1 stddev):
-store_load_msec:46184-47474(46673.4+/-4.5e+02)
+store_load_msec:33934-35251(34531.4+/-5e+02)
-vsz_kb:2638880
+vsz_kb:2637488
-store_rewrite_sec:46.750000-48.280000(47.512+/-0.51)
+store_rewrite_sec:34.720000-35.130000(34.94+/-0.14)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
fixup printing methods in devtools/decodemsg such that TLV's can
now be printed as well. here's how you'd use it:
$ ./devtools/decodemsg --tlv opening_tlv 0120001E020202020202020202020202020202020202020202020202020202020202
> WIRE_OPTION_UPFRONT_SHUTDOWN_SCRIPT (size 32):
> shutdown_scriptpubkey=[020202020202020202020202020202020202020202020202020202020202]
TLV's use var_int's for messages sizes, both internally and
in the top level (you should really stack a var_int inside a var_int!!)
this updates our automagick generator code to understand 'var_ints'
passing back a null TLV was crashing here, because we tried to
dereference a null pointer. instead, we put it into a temporary
struct that we can check for NULL-ness, before assigning to the
passed in pointer.
let's let the fromwire__tlv methods allocate the tlv-objects and
return them. we also want to initialize all of their underlying
messages to NULL, and fail if we discover a duplicate mesage type.
if parsing fails, instead of returning a struct we return NULL.
Suggested-By: @rustyrussell
Since messages in TLV's are optional, the ideal way to deal with
them is to have a 'master struct' object for every defined tlv, where
the presence or lack of a field can be determined via the presence
(or lack thereof) of a struct for each of the optional message
types.
In order to do this, appropriately, we need a struct for every
TLV message. The next commit will make use of these.
Note that right now TLV message structs aren't namespaced to the
TLV they belong to, so there's the potential for collision. This
should be fixed when/where it occurs (should fail to compile).
Add tlv-messages to the general messages set so that their parsing
messages get printed out.
FIXME: figure out how to account for partial message length processing?
Version 1.1 of the lightning-rfc spec introduces TLVs for optional
data fields. This starts the process of updating our auto-gen'd
wireformat parsers to be able to understand TLV fields.
The general way to declare a new TLV field is to add a '+' to the
end of the fieldname. All field type declarations for that TLV set
should be added to a file in the same directory by the name
`gen_<field_name>_csv`.
Note that the FIXME included in this commit is difficult to fix, as
we currently pass in the csv files via stdin (so there's no easy
way to ascertain the originating directory of file)
Otherwise we can't really return a variable sized message with more than 65k
results. This was causing an integer overflow in `listchannels` (see #2504 for
details).
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Basically we tell it that every field ending in '_msat' is a struct
amount_msat, and 'satoshis' is an amount_sat. The exceptions are
channel_update's fee_base_msat which is a u32, and
final_incorrect_htlc_amount's incoming_htlc_amt which is also a
'struct amount_msat'.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
They're generally used pass-by-copy (unusual for C structs, but
convenient they're basically u64) and all possibly problematic
operations return WARN_UNUSED_RESULT bool to make you handle the
over/underflow cases.
The new #include in json.h means we bolt11.c sees the amount.h definition
of MSAT_PER_BTC, so delete its local version.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we have an array of varlen structures (which require a ctx arg), we
should make that arg the array itself (which was tal_arr()), not the
root context.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
tal_count() is used where there's a type, even if it's char or u8, and
tal_bytelen() is going to replace tal_len() for clarity: it's only needed
where a pointer is void.
We shim tal_bytelen() for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This requires a tweak to generate-wire.py too, since it always called the
top-level routine 'print_message'.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
gossip_getnodes_entry was used by gossipd for reporting nodes, and for
reporting peers. But the local_features field is only available for peers,
and most other fields are only available from node_announcement.
Note that the connectd change actually means we get less information
about peers: gossipd used to do the node lookup for peers and include the
node_announcement information if it had it.
Since generate_wire.py can't create arrays-of-arrays, we add a 'struct
peer_features' to encapsulate the two feature arrays for each peer, and
for convenience we add it to lightningd/gossip_msg.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Well, it's generated by shachain, so technically it is a sha256, but
that's an internal detail. It's a secret.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We already work around this by using an array with a 0/1 length convention,
but better to be explicit.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We always hand in "NULL" (which means use tal_len on the msg), except
for two places which do that manually for no good reason.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A convenient alias for char *, though we don't allow control characters
so our logs can't be fooled with embedded \n.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are now logically arrays of pointers. This is much more natural,
and gets rid of the horrible utxo array converters.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The close_info is needed to re-derive the secret key that is supposed
to be used to sign the input spending the output.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_blkid_to_hex() so it's reversed as people expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_txid_to_hex() so it's reversed as people expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If a structure foo has a optional fields opt1 and opt2, this creates
towire_foo, towire_foo_opt1 and towire_foo_opt2 (since opt2 implies opt1),
similarly for fromwire_*.
This requires the callers to be updated to call the correct routines (eg.
try fromwire_foo_opt2, then fromwire_foo_opt1, then finally fromwire_foo),
but this is a minimal change to the generation code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now in sync with 8ee57b97738b1e9467a1342ca8373d40f0c4aca5.
Our tool doesn't need to convert them any more, but we actually had a
mis-typed field in the HSM which needed fixing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Some of the struct array helpers need to allocate data when
deserializing their fields. The `getnodes` reply is one such example
that allocates the hostname. Since the change to calling array helpers
the getnodes call was broken because it was attempting to allocate off
of the entry, which did not have a tal header, thus failing.
We use the fourth value (size) to determine the type, unless the fifth
value is suppled. That's silly: allow the fourth value to be a typename,
since that's the only reason we care about the size at all!
Unfortunately there are places in the spec where we use a raw fieldname
without '*1' for a length, so we have to distingish this from the
typename case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Except for the trivial case of u8 arrays, have the generator create
the loop code for the array iteration.
This removes some trivial helpers, and avoids us having to write more.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The wiregen tool was a bit hard to maintain since it was printing all
over the place, mixing template and processing logic. This commit
tears the two apart, externalizes everything that is not a single code
line, and repackages it into templates. Specifically functions are now
their own template and header/implementation files are a template.
Furthermore this simplifies some of the boilerplate of mapping types
to sizes and back again, by extracting them into dicts.
All changes have been verified to produce identical results on the
current wire definitions, except a bit of whitespace changes.
The spec 4af8e1841151f0c6e8151979d6c89d11839b2f65 uses a 32-byte 'channel-id'
field, not to be confused with the 8-byte short ID used by gossip. Rename
appropriately, and update to the new handshake protocol.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's awkward to handle them differently. But this change means we
need to expose them to the generated code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
