Commit Graph

301 Commits

Author SHA1 Message Date
Rusty Russell
35cca73d2f devtools/fp16: conversion tool for fp16 format.
We use this in gossmap to represent htlc max/min approximations.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-03-22 18:45:41 +10:30
Rusty Russell
43a833e405 lightningd: remove support for legacy onion format.
As per proposal in https://github.com/lightning/bolts/pull/962

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: protocol: support for legacy onion format removed, since everyone supports the new one.
2022-03-18 09:20:11 +10:30
niftynei
7add7ca199 json: reverse parse a coin_mvt tag back into an enum 2022-03-05 15:12:12 +10:30
niftynei
ea36c3a938 db: collapse the db_queries files into the generated ones
There's no reason to have these be independent of the generated files,
all the data contained within them is 'formulaic'
2022-03-05 15:03:34 +10:30
niftynei
ce12d2b8a9 database: pull out database code into a new module
We're going to reuse the database controllers for the accounting plugin
2022-03-05 15:03:34 +10:30
Simon Vrouwe
760c271381 devtools: in decodemsg, add newline and fix decoding a node_announcement with tlvs
Terminal prompt got messed up because missing newline in case of empty fields.

printwire_addresses expected it to be the last field, which is not
the case of a node_announcement with tlv
2022-03-01 09:38:40 +10:30
Rusty Russell
9c0bb444b7 per_peer_state: remove struct crypto_state
Now that connectd does the crypto, no need to hand around crypto_state.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell
6bfc3d8692 gossipwith: create our own internal sync_crypto functions.
This avoids changes to crypto_sync which are coming in next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
Rusty Russell
7e7a63a20d connectd: keep timeout timer around so we can disable it.
connectd will be keeping the conn open, so it needs to free this
"conn_timeout" timer.  Pass it through, so we can do that.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-01-20 15:24:06 +10:30
ZmnSCPxj jxPCSnmZ
d088288daa bitcoin/chainparams.h: Split BIP173 name into onchain and Lightning HRPs.
Fixes: #4937
2021-12-14 11:21:35 +10:30
Rusty Russell
f64df6fbb5 devtools: simple script to look for functions which don't seem to be used.
Gives a rough first-pass, at least.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell
4ffda340d3 check: make sure all files outside contrib/ include "config.h" first.
And turn "" includes into full-path (which makes it easier to put
config.h first, and finds some cases check-includes.sh missed
previously).

config.h sets _GNU_SOURCE which really needs to be done before any
'#includes': we mainly got away with it with glibc, but other platforms
like Alpine may have stricter requirements.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-06 10:05:39 +10:30
Rusty Russell
163d3a16f4 doc: clean up offers bolt quotes
As of 2b923a0367c5f9154fcec706e3302cc4658dd889.

Recurrence quotes need to be marked separately, since they're no longer
in offers main bolt.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell
b3af5f5a2c spec: import latest onionmessage spec, based on routeblinding.
This is from 6e99c5feaf60cb797507d181fe583224309318e9

We renamed the enctlv field to encrypted_recipient_data in the spec, and the
new onion_message is message 513.  We don't handle it until the next patch.

Two renames:
1. blinding_seed -> blinding_point.
2. enctlv -> encrypted_recipient_data.

We don't do a compat cycle for our JSON APIs for these experimental
features only used by our own plugins, we just rename.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell
1ec6346f3d common: rename current onion message structures to obs2_.
Yes, we changed the spec again.  Hopefully for the last time!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Rusty Russell
b74848f6f6 common: remove support for pre v0.10.2 onionmessages.
Temporarily disable sendpay_blinding test which uses obsolete onionmsg;
there's still some debate on the PR about how blinded HTLCs will work.

Changelog-EXPERIMENTAL: onionmessage: removed support for v0.10.1 onion messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-12-01 05:44:28 +10:30
Michael Schmoock
01e8a523e9 bolt7: allow announcement of ADDR_TYPE_DNS 2021-11-30 09:38:17 +10:30
Rusty Russell
1cbdb9cc12 devtools/taldump-analyze.py: script to analyze tal_dump() output.
Example output on my production machine, running v0.10.2rc1-9-g3d8293d:

```
Top sizes by name
backtrace: 1172021248 bytes, 4578208 items
wallet/txfilter.c:111:struct bitcoin_outpoint: 49584636 bytes, 1377351 items
lightningd/plugin.c:1651:char[]: 17261056 bytes, 16 items
lightningd/log.c:282:char[]: 16653908 bytes, 838086 items
wally_tal: 13390257 bytes, 122503 items
lightningd/log.c:250:struct log_entry[]: 7340032 bytes, 1 items
lightningd/jsonrpc.c:939:char[]: 6979648 bytes, 214 items
lightningd/log.c:440:u8[]: 3278359 bytes, 1338 items
common/memleak.c:51:struct memleak_notleak: 2216763 bytes, 2216763 items
Top sizes by path
['']: 2579337788 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd']: 831485962 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet']: 756169631 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter']: 756166227 bytes, 2 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter', 'wallet/txfilter.c:134:struct outpointset']: 706581191 bytes, 2 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable']: 457580181 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book']: 430307262 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology']: 48054597 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block']: 47076437 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]']: 47034285 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins']: 17569217 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins', 'lightningd/plugin.c:246:struct plugin', 'lightningd/plugin.c:1651:char[]']: 16784384 bytes, 5 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx']: 13730127 bytes, 452 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/jsonrpc.c:1189:struct jsonrpc']: 7745890 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book', 'lightningd/log.c:250:struct log_entry[]']: 7340032 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx', 'wally_tal']: 4480328 bytes, 166 items
['', 'tmpctx']: 0 bytes, 1 items
```
2021-11-27 10:38:13 +10:30
Rusty Russell
fcf3d0ce6c db: turn generated queries array into a simple hash table.
Since we have that functionality, let's use it.  Also, make table
const.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
787fbb1228 db: create simple hashtable of fields in SELECT.
This simplistically maps names to numbers, eg:

	SELECT foo, bar FROM tbl;

'foo' -> 0
'bar' -> 1

If a statement is too complex for our simple parsing, we treat it as a
single field (which currently it always is).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-17 10:51:48 +10:30
Rusty Russell
2f247c7bfb torv2: remove support for advertizing and connecting.
October was the date Torv2 is no longer supported by the Tor Project;
it will probably not work at all by next release, so we should remove
it now even though it's not quite the 6 months we prefer for
deprecation cycles.

I still see 110 nodes advertizing Torv2 (vs 10,292 Torv3); we still
parse and display it, we just don't advertize or connect to it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-11-14 18:49:46 +01:00
Christian Decker
15b403531e db: Allow DB-specific queries in the migrations
Sometimes we need some DB-specific magic, and marking migrations with
`/*PSQL*/` or `/*SQLITE*/` will now give us that.
2021-10-31 13:11:34 +01:00
niftynei
1fe829c546 lightningd: new option for htlc dust limit
To reduce the surface area of amount of a channel balance that can be
eaten up as htlc dust, we introduce a new config
'--max-dust-htlc-exposure-msat', which sets the max amount that any
channel's balance can be added as dust

Changelog-Added: config: new option --max-dust-htlc-exposure-msat, which limits the total amount of sats to be allowed as dust on a channel
2021-10-23 12:59:13 +02:00
Rusty Russell
b013b3ab0c patch websocket-address-support.patch 2021-10-22 11:56:30 +02:00
Rusty Russell
c503232cde common: use bitcoin_outpoint.
I started pulling this thread, and the entire codebase got unravelled.

Oh well, it's done now!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
36b66871a2 db: add min/max commitnum fields to channel_htlcs.
And initialize max to current height max when htlcs are already dead.
Turns out (thanks CI!) that MAX() of multiple columns is GREATEST() in
Postgres.  That's clearer (MAX is used elsewhere for single columns),
so translate on the sqlite3 side.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-15 12:09:36 +02:00
Rusty Russell
c92ce59892 BOLT 12: switch invoice_request/invoice to singular chain field.
We keep the now-removed chains field, and in deprecated mode, we set it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: bolt12: `chains` in invoice_request and invoice is deprecated, `chain` is used instead.
2021-10-08 13:47:30 +02:00
Rusty Russell
45bf7a3974 bolt12: update to latest spec.
Main changes are:
1. Uses point32 instead of pubkey32.
2. Uses issuer instead of vendor.
3. Uses byte instead of u8.
4. blinded_path num_hops is now a byte, not u16 (we don't use that yet!).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: bolt12: `vendor` is deprecated: the field is now called `issuer`.
2021-10-08 13:47:30 +02:00
Rusty Russell
8f582e770c BOLT12: use point32 instead of pubkey32.
That's the modern BOLT12 term.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-10-08 13:47:30 +02:00
Rusty Russell
7ef2f4d7fb devtools/features: tool to convert feature bitmap to names.
For example:

```
$ ./devtools/features 80008008226aa2
option_data_loss_protect/odd (optional)
option_upfront_shutdown_script/odd (optional)
option_gossip_queries/odd (optional)
option_var_onion_optin/odd (optional)
option_gossip_queries_ex/odd (optional)
option_static_remotekey/odd (optional)
option_payment_secret/even (compulsory)
option_basic_mpp/odd (optional)
option_anchor_outputs/odd (optional)
option_shutdown_anysegwit/odd (optional)
option_onion_messages/odd (optional)
option_unknown_54/odd (optional)
```


Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-23 15:02:05 +02:00
Rusty Russell
e3ae7883bb channeld: rename onion_message to obs_onion_message.
This splits the existing old-spec pathways to prepare for the
new ones.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-22 06:44:26 +09:30
Rusty Russell
24536c5561 common/autodata: use instead of ccan/autodata
This means it needs to be linked ~everywhere.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-21 18:04:43 +02:00
Rusty Russell
7401b26824 cleanup: remove unneeded includes in C files.
Before:
 Ten builds, laptop -j5, no ccache:

```
real	0m36.686000-38.956000(38.608+/-0.65)s
user	2m32.864000-42.253000(40.7545+/-2.7)s
sys	0m16.618000-18.316000(17.8531+/-0.48)s
```

 Ten builds, laptop -j5, ccache (warm):

```
real	0m8.212000-8.577000(8.39989+/-0.13)s
user	0m12.731000-13.212000(12.9751+/-0.17)s
sys	0m3.697000-3.902000(3.83722+/-0.064)s
```

After:
 Ten builds, laptop -j5, no ccache: 8% faster

```
real	0m33.802000-35.773000(35.468+/-0.54)s
user	2m19.073000-27.754000(26.2542+/-2.3)s
sys	0m15.784000-17.173000(16.7165+/-0.37)s
```

 Ten builds, laptop -j5, ccache (warm): 1% faster

```
real	0m8.200000-8.485000(8.30138+/-0.097)s
user	0m12.485000-13.100000(12.7344+/-0.19)s
sys	0m3.702000-3.889000(3.78787+/-0.056)s
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
ea30c34d82 cleanup: remove unneeded includes in header files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
1ab2b4a3fb devtools/reduce-includes.sh: brute-force script to remove includes.
Tries one at a time.  Slow, but doesn't need to be run often.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-17 09:43:22 +09:30
Rusty Russell
740afb822c common/initial_channel: use channel_type instead of individual option bools.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell
cb22015b2a common/channel_type: wrapper for generated 'struct channel_type'.
We make it a first-class citizen internally, even though we won't use
it over the wire (at least, non-experimental builds).  This scheme
follows the latest draft, in which features are flagged compulsory.

We also add several helper functions.

Since uses the *even* bits (as per latest spec), not the *odd* bits,
we have some other fixups.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-13 15:53:23 +02:00
Rusty Russell
fb4edc2938 Makefile: update bolt version to include option_anchors_zero_fee_htlc_tx.
This touches a lot of text, mainly to change "if `option_anchor_outputs`"
to "if `option_anchors`"

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-09-09 12:04:48 +09:30
Rusty Russell
1b2c6964fd Makefile: update spec version
This includes the new bolt11 test vectors, and also removes the
requirement that HTLCs be less than 2^32 msat.  We keep that for now
because Electrum enforced it on receive: in two releases we will stop
that too.

So no longer warn about needing mpp in that case either.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Deprecated: Protocol: No longer restrict HTLCs to
2021-09-09 12:04:48 +09:30
Rusty Russell
cf1dd779d1 gossmap: don't spew to stderr, include counter for callers.
Fixes: #4722
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Plugins: don't drop complaints about silly channels to stderr.
2021-08-24 06:42:32 +09:30
Rusty Russell
7769903f1a bolt12: import latest spec (timestamp -> created_at).
@shesek points out that we called this field created_at in bolt11 decode,
which makes more sense anyway.

Changelog-EXPERIMENTAL: bolt12 decode `timestamp` field deprecated in favor of new name `created_at`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-21 13:27:27 -04:00
niftynei
265f960cfe liquidity-ads: persist channel blockheight states to disk
Adds new tables to database, backfills, basically copies the fee_rates
state machine for channeld.
2021-07-20 13:28:38 -04:00
niftynei
5989433810 lease_rates: pass in 'lease_expiry' and 'csv' to commitments/channel 2021-07-20 13:28:38 -04:00
niftynei
5b3c02f401 liquidity-ads: import from spec
Import the wires from spec. Here we go!
2021-07-20 13:28:38 -04:00
Rusty Russell
424ac84263 patch remove-payer-backcompat.patch 2021-07-14 14:33:18 -05:00
Rusty Russell
12dd3a439b offers: include the payer_note field in the invoice if it's in the request.
And list it in bolt12-cli decode, and the `decode` command.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Rusty Russell
18b6aa5e66 BOLT 12: update to include replace_invoice options.
We don't support it (yet), but update the spec to include it.

We include the previous field (recurrence_signature) as a shim for the
moment, for compat with existing nodes.  It's ugly, but next release
we'll stop *sending* it, then finally we'll stop accepting it!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2021-07-03 12:13:45 +09:30
Rusty Russell
f9fe814ea3 offers: signatures are now optional.
As per latest spec revision.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: BOLT12 offers can now be unsigned, for really short QR codes.
2021-07-03 12:13:45 +09:30
nathanael
8167af553e sort alphabetically 2021-06-28 14:43:47 +09:30
nathanael
bb64af1a33 add binaries to gitignore 2021-06-28 14:43:47 +09:30