Commit Graph

8665 Commits

Author SHA1 Message Date
Rusty Russell
7a93363b0b pytest: remove duplicate tests.
These tests appear in both test_misc.py and test_wallet.py.

I checked, and they're the only duplcates; they were moved in
0543149b89 then re-added (merge error?)
in 67fc8ffbcf.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
2696ec6ccb pytest: fix assumptions in test_withdraw_misc
First, simplify: amount is set to 1000000, but then we deposit 1000000 + 0.01btc
(i.e. 2000000), and we always use 2 * amount.  Just use a single constant to
make it clear.

Secondly, we assume that the wallet considers outputs spent as soon as
we created the tx: this will not be true once withdraw uses sendpsbt.
So, we generate blocks, but now sometimes withdraw will pick up change
txs, so we need to reserve them to avoid that messing our coinmovements.

Finally, we assumed the withdrawl order was BIP69, which becomes
variable.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
e5f7e5b577 pytest: allow variable-order coin_moves, and give more information on failure.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
f34d0b1cf5 plugins/txprepare: create simple variant for "withdraw".
This is a little lazy, but simpler than extracting the common parts
or making withdraw a plugin which calls txprepare (which should be
deprecated soon in favor of fundpsbt etc).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
8d4557938b plugins/txprepare: move functions higher.
Simple moveonly change for next patch.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
3b8c0a7397 sendpsbt: just reserve (maybe bump) inputs on send, don't mark spent.
Marking spent means if the transaction doesn't confirm for some
reason, the user will need to force a rescan to find the funds.  Now
we have timed reservations, reserving for (an additional) 12 hours
should be sufficient.

We also take this opportunity (now we have our own callback path)
to record the tx in the wallet only on success.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
c319f7697e wallet: wean fundpsbt off tx_spending_utxos.
We only need a subset of its functionality, so reproduce that here.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
182b761a46 walletrpc: remove now-unused txprepare parsing code.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
6b2a3f4dfb txprepare: remove old code, switch to plugin.
Some minor phrasing differences cause test changes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: txprepare reservations stay across restarts: use fundpsbt/reservepsbt/unreservepsbt
Changelog-Removed: txprepare `destination` `satoshi` argument form removed (deprecated v0.7.3)
2020-09-08 10:14:42 +09:30
Rusty Russell
843d3d6745 pytest: remove test_txprepare_restart
This tested that we unreserved everything on restart, which we no
longer do.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
3b1a226f67 txprepare: add txsend functionality.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
522459ba6e txprepare: add txdiscard functionality.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
3f57249f08 reservation: return FUNDING_STILL_SYNCING_BITCOIN if we might be missing funds.
This is useful, and also makes us pass the testsuite.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
74c6307c54 plugins/txprepare: use utxopsbt to handle utxos arg.
Very similar to fundpsbt, but takes 'utxos' instead of 'minconf'.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
8d5ff210bf txprepare: a plugin to replace txprepare (not yet active).
This uses `fundpsbt` and similar to simulate the txprepare command.
It has one difference (when complete), in that it those reservations
are now timed and don't get reset on restart.

It also doesn't have the restriction that `all` can only be used with
no other output, as I didn't realize that when I implemented it!

Note that change is now inserted in a random position, not sorted
into BIP69 order.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
96d260b787 wallet: fix reference to potentially-undefined variable.
If all is set, diff may not be:

    ==1230518== Use of uninitialised value of size 8
    ==1230518==    at 0x4C5781B: _itoa_word (_itoa.c:179)
    ==1230518==    by 0x4C736F4: __vfprintf_internal (vfprintf-internal.c:1687)
    ==1230518==    by 0x4C88119: __vsnprintf_internal (vsnprintf.c:114)
    ==1230518==    by 0x1D44B6: do_vfmt (str.c:66)
    ==1230518==    by 0x1D45A0: tal_vfmt_ (str.c:92)
    ==1230518==    by 0x1D4401: tal_fmt_ (str.c:44)
    ==1230518==    by 0x15D30F: fmt_amount_sat (amount.c:60)
    ==1230518==    by 0x15D338: fmt_amount_sat_ (amount.c:62)
    ==1230518==    by 0x178C45: type_to_string_ (type_to_string.c:35)
    ==1230518==    by 0x1B8F75: json_fundpsbt (reservation.c:394)
    ==1230518==    by 0x12D0EC: command_exec (jsonrpc.c:602)
    ==1230518==    by 0x12D598: rpc_command_hook_callback (jsonrpc.c:712)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 10:14:42 +09:30
Rusty Russell
c34c055d82 Makefile: use completely separate spec-derived files for EXPERIMENTAL_FEATURES
This avoids overwriting the ones in git, and generally makes things neater.

We have convenience headers wire/peer_wire.h and wire/onion_wire.h to
avoid most #ifdefs: simply include those.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 09:42:00 +09:30
Rusty Russell
d8e8426b52 Makefile: remove EXPERIMENTAL_FEATURES marker from generated files
We're going to make experimental versions of these completely separate files.

Also remove the dependency on the Makefile itself: it simply causes
unnecessary churn.  We can always force-rebuild when we change a rule.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-08 09:42:00 +09:30
Michael Schmoock
2816c08036 openchannel: test new hook chainable mechanics 2020-09-08 06:14:29 +09:30
Michael Schmoock
a3e18a6aba openchannel: make openchannel hook chainable
This will make the `openchannel_hook` chainable. Logic is as follows:
 - The first plugin that rejects terminates the chain
 - If more than one plugin uses the `close_to` parameter, take the first
   value and log_unusual for the others.

Changelog-Added: openchannel_hook is now chainable
2020-09-08 06:14:29 +09:30
Vincenzo Palazzo
6e16ee0eaa Alignment doc with the actual implementation
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
2020-09-07 11:07:34 +09:30
Christian Decker
037f7201be make: Do not delete deployable-lightning.pdf on maintainer-clean
It is not part of the default documentation generation, thus will show up in
the git diff every time a maintainer-clean is done. We could also add it to
the default documentation generation, however it requires some rather
heavy-weight latex dependencies, and the file hasn't changed in several years.
2020-09-07 11:03:29 +09:30
Christian Decker
0d5066c48b github: Mark some more files as derived / generated
Since we started committing generated files as well we can mark them as
derived so that they appear collapsed in the diff, reducing noise while
reviewing.
2020-09-07 11:03:29 +09:30
Christian Decker
6fc0433536 make: Do not clean sqlgen files in non-maintainer clean
Generating these requires python tooling which may not be present, so only
clean them in maintainer mode. Also remove `wallet/statements.po` which is no
longer used or generated at all.
2020-09-07 11:03:29 +09:30
Christian Decker
d7ff74e8f6 json: Only start parsing once we're sure we have a message
PR #3957 improved performance considerably, however we still look over the
entire message for the message separator. If instead we just look in the
incrementally read data, we remove the quadratic behavior for large messages.

This is safe since we then loop over the messages which would drain any
message separator from the buffer before we attempt the next read.

Changelog-Fixed: bcli: Significant speedups for block synchronization
2020-09-02 13:21:32 +02:00
Christian Decker
f2797f303a block: Compute the txids only once
We're not going to mutate transactions in a block, so computing the txids
every time we need them is a waste, let's compute them upfront and use them
afterwards.
2020-09-02 13:21:32 +02:00
Christian Decker
31c67f1392 bcli: Avoid tal_fmt when handling a raw hex block
We were using `tal_fmt` to truncate off the last byte of the
response (newline), which required an allocation, a call to `vsnprintf` and a
copy of the block contents. This being >2MB in size on mainnet was rather
expensive.

We now just signal the end of the string by overwriting the trailing newline,
and stealing directly onto the result.
2020-09-02 13:21:32 +02:00
Rusty Russell
e5e0ccaba0 wallet/Makefile: fix typo in rule to regen sqlgen files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell
8670cb333f wallet/Makefile: fix wallet_clean vs wallet-clean typo.
And remove toplevel target we no longer try to support.

Reported-byL @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell
82f2f43425 Generated files: more merge fallout.
We changed the generation templates!

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell
cd4c99d028 .gitignore: don't ignore checked-in files.
It doesn't have any effect, but would confuse if we added one.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 11:10:21 +09:30
Rusty Russell
e4b5a1e0e8 dev: option to set what features we offer.
And use it to smoke-test !option_static_remotekey.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 10:52:49 +09:30
Rusty Russell
3a2191fcfe wallet: db_bind_talarr and db_column_talarr helpers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
39e4796ae3 json_command: command_fail_badparam helper.
It's common to want to complain a token is not what we expected.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
aa441e3b27 tools/generate-wire.py: fix loop logic for towire_xxx_array
We have to handle singletons which are arrays of variable-length entries:
this needs to be a ptr-to-ptr.

```C
struct blinded_payinfo {
        u32 fee_base_msat;
        u32 fee_proportional_millionths;
        u16 cltv_expiry_delta;
        u8 *features;
};
```

Before:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo *blindedpay;
```

After:
```C
struct tlv_invoice_tlvs {
...
	struct blinded_payinfo **blindedpay;
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
897c53ce1c tools/generate-wire.py: fix loop logic for towire_xxx_array
has_len_fields() doesn't cover our blacklist of variable types, so if
we have an array of them, this logic is wrong.  This happens in the
the bolt13 patch:

```C
struct tlv_offer_tlvs_blindedpath {
        struct pubkey blinding;
        struct onionmsg_path **path;
};
```

Before:
wire/gen_bolt13_tlv.c:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path + i);
```
After:
```C
		for (size_t i = 0; i < tal_count(r->blindedpath->path); i++)
		towire_onionmsg_path(&ptr, r->blindedpath->path[i]);
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
76813d6f90 tools/generate-wire.py: fix varsize assignment.
Code like this is suspicious:

	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);

In fact, it is a memory leak since we copy and don't free the fromwire
result.  Really, field_0 should be a pointer.

We don't hit this case (yet!) in spec-generated code, but I did for
bolt13.

Here's the difference in gen_test output:

```patch
  diff -ur /tmp/before/test/gen_test.c /tmp/after/test/gen_test.c
  --- /tmp/before/test/gen_test.c	2020-05-07 16:23:31.651611235 +0930
  +++ /tmp/after/test/gen_test.c	2020-05-07 16:20:54.232574482 +0930
  @@ -214,12 +214,12 @@
   static void towire_subtype_varsize_struct(u8 **p, const struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  -	towire_test_features(p, &subtype_varsize_struct->field_0);
  +	towire_test_features(p, subtype_varsize_struct->field_0);
   }
   static void fromwire_subtype_varsize_struct(const u8 **cursor, size_t *plen, struct subtype_varsize_struct *subtype_varsize_struct)
   {
   
  - 	subtype_varsize_struct->field_0 = *fromwire_test_features(subtype_varsize_struct, cursor, plen);
  + 	subtype_varsize_struct->field_0 = fromwire_test_features(subtype_varsize_struct, cursor, plen);
   }
   
   /* SUBTYPE: SUBTYPE_VAR_LEN */
  @@ -373,7 +373,7 @@
   
   	ptr = tal_arr(ctx, u8, 0);
   
  -	towire_test_features(&ptr, &r->tlv3->features);
  +	towire_test_features(&ptr, r->tlv3->features);
   
   	towire_amount_msat(&ptr, r->tlv3->amount_msat_1);
   
  @@ -385,7 +385,7 @@
   	struct tlv_test_n1 *r = vrecord;
   
   	r->tlv3 = tal(r, struct tlv_test_n1_tlv3);
  -	r->tlv3->features = *fromwire_test_features(r->tlv3, cursor, plen);
  +	r->tlv3->features = fromwire_test_features(r->tlv3, cursor, plen);
   	r->tlv3->amount_msat_1 = fromwire_amount_msat(cursor, plen);
   	r->tlv3->amount_msat_2 = fromwire_amount_msat(cursor, plen);
   }
  @@ -824,11 +824,11 @@
   
   	towire_test_short_id(&ptr, &r->tlv3->subtype);
   
  -	towire_subtype_var_len(&ptr, &r->tlv3->varlen_subtype);
  +	towire_subtype_var_len(&ptr, r->tlv3->varlen_subtype);
   
  -	towire_subtype_var_assign(&ptr, &r->tlv3->varlen_assigned);
  +	towire_subtype_var_assign(&ptr, r->tlv3->varlen_assigned);
   
  -	towire_subtype_varlen_varsize(&ptr, &r->tlv3->test_sbt_varlen_varsize);
  +	towire_subtype_varlen_varsize(&ptr, r->tlv3->test_sbt_varlen_varsize);
   
   		for (size_t i = 0; i < 2; i++)
   		towire_u32(&ptr, r->tlv3->arr_assign[i]);
  @@ -868,9 +868,9 @@
   
   	r->tlv3 = tal(r, struct tlv_test_n3_tlv3);
   	fromwire_test_short_id(cursor, plen, &r->tlv3->subtype);
  -	r->tlv3->varlen_subtype = *fromwire_subtype_var_len(r->tlv3, cursor, plen);
  -	r->tlv3->varlen_assigned = *fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  -	r->tlv3->test_sbt_varlen_varsize = *fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_subtype = fromwire_subtype_var_len(r->tlv3, cursor, plen);
  +	r->tlv3->varlen_assigned = fromwire_subtype_var_assign(r->tlv3, cursor, plen);
  +	r->tlv3->test_sbt_varlen_varsize = fromwire_subtype_varlen_varsize(r->tlv3, cursor, plen);
   		for (size_t i = 0; i < 2; i++) {
   		u32 tmp;
   		tmp = fromwire_u32(cursor, plen);
  diff -ur /tmp/before/test/gen_test.h /tmp/after/test/gen_test.h
  --- /tmp/before/test/gen_test.h	2020-05-07 16:23:30.399617108 +0930
  +++ /tmp/after/test/gen_test.h	2020-05-07 16:20:52.912584680 +0930
  @@ -45,7 +45,7 @@
           struct test_short_id field_1;
   };
   struct subtype_varsize_struct {
  -        struct test_features field_0;
  +        struct test_features *field_0;
   };
   struct subtype_var_len {
           struct test_short_id *field_2;
  @@ -60,15 +60,15 @@
           struct test_short_id field3[2];
   };
   struct tlv_test_n1_tlv3 {
  -        struct test_features features;
  +        struct test_features *features;
           struct amount_msat amount_msat_1;
           struct amount_msat amount_msat_2;
   };
   struct tlv_test_n3_tlv3 {
           struct test_short_id subtype;
  -        struct subtype_var_len varlen_subtype;
  -        struct subtype_var_assign varlen_assigned;
  -        struct subtype_varlen_varsize test_sbt_varlen_varsize;
  +        struct subtype_var_len *varlen_subtype;
  +        struct subtype_var_assign *varlen_assigned;
  +        struct subtype_varlen_varsize *test_sbt_varlen_varsize;
           /*  array assigtest_nable */
           u32 arr_assign[2];
           /*  array structs */
  Binary files /tmp/before/test/gen_test.o and /tmp/after/test/gen_test.o differ
  Binary files /tmp/before/test/run-test-wire and /tmp/after/test/run-test-wire differ
  Binary files /tmp/before/test/run-test-wire.o and /tmp/after/test/run-test-wire.o differ
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:46:37 +09:30
Rusty Russell
be0038e49b Makefile: include common objects in ALL_C_SOURCES.
They didn't have correct dependencies without this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:30:25 +09:30
Rusty Russell
5bedcbeed7 doc: regenerate lightningd-config man page.
Merged as SHA256STAMP was going in.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 09:30:25 +09:30
Rusty Russell
0c7d04bd98 libplugin-pay: fix default CLTV.
This was changed recently, but without a bolt quote, we didn't find
this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-02 06:59:13 +09:30
Rusty Russell
74e960c20b pytest: increase channel sizes so they reach min capacity with option_anchor_outputs
With a feerate of 7500perkw and subtracting 660 sats for anchors, a
20,000 sat channel has capacity about 9800 sat, below our default:

	You gave bad parameters: channel capacity with funding 20000sat, reserves 546sat/546sat, max_htlc_value_in_flight_msat is 18446744073709551615msat, channel capacity is 9818sat, which is below 10000000msat

So bump channel amounts.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell
9fabf614f4 pytest: run test_opening_tiny_channel with option_anchor_outputs.
It's now easy to fix.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell
9b8f08a8a0 lightningd: return min-capacity-sat to its intended purpose.
And document exactly what it does: insist that an HTLC can pass of
this value (module assumptions of feerate).

Note that we remove the "is_opener" test from the capacity calculation
for anchor fees: it doesn't matter which side it is, someone has to pay
for anchor fees to it deducts from capacity.

This change breaks the test, which we rewrite.

Changelog-Changed: config: `min-capacity-sat` is now stricter about checking usable capacity of channels.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-09-01 12:27:39 -05:00
Rusty Russell
005fbbe4a6 travis: don't suppress make output.
It's far less verbose than it used to be!  And it's occasionally useful.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
b3f6a0e78d Travis: don't install requirements.txt if we're just building.
This tests our instructions in INSTALL.md, which say you only need
requirements.txt for development or running tests, not building.

Also removes the unused SOURCE_CHECK_ONLY flag.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
390c6470ad configure: we don't need mako any more.
And guard the mako-specific tests with a check instead.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
2c9ad7ea29 external/Makefile: fix mkdir rule for libbacktrace.a
The $(TARGET_DIR) may not exist: use mkdir -p.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
b3d5220da5 tools/refresh-submodules.sh: don't exit, but wait if called in parallel.
Otherwise make thinks we're done, and we can get errors.  Include
primitive code if we abort build halfway and leave .refresh-submodules dir.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
95a0249272 external/Makefile: fix build from fresh git clone.
```
make: *** No rule to make target 'external/gheap/gheap.h', needed by 'bitcoin/chainparams.o'.  Stop.
make: *** Waiting for unfinished jobs....
```

Just simplify the Makefile to make all the external headers we use
dependent on submodcheck.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2020-08-31 21:33:26 -05:00
Rusty Russell
3c6af3efb4 Makefile: commit and preserve all the wiregenerated files.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Changed: Build: we no longer require extra Python modules to build.
2020-08-31 21:33:26 -05:00