Commit Graph

5828 Commits

Author SHA1 Message Date
trueptolemy
8a2312565c DB: add failcode field in forwarded_payments table 2019-05-03 11:45:57 +02:00
trueptolemy
1c1fb1ca51 wallet: add failcode field in forwarding struct
It's a optional field. We only set failcode for FORWARD_LOCAL_FAILED case, and set this field with 0 in other case.
2019-05-03 11:45:57 +02:00
trueptolemy
473dbd4921 wallet: add 'FORWARD_LOCAL_FAILED' as a new forward_status type
In FORWARD_LOCAL_FAILED case, we can get the failcode and can't get the resolve time, that is different with FORWARD_FAILED case.
2019-05-03 11:45:57 +02:00
Rusty Russell
ec658c1f89 status: suppress status_trace/status_debug messages if queue too long.
We can be spammy, which is good for tests, but bad for our simple message queue.
This avoids breaking our tests but also avoid the worst case (1M entries and counting!)
for gossip status messages in the case where we're syncing a large peer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-03 11:42:48 +02:00
lisa neigut
e3bac6c165 wire-gen: patch in the wire/gen_*_experimental_wire files
use the `wire/gen_*experimental_wire` patch file and apply it to
existing wire message set.

empty for now, but will be generated shortly.
2019-05-02 01:20:41 +00:00
lisa neigut
39153f3122 wire-gen: use staging file for generated wire csvs
for now we straight copy the `extracted_peer_wire_csv` over into the
file that is used to generate the .c/.h files; in the future
we'll use this destination file as a way to modify the
`gen_peer_wire_csv`s from a patch.
2019-05-02 01:20:41 +00:00
Rusty Russell
e51a13975f Makefile: don't generate CSVs unless explicitly told to.
ANd rename them: every other "gen" file can be discarded and rebuilt.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:20:41 +00:00
Rusty Russell
dddfdc7f10 pytest: add dev hack for testing option_upfront_shutdown_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:18:33 +00:00
Rusty Russell
99a0345ca8 channeld: enforce option_upfront_shutdown_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:18:33 +00:00
Rusty Russell
cfebe66762 openingd: support receipt of upfront_shutdown_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:18:33 +00:00
Rusty Russell
7ead29b695 db: add support for remote end specify option_upfront_shutdown_script.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-02 01:18:33 +00:00
Christian Decker
aa9284eaa3 base58: Simplified the address parsing
We were deciding whether an address is a testnet address or not in the parser,
and then checking whether it matches our expectation outside as well. This
just returns the address version instead, and still checks it against our
expectation, but without having the parser need to know about address types.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-05-01 12:37:30 +02:00
Christian Decker
5d185f43a8 base58: Add function to just parse a ripemd160 hash
This is an intermediate step since the only difference between p2pkh and p2sh
is the argument that the parsing functions take, and parsing twice for that
reason alone is quite useless.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-05-01 12:37:30 +02:00
Christian Decker
6d618511fc chainparams: Add p2pkh and p2sh versions to chinparams
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Co-authored-by: bisoge <bisoge@gmx.de>
2019-05-01 12:37:30 +02:00
Rusty Russell
0ca0db765a gossipd: fix crash if we truncate store.
Entries we've already loaded expect to exist in the store.  We could go
back and remove them all, but instead just truncate at the known-good
point.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-05-01 11:59:12 +02:00
Christian Decker
c96cc8a444 pytest: Fix the flaky test_plugin_connected_hook test
It was a bit eager :-)

Signed-off-by: Christian Decker <@cdecker>
2019-04-30 17:56:09 -05:00
Christian Decker
eeecfc983c plugin: Remove autopatch argument from helloworld plugin
Suggested-by: Rusty Russell <@rustyrussell>
2019-04-30 17:56:09 -05:00
Christian Decker
88f425fcc5 pytest: Make directory cleanup robust against setup failures
We were triggering a second exception in the directory cleanup step by
attempting to access a field that'd only be set upon entering the test code
itself. That error did not contribute to the problem resolution, so now we
check whether that field is set before accessing it.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-30 17:56:09 -05:00
Christian Decker
20a2e3ca6d git: Ignore the create-gossipstore tool binary 2019-04-30 17:56:09 -05:00
Christian Decker
be853f563a wallet: Clamp maxheight to positive number for large minconf
Fixes #2518

Signed-off-by: Christian Decker <decker.christian@gmail.com>
Changelog-fixed: `minconf` no longer gets wrapped around for large values, which was causing funds with insufficient confirmations to be selected.
2019-04-30 17:52:52 -05:00
Christian Decker
e40b7c5584 pytest: Add test_minconf_withdraw to reproduce issue #2518
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-04-30 17:52:52 -05:00
GreenAddress
fb07265663 remove libbase58, use base58 from libwally (#2594)
* remove libbase58, use base58 from libwally

This removes libbase58 and uses libwally instead.

It allocates and then frees some memory, we may want to
add a function in wally that doesn't or override
wally_operations to use tal.

Signed-off-by: Lawrence Nahum lawrence@greenaddress.it
2019-04-30 23:07:31 +02:00
arowser
890379d8f1 correct format long long unsigned int on 32bits linux 2019-04-30 13:53:15 +02:00
Michael Schmoock
f99c461fed fix: pylightning msat round multiply and division
Currently, when a multiplication operator is invoked that
does not result in an even integer result but a floating result,
the pylightning code will raise an exception:

Millisatoshi must be string with msat/sat/btc suffix or int

This is because the internal float result will be used as
contructor argument like this:  return Millisatoshi(10000.5)

This happens especially on fee calculations where small uneven amounts
are calculated.
2019-04-29 19:10:56 +02:00
Christian Decker
f82e779f71 check: Run check-manpages in /tmp/ instead of the user config
If we have plugins configured in the default config location, then
`check-manpages` may fail due to plugin cli options being added to the
`--help` output.
2019-04-25 21:16:37 -05:00
Rusty Russell
a552130d3d lightningd: make sure we correctly clear connections from connectd on error.
Without this, the connect command hangs in one of my branches.  This logic
is from the old days when gossipd handled connections, and we wanted
to make sure it didn't hang up on this client due to the error.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-25 21:15:17 -05:00
Christian Decker
62e1423968 fixup! tools/bench-gossipd.sh: make it work (where possible) with DEVELOPER=0 2019-04-24 13:46:39 -05:00
Rusty Russell
14ef3e9565 create-gossipstore: actually use the CSV values.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-24 13:46:39 -05:00
Rusty Russell
b248bb155a tools/bench-gossipd.sh: make it work (where possible) with DEVELOPER=0
Some tests require dev support, but the rest can run.  We simplify
the gossip_store output so it's the same in non-dev mode too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-24 13:46:39 -05:00
Rusty Russell
1ccfbef4e7 tools/bench-gossipd.sh: make sure we wait for gossipd to startup sync.
It can take a while if bitcoind has the regtest chain, and grossly
distorts our benchmarks!

Reported-by: Joe Netti <jnetti@blockstream.io>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-24 13:46:39 -05:00
Joe Netti
902bb22a92 devtools/create-gossipstore: cleanups
added sanity check to make sure scid of csv is the same as scid in gossip.
Revised style, mem allocation, and error checks

[ Minor fixups, and updated benchmark script -- RR ]

With data.tar.gz: 456609740 Apr  2 12:33

store_load_msec:35300-42354(37118.2+/-2.7e+03)
vsz_kb:582832
store_rewrite_sec:12.700000-13.430000(12.988+/-0.27)
listnodes_sec:3.000000-3.160000(3.076+/-0.057)
listchannels_sec:30.790000-31.690000(31.03+/-0.34)
routing_sec:0.00
peer_write_all_sec:63.640000-67.860000(66.294+/-1.4)
2019-04-24 13:46:39 -05:00
lisa neigut
8326f6a35c wire-gen: put tlv's after subtypes
in the output, we need tlv's to come after the subtype declarations
and structs so that a tlv can use a subtype without problems
2019-04-24 14:16:15 +02:00
lisa neigut
560f45cd7f wire generator: reverse subtype struct order
nested subtypes need to come first in the struct declaration. we
reverse the order that we read them off so that this requirement
is met
2019-04-24 14:16:15 +02:00
lisa neigut
0ce287d52e tlvs: include correct reference for nested objects 2019-04-24 14:16:15 +02:00
Hampus Sjöberg
3028964687 pylightning: Fixing incompatibility issue with Python 2.7
Global open doesn't have argument `encoding` in Python 2.7.
open from the io package is needed.
2019-04-23 23:39:57 +02:00
Rusty Russell
ca7864f2f3 invoice_hook: remove nested result.
I misunderstood the API, this ended up nesting a result inside the JSON-RPC
result.

No concerns about backwards compatibility since this is so new.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-23 15:26:42 +02:00
Rusty Russell
7f7ad4f89f connected_hook: allow hook to specify an error message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-23 15:26:42 +02:00
Rusty Russell
a314bc62fc lightningd: remove deserialize step for plugin hooks.
This seems like overkill, at least for now.  Handling the JSON
inline is clearer, for the existing examples at least.

We also remove the dummy hook, rather than fix it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-23 15:26:42 +02:00
Rusty Russell
d3c312860d lightningd: if a plugin gives a JSON-RPC error, print it.
We currently just segfault when the deserialize function to access resulttok.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-23 15:26:42 +02:00
Lawrence Nahum
a9e71d76b5 update libwally to latest version 0.6.9
Also removes a workaround caused by bug in libwally (!95) which has
been fixed.

Signed-off-by: Lawrence Nahum <lawrence@greenaddress.it>
2019-04-23 15:18:57 +02:00
Rusty Russell
0fc42415c2 gossipd/routing: remove BFG implementation.
Now we can benchmark, and remove 500 bytes per node.

MCP results from 5 runs, min-max(mean +/- stddev):
	store_load_msec:35093-37907(36146+/-1.1e+03)
	vsz_kb:555168
	store_rewrite_sec:12.120000-13.750000(12.7+/-0.6)
	listnodes_sec:1.270000-1.370000(1.322+/-0.039)
	listchannels_sec:29.770000-31.600000(30.82+/-0.64)
	routing_sec:0.00
	peer_write_all_sec:63.630000-67.850000(65.432+/-1.7)

MCP notable changes from pre-Dijkstra (>1 stddev):
	-vsz_kb:577456
	+vsz_kb:555168
	-routing_sec:60.70
	+routing_sec:12.04

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
cfdb012b30 gossipd: re-add fuzz logic to routing.
Do it inside the can_reach() function, which is less optimal for BFG
which does 20 ops on the same channel, but fine for Dijkstra.

This does have a measurable cost, so we might want to use
non-cryptographic fuzz in future:

$ gossipd/test/run-bench-find_route 100000 100:

Before:
	100 (100 succeeded) routes in 100000 nodes in 97346 msec (973461784 nanoseconds per route)

After:
	100 (100 succeeded) routes in 100000 nodes in 113381 msec (1133813412 nanoseconds per route)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
e197956032 gossipd/routing: Iterate on Dijkstra when route is too long.
If a route is too long, we try to bias Dijkstra towards choosing a
shorter route by adding a per-hop cost.  We do a naive "shortest path"
pass, then using that cost as a ceiling on per-hop cost, we do a
binary search.

There are some subtleties: we use risk rather than total as our
counter field (we normally bias this by 1 anyway, so it's easy to make
that a variable), and we set riskfactor to a mimimal value once we're
iterating.  It's good enough to get a solution, we don't need to do a
2-dimensional search on riskfactor and riskbias.

Of course, this is extremely slow if we hit it on our benchmark,
though it doesn't happen in a more realistic network:

$ gossipd/test/run-bench-find_route 100000 100:

Before:
	100 (79 succeeded) routes in 100000 nodes in 25341 msec (253412314 nanoseconds per route)

After:
	100 (100 succeeded) routes in 100000 nodes in 97346 msec (973461784 nanoseconds per route)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
f8ffae837d gossipd: speed Dijkstra a little.
Our uintmap can be a little slow with all the reallocation, so leave
NULL entries and walk to find the first one.  Since we don't clean
them up, keep a cache of where the min non-all-NULL value is in the
heap.

It's clearer benefit on really large tests, so here's 1M nodes:

Comparison using gossipd/test/run-bench-find_route 1000000 10:

Before:
	10 (10 succeeded) routes in 1000000 nodes in 91995 msec (9199532898 nanoseconds per route)

After:
	10 (10 succeeded) routes in 1000000 nodes in 20605 msec (2060539287 nanoseconds per route)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
7caa37f0f1 gossipd: implement Dijkstra.
Use a uintmap as our minheap.

Note that Dijkstra can give overlength routes, so some checks are disabled.

Comparison using gossipd/test/run-bench-find_route 100000 10:

Before:
	10 (10 succeeded) routes in 100000 nodes in 120087 msec (12008708402 nanoseconds per route)
After:
	10 (10 succeeded) routes in 100000 nodes in 2269 msec (226925462 nanoseconds per route)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
4d84a436f5 gossipd: temporarily disable fuzz in routing.
This allows precise comparison between Dijkstra and Bellman-Ford without
worrying about fuzz.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
594af8049b gossipd: extract common functionality.
This will be needed by Dijkstra as well.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
6dfa46d65a gossipd/test: add test for handling overlong routes.
This is a weakness with Dijkstra, so write an explicit unit test that
we can find a short enough (but more expensive) route.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
981fa68fe0 tools/bench-gossipd.sh: fix routing.
Channels have a htlc_minimum_msat of 10000, which is why we didn't
find routes.

This makes a significant speed drop:

-routing_sec:26.940000-27.990000(27.616+/-0.39)
+routing_sec:60.70

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:33:09 +00:00
Rusty Russell
d0aefac5c3 wallet: fix crash when we have height requirement.
68fe5eacde introduced a skip in the iteration
of the available funds, which means utxos[i] may be off the end of utxos.

Reported-and-debugged-by: @nitramiz
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-04-18 06:06:30 +00:00