Installing pytest through pip3 (at least sometimes) doesn't create a script.
This means calling `which` won't work.
Changed configure so that it can also test if the module is present by calling python/python3.
Change the error message for when pytest can't be found, so that it's clear to the user `configure` must be ran again after installing pytest.
It's an array: we were only saving the single element; if there was more than
one changed HTLC we'd get a bad signature!
The report in #1907 is probably caused by the other side re-requesting
something we considered already finalized; to avoid this particular error,
we should set the field to NULL if there's no last_sent_commit.
I'm increasingly of the opinion we want to just save all the update
packets to the db and blast them out, instead of doing this
second-guessing dance.
Fixes: #1907
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It no longer has any effect on tal_len(), but it *does* give file and line
of allocations which is much nicer for tracking memory leaks!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is just copying most of gossipd/gossip.c into connectd/connect.c.
It shares the same wire format as gossipd during transition, and changes
are deliberately minimal.
It also has an additional message 'connect_reconnected' which it sends
to the master daemon to tell it to kill a peer; gossipd relied on
closing the gossipfd to do this, but connectd doesn't maintain an fd
with remote peers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Different versions of git have different default lengths. This uses a fixed
length hash prefix.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Make should only run configure if config.var already exists:
$ make
./configure --reconfigure
./configure: 65: .: config.vars: not found
./configure --reconfigure
./configure: 65: .: config.vars: not found
Makefile:179: recipe for target 'ccan/config.h' failed
make: *** [ccan/config.h] Error 2
They are quite chatty and fail very rarely, so let's not have them spam our
logs. Failures should print to stderr anyway.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Previously, by uncommenting NO_VALGRIND in Makefile, the variable
was not propagated and the pytest children processes would still
try to use Valgrind.
Setting these options could fail the `pytest` target if `pytest-test-groups` is
not installed.
Fixes#1508
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is the first example of the py.test style fixtures which should allow us to
write much cleaner and nicer tests.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Since we include the tail of the version in the default aliasname with
DEVELOPER, we want to see more of it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had quite a few users running into issues because the `--dev-xyz` options and
`dev-xyz` RPC calls were available. Before a release we should make sure that
the default compilation flags are safe.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Retrying gives spurious failures, since we see transactions from previous
runs. That makes it near impossible to diagnose the actual problem.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, this gets some MacOS fixes from #1327.
It also includes a major intmap update which fixes corner cases in traversals,
and requires ccan/bitops.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This lets us clearly mark transition features, in a way that they can
be removed after 0.6 is released.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is how the error before the fix looked like on Arch Linux with
GNU libtool 2.4.6.40-6ca5-dirty:
$ make installcheck
...
make[4]: Entering directory '/home/jasan/lightning/external/libwally-core/src/secp256k1'
/usr/bin/mkdir -p 'testinstall/home/jasan/lightning/external/libwally-core/..'
/bin/sh ./libtool --mode=install /usr/bin/install -c libsecp256k1.la 'testinstall/home/jasan/lightning/external/libwally-core/..'
Usage: /home/jasan/lightning/external/libwally-core/src/secp256k1/libtool [OPTION]... [MODE-ARG]...
Try 'libtool --help' for more information.
libtool: error: 'testinstall/home/jasan/lightning/external/libwally-core/..' must be an absolute directory name
make[4]: *** [Makefile:910: install-libLTLIBRARIES] Error 1
make[4]: Leaving directory '/home/jasan/lightning/external/libwally-core/src/secp256k1'
make[3]: *** [Makefile:1253: install-exec-recursive] Error 1
make[3]: Leaving directory '/home/jasan/lightning/external/libwally-core/src'
make[2]: *** [Makefile:429: install-exec-recursive] Error 1
make[2]: Leaving directory '/home/jasan/lightning/external/libwally-core'
make[1]: *** [external/Makefile:41: external/libwallycore.a] Error 2
rm external/libwally-core/src/secp256k1/libsecp256k1.la
make[1]: Leaving directory '/home/jasan/lightning'
make: *** [Makefile:430: installcheck] Error 2
Then there was linking error which needed to be fixed by correcting
the idea of installing external libraries to a particular directory.
This should solve what @icota wanted in https://github.com/ElementsProject/lightning/pull/1035 in a much cleaner way.
In particular, this allows you to say what configurator should use, independent
of what other compilation should use, and reverts the '-static' which broke
MacOS.
Fixes: #1059
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
As of version 5.0 Android requires all dynamically linked executables to support PIE. This allows programs to be loaded at a different addresses, making it harder for attackers to target.
Enable with PIE=1
When cross compiling it's important that the resulting config.h reflects the platform we are building for and not the one we are building on.
Otherwise we end up with a config.h that defines headers that are not there on the target platform, wrong endnianness and so on.
The -static flag is there to be able to easily run the configurator test executables on the build machine with qemu-*.
E.g. Without the -static flag the resulting dynamically linked ARM executables complain about the lack of linker (/lib/ld-linux-armhf.so or /system/bin/linker for Android), since these files are not usually available on the build machine building statically avoids this problem and results in a proper config.h for cross compiling.
This doesn't cover external libs in general (which *could* be effected by
CFLAGS), but at least all our own objects are rebuilt.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Adds two new environment variables TEST_GROUP_COUNT and TEST_GROUP to
split the integration tests into groups and run only a selected group.
This allows us to increase the TEST_GROUP_COUNT and add a new
TEST_GROUP to avoid bumping up against the time limit when running in
valgrind.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The py.test unit test runner offers a number of more advanced features
than simply running using unittest.main. In particular it allows us to
capture a tests output and print it if it fails. This change checks
whether we have pytest available and if yes, enables verbose tests and
runs using pytest. This'll give the usual experience (with colors!)
and show us the stdout if a test fails making travis a lot easier to
handle.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
And nail "make check-source" to that specific version (which is a commit id,
not a branch name, so needs a different syntax for git).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is a bit messier than I'd like, but we want to clearly remove all
dev code (not just have it uncalled), so we remove fields and functions
altogether rather than stub them out. This means we put #ifdefs in callers
in some places, but at least it's explicit.
We still run tests, but only a subset, and we run with NO_VALGRIND under
Travis to avoid increasing test times too much.
See-also: #176
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
update-mocks was broken, since it assumed the daemon/ directory.
We now use "make" directly to build the test file and harvest errors,
and are more robust if it simply doesn't compile (ie. fails, but no
linker errors).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
You will want to 'make distclean' after this.
I also removed libsecp; we use the one in in libwally anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Also, we split the more sophisticated json_add helpers to avoid pulling in
everything into lightning-cli, and unify the routines to print struct
short_channel_id (it's ':', not '/' too).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To avoid everything pulling in HTLCs stuff to the opening daemon, we
split the channel and commit_tx routines into initial_channel and
initial_commit_tx (no HTLC support) and move full HTLC supporting versions
into channeld.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This moves all the non-legacy blackbox testing into python.
Before:
real 10m18.385s
After:
real 9m54.877s
Note that this doesn't valgrind the subdaemons: that patch seems to cause
some issues in the python framework which I am still chasing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The Dockerfile is now stored in contrib and built using the Docker
Hub. This allows us to simply pull in the finished image from the hub
instead of having to build it ourself. Should shave off about 2
minutes from the build time.
I also switched to running the individual build and check steps in
their own containers, but on the same volume, so travis can group the
commands and run them independently.
libwally's tools/cleanup.sh doesn't actually remove files if it can't
run make, so do that manually. Also clear some other cruft.
Also, we weren't deleting wire/gen_onion_wire.c in "make clean".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We had a hack for 'struct rval' in protobuf_convert.h; make an
explicit header and put it in bitcoin/preimage.h. It's not really
bitcoin-specific, but it's better than having bitcoin/script depend on
an external header.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can't run them in parallel, but we can at least have 'make check'
run them all.
Developers should be running "make check-source && make check".
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The API formalizes how daemons should report their statuses back to
the main lightningd. It's a simple write API, which includes tracing
support (currently it always sends traces, later it could send iff
there's a failure, for example).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Setting the environment variable `COVERAGE=1` allows us to turn
coverage measurements on. This produces a number of auxiliary gcno and
gcda files which can then be converted into coverage reports.
I know line coverage and branch coverage is not a perfect metric, but
it might get us some more visibility into where more tests might be
needed.
The object file should not be built inside the submodule, as that can
confuse git.
Not everything depends on the libbase58 header (CCAN doesn't), so
move that to the everything-else depends line.
The BITCOIN_SRC etc should also move to bitcoin/Makefile, but that's
a bigger change.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We add a "dev-restart" command which causes the daemon to close fds
and exec itself. Then we do it after every command, with the caveat
that we always send a commit before newhtlc, because if not committed,
that is forgotten. Fulfillhtlc and failhtlc get resent, since they're
idempotent.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
To do this we keep an order counter so we know how to retransmit. We
could simply keep old packets, but this is a little clearer for now.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We no longer need it anywhere. This simplifies things to the point where
we might as well just not include dust outputs as we go, rather than
explicitly removing them, which gets rid of remove_dust.c as well.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Add Makefile target update-secp256k1, and run it.
The only API change is that len is now an IN-OUT parameter to serialization
functions.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If upgrade changes configure flags, 'make distclean' can fail as it
tries to reconfigure. Deleting secp256k1/libsecp256k1.la forces a
full autogen/configure/make cycle.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's a more logical name, and a more logical place. We change
"funding" to "channel" in the remaining exposed symbols, too.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This would have revealed the previous breakage (and I tested that!),
plus now we test negotiate on closing.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
While useful for testing, it doesn't make sense to have an explicit commit
command; we should commit whenever there are outstanding changes.
We have a 10ms timer to allow limited batching, however.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We use dev-disconnect to convince one node the other has disconnected
(but not vice versa), to get deterministic behaviour. We do this with
one HTLC outstanding, to test the HTLC timeout path.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Once we see an on-chain tx, we ignore the state machine and handle it
as per the onchain.md draft. This specifies a *resolution* for each
output, and we're done when they're irrevocable.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Occasionally make seems to rebuild this in parallel, so make sure to
never have a partially complete one.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I had already disabled it, and this clears the decks for Segregated Witness
which gives us everything we want.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This uses libsodium (we could use openssl, but the required primitives
are only in 1.1.0 which is still in alpha).
It doesn't handle reconnections yet, either.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're going to change the protocol to send multiple changes at once, so
disable this compilation and testing for now.
We can revisit it afterwards once the protocol is stable again.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to change the protocol again, and I don't want to do the
grunt work to update these. They were useful for pre-build protocol
testing, though.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We abort when this happens, but still worth testing.
This involves a refactor so we can allocate watches off a specific context,
for easy freeing when they're no longer wanted.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We could probably just use Openssl's RAND_pseudo_bytes, but this gives
a nice interface to get a number between 1 and N.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This enforces the include order implemented by the previous patch, and
also make sure we haven't missed any headers.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A halfway house between the horror of dynamic generation and the fail of
making everything depend on every header.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I haven't implemented getting transactions from alphad (it needs the values
for the inputs, too), so switch to bitcoind.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I got confused navigating these, especially since Alpha and Bitcoin
have diverged (BIP68 was proposed after Elements Alpha).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The latest version of the BIP doesn't use inversion, but does use
bitshifts.
It also uncovered a bug in the test scripts: the block timestamps
creep forward when we generate large numbers of blocks (UpdateTime
insists it be > GetMedianTimePast() so it's valid). We need to take
this into account when waiting for the median to move (reduced it from
60 to 30 seconds, since that adds about 14 seconds).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Most complex change was gather_updates(), which handles all the "what
is the current state of the channel" logic for our dumb test utils.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Alpha has segregated witnesses, so txid doesn't include the inputs.
That means we can create the first commit transaction before we sign
the anchor tx.
Bitcoin will need to do this differently: presumbly via new sipops
(a OP_CHECKSIG2VERIFY I imagine).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise valgrind tells you when you test a hash; you want to
know if you hash uninitialized memory long before that.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I had each side using the other side's hash secret. That's a very
dumb idea, since it means you can steal from a unilateral close!
A's secret applies to A's commit transaction: it needs the
secret and B's final signature to steal funds, and that should
never happen (since A doesn't have the B's final signature, and
once A has given B the secret, they never broadcast the commit tx).
This makes the update a 4 step dance, since you need the new
revocation hash to make the other side's TX to sign.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
