Commit Graph

6382 Commits

Author SHA1 Message Date
Christian Decker
e65e4fcf4b wallet: Call db_stmt_free from the db_stmt destructor automatically
This is much more in line with the rest of our memory management.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
093f5bfbec db: Switch to new DB asbtraction for DB migrations
These do not require the ability to iterate over the result, hence they can be
migrated already.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
50a1af8908 db: Implement the sqlite3 driver
This is the DB-specific counterpart to the previous commit.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
38601afd17 db: Implement skaffolding for the dispatch of DB-specific functions
These functions implement the lookup of the query, and the dispatch to the
DB-specific functions that do the actual heavy lifting.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
70e8da4fbd wallet: Add read-only flag to extracted queries
This gets rid of the two parallel execution paths of read-only and write
queries, by explicitly stating with each query whether it is a read-only
query, we only need to remember the ones marked as write queries.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
455aa48da0 wallet: Move the struct db definition to db_common.h
All drivers will have to reach into it, so put it in a place that is reachable
from the drivers, along with all other definitions.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
e04772ec9c wallet: Move the db_fatal definition so we can use it in drivers
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
acedcc593f wallet: Look up the db_config for the given driver in db_open
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
e4ab98459c wallet: Add tooling to extract SQL queries and generate driver info
This is the counterpart of the annotations we did in the last few commits. It
extracts queries, passes them through a driver-specific query rewriter and
dumps them into a driver-specific query-list, along with some metadata to
facilitate processing later on. The generated query list is then registered as
a `db_config` and will be loaded by the driver upon instantiation.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
803007ecdf db: Make the db struct private and provide accessors instead
We will soon generalize the DB, so directly reaching into the `struct db`
instance to talk to the sqlite3 connection is bad anyway. This increases
flexibility and allows us to tailor the actual implementation to the
underlying DB.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
66a47d2761 wallet: Annotate migrations using the SQL macro
Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
5270c149e3 wallet: Add macro to annotate SQL statements
These two simple macros have a twofold use:

 1) They serve as annotations for the query extraction tool to find them when
 extracting queries from the C source code.
 2) They replace the actual queries with names that can be used to lookup the
 queries in a table again, once they have been rewritten into the target SQL dialect.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
57f91246cf wallet: Change db_select_prepare to check for select instead
`db_select_prepare` was prepending the "SELECT" part in an attempt to limit
its use to read-only statements. This is leads to the queries in the code not
actually being well-formed, which we'll need in a later commit, and was also
resulting in extra allocations. This switches the behavior to just enforce a
"SELECT" prefix being present which allows us to have well-formed queries in
the code again and avoids the extra allocation.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
f08fd9eaab wallet: Remove printf-like db_select variant
We need to have full DB queries that can be extracted at compile time later in
order to be able to rewrite them in other SQL dialects. In addition we had a
bit of unnecessary code-duplication in db_select and db_select_prepare. Now
the former uses the latter internally.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
Christian Decker
b856bb2f54 wallet: Expand select query field lists from macros
These will interfere with our query extraction process later on, and they were
really separating definition from use anyway, so let's expand these field lists.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-05 23:41:05 +00:00
trueptolemy
b5600b5403 CHANGELOG: And the entry of the change of txprepare 2019-09-05 16:05:36 -05:00
trueptolemy
d813dff3b4 doc: Update the document of txprepare 2019-09-05 16:05:36 -05:00
trueptolemy
74055eada5 pytest: Test txprepare with multiple outputs 2019-09-05 16:05:36 -05:00
trueptolemy
42d5d753db cleanup: Remove unused structure struct withdrawal 2019-09-05 16:05:36 -05:00
trueptolemy
cdcafdaf74 API: txprepare now support mutiple outputs 2019-09-05 16:05:36 -05:00
trueptolemy
0ab3817715 common: Fix a memleak in withdraw_tx 2019-09-05 16:05:36 -05:00
trueptolemy
b660531216 common: withdraw_tx() now use the array of struct bitcoin_tx_output as parameter 2019-09-05 16:05:36 -05:00
trueptolemy
6570c743c4 bitcoin: Add the interface to add multiple outputs to bitcoin transaction 2019-09-05 16:05:36 -05:00
darosior
4672795e1a lightningd: don't join the config_dir to the pid_file path 2019-09-05 19:10:33 +02:00
lisa neigut
10ed2ebdb4 wallet: track our change outputs
Add change scripts to our txfilter so that we mark them as confirmed
(and can subsequently spend them)
2019-09-05 15:48:49 +02:00
lisa neigut
01e0ba8232 test: add check that withdraw's change is confirmed
We're not tracking change outputs for withdraws, so they're not
confirming. This test exposes the problem.
2019-09-05 15:48:49 +02:00
lisa neigut
4030ccf94c fix memleak: remove duplicated new_channel creation
We call this same exact code inside `funder_finalize_channel_setup`;
which results in a memleak since we don't free the previous before
clobbering it.
2019-09-05 01:32:48 +00:00
lisa neigut
4d2f91f7ae test-fixtures: use helper for checking errors
log files were being deleted on memleak errors, since
we weren't marking the node has having an error.

this helper function is designed to exactly handle this, so
we use the helper function and modify it to print any additional
error messages that are handed back from killall.
2019-09-05 01:32:48 +00:00
lisa neigut
df1d92a7a2 test-fixtures: return error set, don't throw exception
Throwing an exception while killing all nodes meant that
we aren't cleaning up all the nodes properly. Instead,
collect the errors, and return them back to the upper level,
where we report them and terminate as expected.
2019-09-05 01:32:48 +00:00
lisa neigut
63c80e7aa9 test-fixtures: move 'broken' check to after memleak
Memleaks appear in the logs as 'broken', so the broken log
check captures them as well. This moves broken to after memleak
so we get more informative error messages.
2019-09-05 01:32:48 +00:00
lisa neigut
5663ecc599 listfunds: add 'blockheight' for confirmed transactions
Needed to calculate the value of all inputs for 'all' in
externalized fundchannel
2019-09-04 14:08:08 +02:00
Michael Schmoock
76f27f47dc doc: HACKING - point out how to run the pylightning tests 2019-09-03 19:02:04 +02:00
Michael Schmoock
bc5add399b pylightning: adds a millisatoshi test class 2019-09-03 19:02:04 +02:00
Michael Schmoock
c0069d8944 pylightning: adds Millisatoshi __radd__ method for sum() 2019-09-03 19:02:04 +02:00
darosior
0e2ec27576 doc/requirements.txt: update 'mrkd' dependency 2019-09-02 16:31:36 +02:00
Christian Decker
7fafd934cf pytest: Remove --duration argument
We're not going to optimize travis any longer so let's no fill logs with
useless measurements.
2019-09-02 16:31:36 +02:00
Christian Decker
fd1e8a5999 pytest: Fix directory deletion on passed tests
We were checking the test request against the searched for string. This fixes
it by actually looking at the outcome instead and should clean up correctly
if tests do not fail.

Signed-off-by: Christian Decker <decker.christian@gmail.com>
2019-09-02 16:31:36 +02:00
Christian Decker
245dff26e0 pytest: Updated requirements.txt to latest versions
This is an issue that was raised in #2665: some of the dependencies where
causing warnings to be added to the logs about deprecated dependencies. Since
I did not get these warnings I just blanket updated all the dependencies in
the hopes of getting the warnings to resolve.

Signed-off-by: Christian Decker <@cdecker>
2019-09-02 16:31:36 +02:00
Christian Decker
c3254e6639 docs: Update manpages to be recognized correctly by mrkd
mrkd started enforcing the `name -- short description` style of top-level
headings somewhere, and was thus failing to build the man-pages. I swapped
the title and with the existing short description to make it work
again. `mrkd` will automatically infer the section from the filename so no
need to put it in the title as well.

In addition I removed the "last updated" lines at the bottom since they are
out of date at best, and misleading at the worst. If we want to keep them, I'd
suggest generating them from the commit that last touched them.
2019-09-02 16:31:36 +02:00
Rusty Russell
049529542a lightningd: delay reprocessing of incoming htlcs at startup until plugins ready.
Fixes: #2923
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-01 16:55:32 +02:00
Rusty Russell
f0ab4d72da pytest: add test that htlc_accepted hook only gets called after init.
We fail this.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-09-01 16:55:32 +02:00
nicolas.dorier
49496ab3ba [Docker] Use stretch-slim instead of Alpine 2019-08-30 16:17:51 +02:00
darosior
8f8e955421 Update installation instructions and Dockerfile
Get rid of the 'asciidoc' dependency, make developers also install 'doc/requirements.txt'
2019-08-30 00:34:11 +00:00
darosior
092f20aeeb README: Replace broken links to manpages 2019-08-30 00:34:11 +00:00
Rusty Russell
3ef5160fa7 travis: add --enable-experimental-features (build-only) test.
Also, add a 32-bit build-only test, and move all the build-only tests
to the top.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Rusty Russell
02b9b7f6e6 tests: update mocks for --enable-experimental-features builds.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 15:51:36 +02:00
Christian Decker
b77270fa46 docs: Add the title of the manpage to TOC and sort alphabetically
Fixes #2987
2019-08-29 14:56:24 +02:00
Rusty Russell
5e3e2f4e17 run-secret_eq_consttime: try harder.
This will never be reliable under high load, without making it unable
to detect real errors.  But the test is useful because if we don't
have this test we'll never notice if we break the const-timedness of
our implementation.

So, move the calloc out of the test loop (which seems to make it more
reliable), and then after we've run it, check the 1-minute load
average.  Too high, we don't complain about results.  It's not
perfect, but it's better.

Running 100 times (-O3) serially gave 100 successes with the following results:
	Constant: Within 5% 562-926(832.89+/-73)/1000 times
        Non-constant: More than 5% slower 860-990(956.35+/-26)/1000 times

More importantly, if we swap the const and non-const tests, we get
the expected 100 failures:
	Non-constant: Within 5% 14-79(41.17+/-14)/1000 times
	Constant: More than 5% slower 44-231(111.89+/-33)/1000 times

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
6901732ee0 lightningd: create --list-features-only which lists what features we support.
This allows the lightning-rfc protocol tests to automatically query what
features we support.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00
Rusty Russell
17edcfa24d features: don't assume we'll always only advertize optional features.
Generalize things a bit so OPTIONAL_FEATURE() and COMPULSORY_FEATURE()
work with either odd or even features, then explicitly use OPTIONAL_FEATURE
in our internal feature array.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2019-08-29 09:01:48 +02:00