Commit Graph

11849 Commits

Author SHA1 Message Date
Rusty Russell
66d8ce7c8c pytest: clarify test_onchain_multihtlc_our_unilateral / test_onchain_multihtlc_their_unilateral
Use the names, not calculations on an array.  It's simply clearer,
especially when debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-04 11:16:01 -05:00
Christian Decker
2632cddfe4 pay: Fix a memory leak when retrying getroute 2022-08-04 11:14:45 -05:00
Rusty Russell
645b1b505b lightningd: fix funding_locked in channel_opened notification.
Previously, "funding_locked" was always "true"!

(It's actually been wrong since its introduction in v0.7.3!)

Changelog-Fixed: Plugins: `channel_opened` notification `funding_locked` field is now accurate: was always `true`.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-04 13:08:25 +09:30
Seth For Privacy
498f9b75f2 Correct basics of backup plugin usage 2022-08-03 21:22:41 -05:00
Rusty Russell
4ffae2a8c6 CHANGELOG.md: note the Great Msat Migration!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-04 11:47:51 +09:30
Rusty Russell
cb496acb49 CHANGELOG.md: neaten entries.
1. Unbalanced backticks and missing space in bookkeeper line.
2. Rephrase emergency backup to not overpromise.
3. "logging" and "lightnignd" are not correct title, use "Config".
4. JSON-RPC not JSONRPC in one place.
5. Don't include empty EXPERIMENTAL section (I checked, it really was empty!)

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-04 11:47:51 +09:30
Rusty Russell
aea2ec5c0d CHANGELOG.md: add in CHANGELOG from 0.11.2 branch.
And manually remove those which were claimed for 0.12.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-04 11:47:51 +09:30
niftynei
071b1bc4f1 pyln: update versions to v0.12.0 2022-08-02 10:00:26 -07:00
niftynei
31ba3007bd changelog: v0.12.0rc1 release notes
Changelog for v0.12.0rc1

Changelog-None
2022-08-02 10:00:26 -07:00
niftynei
75ccce7808 devtools: if there's a message in the API call, print and exit
If you've got bad credentials, you'll get an error message.
2022-08-02 10:00:26 -07:00
niftynei
dfa325dc4d bkpr: make unit tests not fail if !HAVE_SQLITE3
We rely on sqlite3 being present to run unit tests for some bookkeeper
tests; here we effectively disable these tests if not available

Fixes report in #4928
Reported-By: @whitslack
2022-08-02 11:52:57 +09:30
Rusty Russell
2971b2af79 bkpr: insert obscure 60s pop references.
The initial snapshots on an already-running lightningd are expected to
be unbalanced, but this shouldn't cause users to long for the green,
green grass of home.

This controls the Art of Noise.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-08-01 09:01:08 -07:00
niftynei
3445882ee4 bkpr: use long-uint not size_t for time_t
Reported-By: @endothermicdev
2022-07-31 21:53:05 +09:30
niftynei
4e503f7d0a bkpr/listpeeers: add lease_fees back to funds; separate out in listpeers
First, how we record "our_funds" and then apply pushes vs lease_fees
(for liquidity ad buys/sales) was exactly opposite.

For pushes we were reporting the total funded into the channel, with the
push representing how much we'd later moved to the peer.

For lease_fees we were rerporting the total in the channel, with the
push representing how much was already moved to the peer.

We fix this (from a view perspective) by re-adding lease fees to what's
reported in the channel funding totals. Since this is now new behavior
(for leased channel values), we added new fields so we can take the old
field names thru a deprecation cycle.

We also make it possible to differentiate btw a push and a lease_fee
(before they were all the same), by adding to new fields to `listpeers`:
`fee_paid_msat` and `fee_rcvd_msat`.

This allows us to avoid math in the bookkeeper, instead we just pick
the numbers out directly and record them.

Fixes #5472

Changelog-Added: JSON-RPC: `listpeers` now has a few new fields for `funding` (`remote_funds_msat`, `local_funds_msat`, `fee_paid_msat`, `fee_rcvd_msat`).
Changelog-Deprecated: JSON-RPC: `listpeers`.`funded` fields `local_msat` and `remote_msat` are now deprecated.
2022-07-31 21:53:05 +09:30
niftynei
a675f4c24e balance_snapshot: emit balances for channels that are awaiting_lockin
9cad7d6a6a changed the behavior of `channel_active`, which slightly
broke how our balance snapshots work (we need info about channels that
aren't on-chain yet).

This patches adds AWAITING_UNILATERAL back in.
2022-07-31 21:53:05 +09:30
William Casarin
10d66c25c4 commando-rune: show warning when creating runes with no restrictions
Changelog-Added: Show warning when creating runes with no restrictions
Signed-off-by: William Casarin <jb55@jb55.com>
2022-07-29 08:46:14 -07:00
CC
78804d9ea8 fix doc: deschashonly 2022-07-29 16:33:06 +09:30
Rusty Russell
8c9fa457ba pytest: fix flake in test_gossip_timestamp_filter
```
        # 0x0100 = channel_announcement
        # 0x0102 = channel_update
        # (Node announcement may have any timestamp)
        types = Counter([m[0:4] for m in msgs])
        assert types['0100'] == 1
>       assert types['0102'] == 2
E       assert 1 == 2

tests/test_gossip.py:324: AssertionError
```

Examining the logs shows that we ask l4 for timestamps
"first_timestamp=1658892115 timestamp_range=13", and the timestamp on
the missing update is exactly `1658892128`.

So round the end time up by 1 for filtering.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 15:08:44 +09:30
Rusty Russell
22ff007d64 connectd: control connect backoff from lightningd.
We used to tell connectd to remember our connect delay, and hand it
back (increased if necessary).

Instead, simply record when we last tried to connect.  If it was less
than 10 minutes ago, double delay (up to 5 minutes max), otherwise
reset delay to 1 second.

This covers all scenarios: whether we reconnect then immediately
disconnect, or never successfully connect, it doesn't matter.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Fixes: #5453
2022-07-28 15:08:44 +09:30
Rusty Russell
9cad7d6a6a lightningd: don't consider AWAITING_UNILATERAL to be "active".
It's not active: we don't want to connect.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 15:08:44 +09:30
Rusty Russell
a259698906 pytest: test that we don't try to reconnect in AWAITING_UNILATERAL.
We would just send an error, and it's annoying.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 15:08:44 +09:30
niftynei
6e9af1ef3e bkpr: cleanup csv_safe_str 2022-07-28 12:08:18 +09:30
niftynei
c83c1521ce memleak: throw away things when we're done with them
memleak was complaining about dangling refs; they were all allocated off
of `cmd` but not technically unreachabe from any in-context memory.

Instead we move things over to tmpctx to clean up (in the paths where
we're going to move the objects out of reach via intermediate RPC call)

ALSO: cleans up unused param in `lookup_invoice_desc`

lightningd-1 2022-07-23T19:17:11.192Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x1511f68
lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper:   label=plugins/bkpr/recorder.c:1048:struct account
lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper:   backtrace:
lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/recorder.c:1048 (stmt2account)
lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/recorder.c:1109 (find_account)
lightningd-1 2022-07-23T19:17:11.196Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1606 (parse_and_log_channel_move)
lightningd-1 2022-07-23T19:17:11.197Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1713 (json_coin_moved)
lightningd-1 2022-07-23T19:17:11.198Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1421 (ld_command_handle)
lightningd-1 2022-07-23T19:17:11.198Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1491 (ld_read_json_one)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1511 (ld_read_json)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:59 (next_plan)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:407 (do_plan)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:417 (io_ready)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/poll.c:453 (io_loop)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1708 (plugin_main)
lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1812 (main)
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper:     ../csu/libc-start.c:308 (__libc_start_main)
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper:   parents:
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1378:struct command
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x15305b8
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper:   label=plugins/bkpr/bookkeeper.c:1643:enum mvt_tag[]
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper:   backtrace:
lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     ccan/ccan/tal/tal.c:471 (tal_alloc_arr_)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1643 (parse_tags)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1686 (json_coin_moved)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1421 (ld_command_handle)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1491 (ld_read_json_one)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1511 (ld_read_json)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:59 (next_plan)
lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:407 (do_plan)
lightningd-1 2022-07-23T19:17:11.202Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:417 (io_ready)
lightningd-1 2022-07-23T19:17:11.202Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/poll.c:453 (io_loop)
lightningd-1 2022-07-23T19:17:11.202Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1708 (plugin_main)
lightningd-1 2022-07-23T19:17:11.203Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1812 (main)
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:     ../csu/libc-start.c:308 (__libc_start_main)
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:   parents:
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1378:struct command
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x1508da8
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:   label=plugins/bkpr/bookkeeper.c:1568:struct channel_event
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:   backtrace:
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:     ccan/ccan/tal/tal.c:442 (tal_alloc_)
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1568 (parse_and_log_channel_move)
lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1713 (json_coin_moved)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1421 (ld_command_handle)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1491 (ld_read_json_one)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1511 (ld_read_json)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:59 (next_plan)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:407 (do_plan)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/io.c:417 (io_ready)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     ccan/ccan/io/poll.c:453 (io_loop)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1708 (plugin_main)
lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper:     plugins/bkpr/bookkeeper.c:1812 (main)
lightningd-1 2022-07-23T19:17:11.206Z **BROKEN** plugin-bookkeeper:     ../csu/libc-start.c:308 (__libc_start_main)
lightningd-1 2022-07-23T19:17:11.206Z **BROKEN** plugin-bookkeeper:   parents:
lightningd-1 2022-07-23T19:17:11.206Z **BROKEN** plugin-bookkeeper:     plugins/libplugin.c:1378:struct command
2022-07-28 12:08:18 +09:30
niftynei
2f72bbbbc5 nit: send_outreq returns &pending, no need to call sep command 2022-07-28 12:08:18 +09:30
niftynei
e048292fdf bkpr-zeroconf: Zeroconfs will emit 'channel_proposed' event
Keep the accounts as an 'append only' log, instead we move the marker
for the 'channel_open' forward when a 'channel_open' comes out.

We also neatly hide the 'channel_proposed' events in 'inspect' if
there's a 'channel_open' for that same event.

If you call inspect before the 'channel_open' is confirmed, you'll see
the tag as 'channel_proposed', afterwards it shows up as
'channel_open'. However the event log rolls forward -- listaccountevents
will show the correct history of the proposal then open confirming (plus
any routing that happened before the channel confirmed).
2022-07-28 12:08:18 +09:30
niftynei
30aa1d79fb bkpr: for zerconfs, we still wanna know you're opening a channel
We need a record of the channel account before you start sending
payments through it. Normally we don't start allowing payments to be
sent until after the channel has locked in but zeroconf does away with
this assumption.

Instead we push out a "channel_proposed" event, which should only show
up for zeroconfs.
2022-07-28 12:08:18 +09:30
Rusty Russell
1b5dc4409a doc: remove two more generated manpages.
Somehow these two were missed in 79e09b92ef.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 12:08:18 +09:30
Rusty Russell
6a22411f7e doc: note that bookkeeper-dir and bookkeeper-db are in bookkeeper plugin.
We do this for bcli options, too.  Good to note if people disable/replace the
plugin.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 12:08:18 +09:30
Rusty Russell
305a238810 plugins/Makefile: put bitcoin/chainparams.o in PLUGIN_COMMON_OBJS since everyone needs it.
This was originally done as part of the bookkeeper introduction, but it deserves its
own patch (and that one didn't remove the bitcoin/chainparams.o from the individual
requirements lines).

Suggested-by: @niftynei
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
2022-07-28 12:08:18 +09:30
niftynei
71c03bc082 bkpr: Add an option to set the database to something else (postgres)
`lightningd` has an option --wallet that lets you supply a database dsn
string to connect to a sqlite3/postgres database that's hosted/stored
elsewhere.

This adds the `--bookkeeper-db` option which does the same, except for
the bookkeeping data for a node!

Note that the default is to go in the `lightning-dir` in a database
called `accounts.sqlite3`
2022-07-28 12:08:18 +09:30
niftynei
3c79a456c0 test-db-provider: if postgres in tests, startup a bookkeeper db
FXIME: Has a edge case where if you disable the bookkeeper, it'll
blowup because you've got an option that isn't present anywhere...
2022-07-28 12:08:18 +09:30
niftynei
e5d3ce3b1f bkpr incomestmt: properly escape things for the CSVs
First off, when we pull data out of JSON, unescape it so we don't end up
with extraneous escapes in our bookkeeping data. I promise, it's worth
it.

Then, when we print descriptions out to the csvs, we gotta wrap
everything in quotes... but also we have to change all the double-quotes
to singles so that adding the quotes doesn't do anything untoward.

We also just pass it thru json_escape to get rid of linebreaks etc.

Note that in the tests we do a byte comparison instead of converting the
CSV dumps to strings because python will escape the strings on
conversion...
2022-07-28 12:08:18 +09:30
niftynei
5146baa00b bkpr csvs: koinly + cointracker only accept fees on the same line
So we print out invoice fees on the same line for those CSVs! This means
we have to do a little bit of gymnastics (but not too bad):
	- we save the fee amount onto the income event now so we can use
it later
	- we ignore every "invoice_fee" event for the koinly/cointracker

Note that since we're not skipping income events in the loops we also
move the newline character to the start of every `_entry` function so
skipped records dont incur empth lines.

Changelog-Added: bkpr: print out invoice fees on the same line for `koinly` and `cointracker` csv types
2022-07-28 12:08:18 +09:30
niftynei
352b419755 bkpr: save invoice description data to the database and display it
It'll be really nice to be able to read description data about an
invoice, if we've got it!
2022-07-28 12:08:18 +09:30
niftynei
5c45939acf test-utils: add the bolt11 invoice
nice to have for making tests that assert on description/bolt data
2022-07-28 12:08:18 +09:30
niftynei
ab94c557c7 bkpr: add test for bookkeeper being added after channel has closed
We rescan and pick up the channel's close tx, but can't go back and
get the open info, since we've already deleted the channel from the
database.
2022-07-28 12:08:18 +09:30
niftynei
2d22bab879 bkpr: fetch originating account, if exists, and make sure is populated
It's possible we'll get an "external" event for an account/channel and
then try to do close checks on it and it'll bomb out because we didn't
go look up the originating account to make sure that that had open info

Now, we make sure and do that when we hear about an originating account
2022-07-28 12:08:18 +09:30
niftynei
55e15c5f10 bkpr: correctly pass in command for jsonrpc_request_start
We weren't passing the command, which meant that it wasn't getting
populated correctly.

We do that now, it fixes some crashes
2022-07-28 12:08:18 +09:30
niftynei
67ce0ee3b2 bkpr: prevent crash when updating same account all at once
If two events for the same (unlogged) account come in and get run
through the "lookup peer" code, we should anticipate that.

We do two things here:
	- one, if it's a duplicate "event" to create the channel open
	  we check for that and just exit early
	- two, we were using a copy of the account that was
	  fetched/pulled from before the RPC ran, so instead we
	  just re-pull the most up to date account info for the
	  close checks.

This fixes the crash I was getting in re-running these things
2022-07-28 12:08:18 +09:30
niftynei
38eb95ffee bkpr: more logging 2022-07-28 12:08:18 +09:30
niftynei
9346158290 bkpr: new method, "is_external_account"
Utility method to figure out if an account is "external"
2022-07-28 12:08:18 +09:30
niftynei
97204d6e0a bkpr: duplicate the name, dont steal it 2022-07-28 12:08:18 +09:30
niftynei
0617690981 coin_mvt/bkpr: add "stealable" tag to stealable outputs
If we expect further events for an onchain output (because we can steal
it away from the 'external'/rightful owner), we mark them.

This prevents us from marking a channel as 'onchain-resolved' before
all events that we're interested in have actually hit the chain.

Case that this matters:
Peer publishes a (cheating) unilateral close and a timeout htlc (which
we can steal).
We then steal the timeout htlc.

W/o the stealable flag, we'd have marked the channel as resolved when
the peer published the timeout htlc, which is incorrect as we're still
waiting for the resolution of that timeout htlc (b/c we *can* steal it).
2022-07-28 12:08:18 +09:30
niftynei
c1cef773ca bkpr: make sure there's always at least on difference in blockheights
We can't divide by zero, so make sure it's always 1 (with a small loss
of precision for other cases, ce la vie)
2022-07-28 12:08:18 +09:30
niftynei
0c2b43b6b2 bkpr: add more data to listaccountevents printout 2022-07-28 12:08:18 +09:30
niftynei
d72033882f bkpr: check for channel resolution for any "originated" event
We were failing to mark channels as resolved b/c we weren't using later
events to external (but originated from this account) events as signals
to run the channel resolution check.

This fixes that, and adds a test.
2022-07-28 12:08:18 +09:30
niftynei
cf8b30d2e8 bkpr: print more info in bkpr-listbalances 2022-07-28 12:08:18 +09:30
niftynei
3dcfd2d0e4 bkpr: account name is required for bkpr-inspect 2022-07-28 12:08:18 +09:30
niftynei
563910e667 bkpr: add docs, change names to 'bkpr-*'
Adds schema definitions and manpages for bkpr- commands; also renames
the commands to all start with 'bkpr-', so they're easier to identify/
make runes about.
2022-07-28 12:08:18 +09:30
niftynei
6dfba2468a json-schema: allow 'required' to not be present in if switches 2022-07-28 12:08:18 +09:30