Rather than reaching into data structures, let them register their own
callbacks. This avoids us having to expose "memleak_remove_xxx"
functions, and call them manually.
Under the hood, this is done by having a specially-named tal child of
the thing we want to assist, containing the callback.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We now have a much stronger consistency check from the combination of
transaction wrapping, tal memory leak detection. Tramsaction wrapping ensures
that each statement is executed before the transaction is committed. The
commit is also driven by the `io_loop`, which means that it is no longer
possible for us to have statements outside of transactions and transactions
are guaranteed to commit at the round's end.
By adding the tal-awareness we can also get a much better indication as to
whether we have un-freed statements flying around, which we can test at the
end of the round as well.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We will soon generalize the DB, so directly reaching into the `struct db`
instance to talk to the sqlite3 connection is bad anyway. This increases
flexibility and allows us to tailor the actual implementation to the
underlying DB.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This removes the WIRE_FINAL_EXPIRY_TOO_SOON which leaked too much info,
and adds the blockheight to WIRE_INCORRECT_OR_UNKNOWN_PAYMENT_DETAILS.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently end up sleeping for 1 second for channeld and gossipd:
better to use a normal blocking waitpid and an alarm to wake us in
case they don't exit.
This speeds up `lightning-cli stop` on my machine from 2.008s to 0.008s:
a 286 times speedup!
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
During sync it is highly likely that we can coalesce multiple calls and share
results among them. We also report back failures for non-existing blocks early
on, so we don't run into issues with blocks that our bitcoind doesn't have
yet.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was caused by us not checking against the max_blockheight, but rather the
min_blockheight which can be negative with a newly created node. This is still
safe since we check for duplicates anyway in `wallet_filteredblock_add`.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is probably worth preventing.
1. Our depth estimate would be inaccurate possibly leading to us
timing out too early.
2. If we're not up-to-date our onchain funds are unknown.
3. We wouldn't be able to send or receive HTLCs until we're synced anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We want to still allow incoming connections, and reestablishment of
channels, but if one tries to give us an HTLC, stall until we're
synced.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we don't know block height, we shouldn't be sending HTLCs. This
stops us forwarding HTLCs as well as new payments.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I suspect multiple plugins trying to connect at the same
time are overrunning the 1-deep listen queue:
From man listen(2):
The backlog argument defines the maximum length to which the queue of
pending connections for sockfd may grow. If a connection request ar‐
rives when the queue is full, the client may receive an error with an
indication of ECONNREFUSED
Fixes: #2922
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`close` takes two optional arguments: `force` and `timeout`.
`timeout` doesn't timeout the close (there's no way to do that), just
the JSON call. `force` (default `false`) if set, means we unilaterally
close at the timeout, instead of just failing.
Timing out JSON calls is generally deprecated: that's the job of the
client. And the semantics of this are confusing, even to me! A
better API is a timeout which, if non-zero, is the time at which we
give up and unilaterally close.
The transition code is awkward, but we'll manage for the three
releases until we can remove it.
The new defaults are to unilaterally close after 48 hours.
Fixes: #2791
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we were to just insert filtered blocks in the range that we will scan later
we'd be hitting the uniqueness constraints later.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Instead of allowing all calls to `getfilteredblock` to be scheduled on the
`bitcoind` queue right away we instead add them in a separate queue, and
process a single call at a time. This limits the concurrency and avoids
thrashing `bitcoind`. At the same time we dispatch incoming results back to
all calls that were queued for that particular blockheight, reducing the
overall number of calls and an increase in overall speed.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We will be calling the callback out of order once we fan out the results of a
single lookip to multiple calls, so being sure that everything is allocated
ahead of time is necessary.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Since we now check all P2WSH outputs in a block, this is getting quite a
common occurence, so logging just produces lots of noise.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This will eventually replace the multi-step `getblockhash` + `getblock` +
`gettxout` mechanism, and return entire filtered blocks which can be added to
the DB, and represent the full set of P2WSH UTXOs.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This was causing `--help` to fail if we already had a `lightningd` running
with the same `--lightning-dir`.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
1. Now checking the pid file really does precede touching the db and
starting plugins, which is far safer.
2. Crashlog is now activated just after daemon parent release, and just
before the main loop, which means no "crash" on startup if we call fatal().
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Dumb programs which have a --daemon option call fork() early. This is
terrible UX since startup errors get lost: the program exits with
"success" immediately then you discover via the logs that it didn't
start at all.
However, forking late introduced a heap of problems with changing
pids. Instead, fork early but keep stderr and the parent around: if
we fail early on, the parent fails with us. We release our parent
with an explicit action just before the main loop.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we are walking the entire allocation tree anyway, and access the tal
metadata anyway, we can just as well also track the size of the memory
allocations to simplify debugging of memory use.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Otherwise it creates the lightning-dir. This can't be helped for --help
(at least, if plugins are present), but --version simply prints and exits.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Note that we move adding the plugin to the plugins list to the end, otherwise
the hook from logging can examine the (uninitialized) plugin.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is easy since we did the option parsing cleanup, but it has the
effect that plugins are launched from the lightning-dir. Now
we have dynamic plugins, this means startup and post-startup plugins
experience the same environment.
This is absolutely a desirable thing: they can just drop files in
their cwd rather than having to move (including, I might note, core
files!).
We also highlight the change in various places (and a drive-up update
of PLUGINS.md which says you have to use --plugin).
The next patch adds a backwards compatibility wedge for old users of
relative plugin paths.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We currently send channel_announcement as soon as we and our
peer agree it's 6 blocks deep. In theory, our other peers might
not have seen that block yet though, so delay a little.
This is mitigated by two factors:
1. lnd will stash any "not ready yet" channel_announcements anyway.
2. c-lightning doesn't enforce the 6 depth minimum at all.
We should not rely on other nodes' generosity or laxity, however!
Next release, we can start enforcing the depth limit, and maybe stashing
ones which don't quite make it (or simply enforce depth 5, not 6).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In file included from wallet/test/run-wallet.c:15:0:
./lightningd/peer_htlcs.c: In function ‘htlcs_reconnect’:
./lightningd/peer_htlcs.c:2060:15: error: ‘failcode’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
} else if (failcode) {
^~~~~~~~
./lightningd/peer_htlcs.c:2056:19: error: ‘failcode’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
failcode != 0
~~~~~~~~~^~~~
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
`forward_event`
A notification for topic `forward_event` is sent every time the status
of a forward payment is set. The json format is same as the API
`listforwards`.
```json
{
"forward_event": {
"payment_hash": "f5a6a059a25d1e329d9b094aeeec8c2191ca037d3f5b0662e21ae850debe8ea2",
"in_channel": "103x2x1",
"out_channel": "103x1x1",
"in_msatoshi": 100001001,
"in_msat": "100001001msat",
"out_msatoshi": 100000000,
"out_msat": "100000000msat",
"fee": 1001,
"fee_msat": "1001msat",
"status": "settled",
"received_time": 1560696342.368,
"resolved_time": 1560696342.556
}
}
```
or
```json
{
"forward_event": {
"payment_hash": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
"in_channel": "103x2x1",
"out_channel": "110x1x0",
"in_msatoshi": 100001001,
"in_msat": "100001001msat",
"out_msatoshi": 100000000,
"out_msat": "100000000msat",
"fee": 1001,
"fee_msat": "1001msat",
"status": "local_failed",
"failcode": 16392,
"failreason": "WIRE_PERMANENT_CHANNEL_FAILURE",
"received_time": 1560696343.052
}
}
```
- The status includes `offered`, `settled`, `failed` and `local_failed`,
and they are all string type in json.
- When the forward payment is valid for us, we'll set `offered`
and send the forward payment to next hop to resolve;
- When the payment forwarded by us gets paid eventually, the forward
payment will change the status from `offered` to `settled`;
- If payment fails locally(like failing to resolve locally) or the
corresponding htlc with next hop fails(like htlc timeout), we will
set the status as `local_failed`. `local_failed` may be set before
setting `offered` or after setting `offered`. In fact, from the
time we receive the htlc of the previous hop, all we can know the
cause of the failure is treated as `local_failed`. `local_failed`
only occuors locally or happens in the htlc between us and next hop;
- If `local_failed` is set before `offered`, this
means we just received htlc from the previous hop and haven't
generate htlc for next hop. In this case, the json of `forward_event`
sets the fields of `out_msatoshi`, `out_msat`,`fee` and `out_channel`
as 0;
- Note: In fact, for this case we may be not sure if this incoming
htlc represents a pay to us or a payment we need to forward.
We just simply treat all incoming failed to resolve as
`local_failed`.
- Only in `local_failed` case, json includes `failcode` and
`failreason` fields;
- `failed` means the payment forwarded by us fails in the
latter hops, and the failure isn't related to us, so we aren't
accessed to the fail reason. `failed` must be set after
`offered`.
- `failed` case doesn't include `failcode` and `failreason`
fields;
- `received_time` means when we received the htlc of this payment from
the previous peer. It will be contained into all status case;
- `resolved_time` means when the htlc of this payment between us and the
next peer was resolved. The resolved result may success or fail, so
only `settled` and `failed` case contain `resolved_time`;
- The `failcode` and `failreason` are defined in [BOLT 4][bolt4-failure-codes].
Warp this process as a new function: 'void json_format_forwarding_object()'. This function will be used in 'forward_event' next, and can ensure the consistent json object structure for forward_payment between 'listforwards' API and 'forward_event' notification.
The reason lnd was sending sync error was that we were taking more than
30 seconds to send the channel_reestablish after connect. That's
understandable on my test node under valgrind, but shouldn't happen normally.
However, it seems it has at least once,
(see https://github.com/ElementsProject/lightning/issues/2847)
: space out startup so it's less likely to happen.
Suggested-by: @cfromknecht
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is the other origin, besides `bitcoin_tx`, where we create `bitcoin_tx`
instances, so add the context as soon as possible. Sadly I can't weave the
chainparams into the deserialization code since that'd need to change all the
generated wire code as well.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The way we build transactions, serialize them, and compute fees depends on the
chain we are working on, so let's add some context to the transactions.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This is just taking the existing serialization code and repackaging it in a
more useful form.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This adds a new pair of files : lightningd/plugin_control, along with a new RPC
command : 'plugin'. This command can be used to manage plugins without restarting lightningd:
lightning-cli plugin start helloworld.py
lightning-cli plugin stop helloworld.py
This adds 'plugin_unregister_hook' and 'plugin_unregister_hook_all'
functions to unregister a given hook a plugin registered, or all hooks a
plugin registered for. Since hooks can only be registered once, it's
useful in the case a new plugin is added which would be prefered for
hook registration over an already loaded plugin.
This adds a 'configured' boolean member to the plugin struct so that we can add plugins to ld->plugins' list and differenciate fresh plugins.
This also adds 'plugins_start' so that new plugins can be started without calling 'plugins_init' and running an io loop
It assumes the head of the array is the object/array we want to remove from,
but that's not true if we're trying to remove from a sub-object.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No code changes, just move.
Put all the dev options into the one function, and register (and
comment on) the early args first.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed that --network=regtest didn't override 'network=bitcoin' in
the config file.
Normally we parse the config file first, then the commandline (so the cmdline
wins). But for early options, we do cmdline first so we can find the config
file. That was fine when the only early option was the location of the
config file, but now it includes plugins and the network setting.
So do a boutique cmdline parse *just* to find the config file, then parse
the config file early options, then the cmdline early options.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
