This hook is called when the queue is empty; we should only send gossip
according to the gossip timer. We're currently dribbling it out after
every message, in violation of the spec.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we have them, let's use them. I missed one case deliberately, since
that causes merge conflicts when I replace it in a following patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I'm not completely conviced that we can't end up removing pending things,
so change asserts to simple returns.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we make destroy_node() remove itself from the map, then we simply
need to free it.
We can batch the frees (as we need) simply by reparenting all the pruned
nodes onto a single temporary parent, then freeing it, relying on tal's
internal datastructures.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
get_connection_by_scid() and update_to_pending() both do the same
lookup which we did in handle_channel_update().
Do the lookup once, and simplify the others.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We always hand in "NULL" (which means use tal_len on the msg), except
for two places which do that manually for no good reason.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We usually did this, but sometimes they were named after what they did,
rather than what they cleaned up.
There are still a few exceptions:
1. I didn't bother creating destroy_xxx wrappers for htable routines
which already existed.
2. Sometimes destructors really are used for side-effects (eg. to simply
mark that something was freed): these are clearer with boutique names.
3. Generally destructors are static, but they don't need to be: in some
cases we attach a destructor then remove it later, or only attach
to *some* cases. These are best with qualifiers in the destroy_<type>
name.
Suggested-by: @ZmnSCPxj
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
DEBUG:root:lightningd(16333): 2018-02-08T02:12:21.158Z lightningd(8262): lightning_openingd(0382ce59ebf18be7d84677c2e35f23294b9992ceca95491fcf8a56c6cb2d9de199): Failed hdr decrypt with rn=2
We only hand off the peer if we've not started writing, but that was
insufficient: we increment the sn twice on encrypting packet, so there's
a window before we've actually started writing where this is now
wrong.
The simplest fix is only to hand off from master when we've just written,
and have the read-packet path simply wake the write-packet path.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We get intermittant failure: WIRE_UNKNOWN_NEXT_PEER (First peer not ready)
because CHANNELD_NORMAL and actually telling gossipd that the channel
is available are distinct things: we need both.
(For test_closing_different_fees, we were testing CHANNELD_NORMAL on
the peer, not on l1, too).
But we may also directly send the announcement sigs if the height is
sufficient, so the simplest is to unify the messages.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now we have wirestring, this is much more natural. And with the
24M length limit, we needn't be so concerned about dumping 64k peer
messages in hex.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
These are now logically arrays of pointers. This is much more natural,
and gets rid of the horrible utxo array converters.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Commit a57a2dcb86 introduced a time_t
in routing.h. So also move the time.h include to the header. This
fixes the build on FreeBSD.
Signed-off-by: Wladimir J. van der Laan <laanwj@gmail.com>
We were dropping these on the floor while checking for txout. So now
we add a map that holds announcements while we are checking.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We are wasting way too much time looking for announcements and updates
in the broadcast. We can just hint where to find the message to be
evicted and safe the traversal.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Adding channels that we are currently verifying to the map, and
skipping if we already have a channel at that position.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We use this technique for the other tags, so use it here too.
This was drawn to my attention when I made more than 10 channels in a
block, and the string changed length:
Valgrind error file: valgrind-errors.31415
==31415== Conditional jump or move depends on uninitialised value(s)
==31415== at 0x4C35E20: bcmp (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31415== by 0x11A624: queue_broadcast (broadcast.c:40)
==31415== by 0x118D93: handle_pending_cannouncement (routing.c:704)
==31415== by 0x1109E3: handle_txout_reply (gossip.c:1796)
==31415== by 0x111177: recv_req (gossip.c:1955)
==31415== by 0x136723: next_plan (io.c:59)
==31415== by 0x137220: do_plan (io.c:387)
==31415== by 0x13725E: io_ready (io.c:397)
==31415== by 0x138B97: io_loop (poll.c:305)
==31415== by 0x111352: main (gossip.c:2022)
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We drop all but the first announcement, so any work that is done for a
channel that we already know is wasted. Pulling this up duplicates
some of the work but allows us to skip the costly txout check.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
`tal_fmt` overallocates the returned string under some circumstances,
meaning that the trailer of the formatted string is unset, but still
considered in `tal_len`. The solution then is to truncate the
formatted string to the real string length. Only necessary here, since
we mix strings and `tal_len`.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We need to make sure all the updates are known to gossip. Since
one is the local update, we change that message to look the same.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Otherwise, we otherwise end up with out-of-order updates
(ie. preceeding announcements).
I assume that is because of the locally-inserted connections.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is done it two parts, since we have to ask the main daemon to do
the lookup for us.
If this becomes a bottleneck, we can have a separate daemon, or even
an RPC pipe to bitcoind ourselves.
Fixes: #403
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our handling of SIGPIPE was incoherent and inconsistent, and we had much
cut & paste between the daemons. They should *ALL* ignore SIGPIPE, and
much of the rest of the boilerplate can be shared, so should be.
Reported-by: @ZmnSCPxj
Fixes: #528
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sometimes we could get into a situation in which we knew the channel
but couldn't find it via the short_channel_id. That'd result in a
replacement which triggered an assert.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
The use of status_failed() requires a stubs update, which fails
with unnamed parameters, so tweak the status.h header as well.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If we side-load a channel, using local-add or the removed JSON-RPC
call, then we could end up in a situation in which a channel is
present, but has no associated channel_announcement. The presence of
the channel_announcement was used to identify new channels, so this
could lead to channels always being considered new. This then caused
the announcements being added to the queue always, resulting in
channel_updates preceeding the announcement.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
We should never be evicting channel_announcements because a) they were
deeply buried and should not change the short_channel_id/tag, b) they
are static.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
It's just a sha256_double, but importantly when we convert it to a
string (in type_to_string, which is used in logging) we use
bitcoin_blkid_to_hex() so it's reversed as people expect.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
If channel_announce is rebroadcast, it should replace the existing one
in-place. We currently only do this if we start from the unsigned one
and replace it with the signed one when we hit 6 confirms.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This would fail, and we'd free an uninitialized pointer.
Also, add us to .gitignore and clear up a comment.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This adds the channel from us to the remote node and activates it with
our local parameters.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Couldn't find a good place to put these messages, we probably want to
do the same capability based request routing that we did for the HSM,
but for now this just defines the message in the master messages file.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
This check is expensive, so just restrict msatoshi going in, as well
as turn off channels charging more than 24x fee.
# 1M nodes:
$ /gossipd/test/run-bench-find_route 1000000 1 > /tmp/out
=> 44164 msec
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can't get them; channel_update doesn't support it.
# 1M nodes:
$ /gossipd/test/run-bench-find_route 1000000 1 > /tmp/out
=> 47677 msec
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Compile this, and link from perfme-start and perfme-stop in your path:
/* Simple wrapper to allow a program to perf itself.
* Copyright Rusty Russell, Blockstream 2015.
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* See <http://www.gnu.org/licenses/>.
*/
#include <ccan/err/err.h>
#include <ccan/str/str.h>
#include <sys/types.h>
#include <unistd.h>
#include <signal.h>
#include <errno.h>
#include <stdio.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#define PERFME_PREFIX "/tmp/perfme."
#define MAX_ENV_ARGS 20
static void write_noerr(int fd)
{
int e = errno;
if (write(fd, "", 1) != 1)
/* Complain about warn_unused_result fascist bullshit */ ;
errno = e;
}
/* Child. Setup pid, run perf. */
static void exec_perf(int pfd[2], const char *perfpid, const char *perfout,
pid_t parent)
{
char pid[STR_MAX_CHARS(pid_t)];
int i, fd;
char *cmd, *args[MAX_ENV_ARGS + 5];
fd = open(perfpid, O_CREAT|O_EXCL|O_WRONLY, 0400);
if (fd < 0) {
write_noerr(pfd[1]);
err(1, "opening %s", perfpid);
}
sprintf(pid, "%u", getpid());
if (write(fd, pid, strlen(pid)) != strlen(pid)) {
write_noerr(pfd[1]);
err(1, "writing to %s", perfpid);
}
close(fd);
sprintf(pid, "%u", parent);
cmd = getenv("PERFME");
if (!cmd)
cmd = "perf record --call-graph dwarf -q";
cmd = strdup(cmd);
for (i = 0; i < MAX_ENV_ARGS; i++) {
args[i] = strtok(i == 0 ? cmd : NULL, " ");
if (!args[i])
break;
}
if (i == 0 || i == MAX_ENV_ARGS)
errx(1, "Too %s args in $PERFME: '%s'",
i ? "many" : "few", getenv("PERFME"));
args[i++] = "-p";
args[i++] = pid;
args[i++] = "-o";
args[i++] = (char *)perfout;
args[i++] = NULL;
execvp(args[0], args);
write_noerr(pfd[1]);
err(1, "Execing %s", args[0]);
}
int main(int argc, char *argv[])
{
pid_t parent = argv[1] ? atoi(argv[1]) : getppid();
char perfout[sizeof(PERFME_PREFIX) + STR_MAX_CHARS(parent)];
char perfpid[sizeof(perfout) + sizeof(".pid")];
err_set_progname(argv[0]);
sprintf(perfpid, PERFME_PREFIX "%u.pid", parent);
if (strends(argv[0], "perfme-stop")) {
char pid[STR_MAX_CHARS(pid_t)];
int r, fd = open(perfpid, O_RDONLY);
if (fd < 0)
err(1, "Opening %s", perfpid);
r = read(fd, pid, sizeof(pid) - 1);
if (r < 0)
err(1, "Reading %s", perfpid);
pid[r] = 0;
if (unlink(perfpid) != 0)
warn("Unlinking %s", perfpid);
if (atoi(pid) <= 0)
errx(1, "Invalid pid '%s' from %s", pid, perfpid);
if (kill(atoi(pid), SIGTERM) != 0)
err(1, "Stopping %s", pid);
exit(0);
} else if (strends(argv[0], "perfme-start")) {
int pfd[2];
sprintf(perfout, PERFME_PREFIX "%u", parent);
/* Use pipe to detect successful exec. */
if (pipe(pfd) != 0)
err(1, "Creating pipe");
switch (fork()) {
case 0:
close(pfd[0]);
fcntl(pfd[1], F_SETFD,
fcntl(pfd[1], F_GETFD)|FD_CLOEXEC);
exec_perf(pfd, perfpid, perfout, parent);
case -1:
err(1, "Forking");
default:
/* Parent. Wait for child. */
close(pfd[1]);
if (read(pfd[0], perfpid, 1) == 1)
exit(1);
fprintf(stderr, "Perf recording into %s\n", perfout);
sleep(1);
exit(0);
}
}
errx(1, "Unknown name: am I perfme-start or perfme-stop?");
}
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
When gossipd sends a message, have a gossip_index. When it gets back a
peer, the current gossip_index is included, so it can know exactly where
it's up to.
Most of this is mechanical plumbing through openingd, channeld and closingd,
even though openingd and closingd don't (currently) read gossip, so their
gossip_index will be unchanged.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
All peers come from gossipd, and maintain an fd to talk to it. Sometimes
we hand the peer back, but to avoid a race, we always recreated it.
The race was that a daemon closed the gossip_fd, which made gossipd
forget the peer, then master handed the peer back to gossipd. We stop
the race by never closing the gossipfd, but hand it back to gossipd
for closing.
Now gossipd has to accept two fds, but the handling of peers is far
clearer.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We should also go through and use consistent nomenclature on functions which
are used with a local peer ("lpeer_xxx"?) and those with a remote peer
("rpeer_xxx"?) but this is minimal.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This will later be used to determine whether or not we should announce
ourselves as a node.
Signed-off-by: Christian Decker <decker.christian@gmail.com>
Test objects must be added to $(ALL_OBJS) so they correctly depend on
CCAN headers etc.
Also, each test in a subdir must depend on headers and src in the parent
directory, as it will often #include them directly.
Reported-by: Christian Decker
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And nail "make check-source" to that specific version (which is a commit id,
not a branch name, so needs a different syntax for git).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And we report these through the getpeers JSON RPC again (carefully: in
our reconnect tests we can get duplicates which this patch now filters
out).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In future it will have TOR support, so the name will be awkward.
We collect the to/fromwire functions in common/wireaddr.c, and the
parsing functions in lightningd/netaddress.c.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We need to derive this from the fd when they connect in, but we already
know it if we're connecting out.
We want this so we can tell (in next few patches) master the peer's address.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
954a3990fa had two errors:
1) We created the handoff message *before* we sent the final packet, meaning
that the cryptostate was out-of-sync.
2) We called io_wait() on the output side of a duplex connection: it has
to be io_wait_out().
This time, stress testing for 2 hours revealed no more problems.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In this case, it was a gossip message half-sent, when we asked the peer
to be released. Fix the problem in general by making send_peer_with_fds()
wait until after the next packet.
test_routing_gossip/lightning-4/log:
b'lightning_openingd(8738): TRACE: First per_commit_point = 02e2ff759ed70c71f154695eade1983664a72546ebc552861f844bff5ea5b933bf'
b'lightning_openingd(8738): TRACE: Failed hdr decrypt with rn=11'
b'lightning_openingd(8738): STATUS_FAIL_PEER_IO: Reading accept_channel: Success'
test_routing_gossip/lightning-5/log:
b'lightning_gossipd(8461): UPDATE WIRE_GOSSIP_PEER_NONGOSSIP'
b'lightning_gossipd(8461): UPDATE WIRE_GOSSIP_PEER_NONGOSSIP'
b'lightningd(8308): Failed to get netaddr for outgoing: Transport endpoint is not connected'
The problem occurs here on release, but could be on any place where we hand
a peer over when using ccan/io. Note the other case (channel.c).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It makes it impossible to embed an ipaddr in another structure, since we
always try to skip over any zeroes, which may swallow a following field.
Do the skip specially for the case where we're parsing routing messages:
we never use padding for our own internal messages anyway.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Now the flow is much simpler from a lightningd POV:
1. If we want to connect to a peer, just send gossipd `gossipctl_reach_peer`.
2. Every new peer, gossipd hands up to lightningd, with global/local features
and the peer fd and a gossip fd using `gossip_peer_connected`
3. If lightningd doesn't want it, it just hands the peerfd and global/local
features back to gossipd using `gossipctl_handle_peer`
4. If a peer sends a non-gossip msg (eg `open_channel`) the gossipd sends
it up using `gossip_peer_nongossip`.
5. If lightningd wants to fund a channel, it simply calls `release_channel`.
Notes:
* There's no more "unique_id": we use the peer id.
* For the moment, we don't ask gossipd when we're told to list peers, so
connected peers without a channel don't appear in the JSON getpeers API.
* We add a `gossipctl_peer_addrhint` for the moment, so you can connect to
a specific ip/port, but using other sources is a TODO.
* We now (correctly) only give up on reaching a peer after we exchange init
messages, which changes the test_disconnect case.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This fixes the only case where the master currently has to write directly
to the peer: re-sending an error. We make gossipd do it, by adding
a new gossipctl_fail_peer message.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
In particular, the main daemon needs to pass it about (marshal/unmarshal)
but it won't need to actually use it after the next patch.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We were sending a channeld message to onchaind, which was v. confusing
due to overlap. We make all the numbers distinct, which means we can
also add an assert() that it's valid for that daemon, which catches
such errors immediately.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This change is really to allow us to have a --dev-fail-on-subdaemon-fail option
so we can handle failures from subdaemons generically.
It also neatens handling so we can have an explicit callback for "peer
did something wrong" (which matters if we want to close the channel in
that case).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. The code to skip over padding didn't take into account max.
2. It also didn't use symbolic names.
3. We are not supposed to fail on unknown addresses, just stop parsing.
4. We don't use the read_ip/write_ip code, so get rid of it.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I missed these when I removed the legacy daemon. We also remove the
min_blocks field which was always 0.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Use a negative timestamp as the flag for this, making the test simple.
This allows valgrind to detect that we're accessing them prematurely,
including across the wire on gossip_getchannels_entry.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
The test is could actually go each way, since for 1000000 the fee is
the same either way.
Increase to 300000, and add an extra test when the alternate path
is disabled.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I had a routing problem, and wrote a simple unit test which passed. So
I wrote one which copied the failure case (and importantly, had a non-1
fee factor), which triggerd it.
In that real example, we underflowed which resulted in us not finding
a route. Simply don't consider routes which are infinite.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Since we initialize last_timestamp to 0, we ignore any initial update
with this timestamp. Don't compare it if we don't already have an
update, and don't initialize it, so valgrind can tell us if we use
it accidentally.
b'lightning_gossipd(3368): TRACE: Received channel_update for channel 6892:2:1(0)'
b'lightning_gossipd(3368): TRACE: Ignoring outdated update.'
b'lightning_gossipd(3368): TRACE: Received channel_update for channel 6893:2:1(1)'
b'lightning_gossipd(3368): TRACE: Channel 6893:2:1(1) was updated.'
The same logic applies to node_updates, so we do the same there.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
