lightning-bolts/07-routing-gossip.md

BOLT #7: P2P Node and Channel Discovery

This specification describes simple node discovery, channel discovery and channel update mechanisms which do not rely on a third-party to disseminate the information.

Node and channel discovery serve two different purposes:

• Channel discovery allows the creation and maintenance of a local view of the network's topology such that the node can discover routes to the desired destination.
• Node discovery allows nodes to broadcast their ID, host and port, such that other nodes can open connections and establish payment channels.

Peers in the network exchange channel_announcement messages that contain information about new channels between two nodes. They can also exchange node_announcement messages which supply additional information about nodes, and channel_update messages which update information about a channel.

There can only be one valid channel_announcement for any channel, but multiple node_announcement messages are possible (to update node information), and at least two channel_update messages are expected.

The announcement_signatures message

This is a direct message between two endpoints of a channel and serves as an opt-in mechanism to allow the announcement of the channel to the rest of the network. It contains the necessary signatures by the sender to construct the channel_announcement message.

1. type: 259 (announcement_signatures)
2. data:
   • [32:channel-id]
   • [8:short-channel-id]
   • [64:node-signature]
   • [64:bitcoin-signature]

The willingness of the endpoints to announce the channel is signaled during the connection setup by setting a channels_public bit in the localfeatures field.

Requirements

If both endpoints have signaled that they'd like to publish the channel then the announcement_signatures message MUST be sent, otherwise they MUST NOT be sent.

If sent, announcement_signatures messages MUST NOT be sent until funding_locked has been sent, and the funding transaction is has at least 6 confirmations.

The short-channel-id is the unique description of the funding transaction. It is constructed with the most significant 3 bytes as the block height, the next 3 bytes indicating the transaction index within the block, and the least significant two bytes indicating the output index which pays to the channel.

The announcement_signatures message is created by constructing a channel_announcement message corresponding to the newly established channel, and sign it with the secrets matching their node-id and bitcoin-key, and send them using an announcement_signatures. The recipient MAY fail the channel if the node-signature or bitcoin-signature is incorrect. The recipient SHOULD queue the channel_announcement message for its peers if it has sent and received a valid announcement_signatures message.

On reconnection, a node SHOULD retransmit the announcement_signatures message if it has not received an announcement_signatures message, and MUST respond to the first announcement_signatures message after reconnection with its own announcement_signatures message.

The channel_announcement message

This message contains ownership information about a channel. It ties each on-chain Bitcoin key to the lightning node key, and vice-versa. The channel is not really usable until at least one side has announced its fee levels and expiry using channel_update.

In order to prove the existence of channel between node-1 and node-2 we need to:

1. Prove that the funding transaction pays to bitcoin-key-1 and bitcoin-key-2.
2. Prove node-1 owns bitcoin-key-1
3. Prove node-2 owns bitcoin-key-2

The first one is done by assuming that all nodes know the unspent transaction outputs, and thus can find the output given by short-channel-id and validate that it is indeed a P2WSH funding transaction output as to those keys specified in BOLT #3.

The second two are done by explicit signatures (bitcoin-signature-1 and bitcoin-signature-2, generated by each bitcoin-key and signing the corresponding node-id.

We also need to prove that node-1 and node-2 both agree on this announcement message; that is done by having a signature from each node-id signing the message (node-signature-1 and node-signature-2).

1. type: 256 (channel_announcement)
2. data:
   • [64:node-signature-1]
   • [64:node-signature-2]
   • [64:bitcoin-signature-1]
   • [64:bitcoin-signature-2]
   • [8:short-channel-id]
   • [33:node-id-1]
   • [33:node-id-2]
   • [33:bitcoin-key-1]
   • [33:bitcoin-key-2]
   • [2:len]
   • [len:features]

Requirements

The creating node MUST set short-channel-id to refer to the confirmed funding transaction as specified in BOLT #2. The corresponding output MUST be a P2WSH as described in BOLT #3.

The creating node MUST set node-id-1 and node-id-2 to the public keys of the two nodes who are operating the channel, such that node-id-1 is the numerically-lesser of the two DER encoded keys. ascending numerical order, and MUST set bitcoin-key-1 and bitcoin-key-2 to funding-pubkeys of node-id-1 and node-id-2 respectively.

The creating node MUST compute the double-SHA256 hash h of the message, starting at offset 256, up to the end of the message. Thus the hash skips the 4 signatures, but hashes the rest of the message, including any future fields appended to the end. node-signature-1 and node-signature-2 MUST be valid signatures of the hash h using the secret associated with node-id-1 and node-id-2 respectively. bitcoin-signature-1 and bitcoin-signature-2 MUST be valid signatures of the hash h using the secret associated with bitcoin-key-1 and bitcoin-key-2 respectively.

The creating node SHOULD set len to the minimum length required to hold the features bits it sets.

The receiving node MUST ignore the message if the output specified by short-channel-id does not correspond to a P2WSH using bitcoin-key-1 and bitcoin-key-2 as specified in BOLT #3. The receiving node MUST ignore the message if this output is spent.

Otherwise, the receiving node SHOULD fail the connection if bitcoin-signature-1, bitcoin-signature-2, node-signature-1 or node-signature-2 are invalid or not correct.

Otherwise, if node-id-1 or node-id-2 are blacklisted, it SHOULD ignore the message.

Otherwise, if the transaction referred to was not previously announced as a channel, the receiving node SHOULD queue the message for rebroadcasting, but MAY choose not to for messages longer than the minimum expected length. If it has previously received a valid channel_announcement for the same transaction in the same block, but different node-id-1 or node-id-2, it SHOULD blacklist the previous message's node-id-1 and node-id-2 as well as this node-id-1 and node-id-2 and forget channels connected to them, otherwise it SHOULD store this channel_announcement.

The receiving node SHOULD NOT route through a channel which has an unknown feature bit set which is even.

The receiving node SHOULD forget a channel once its funding output has been spent or reorganized out.

Rationale

Requiring both nodes to sign indicates they are both willing to route other payments via this node (ie. take part of the public network). Requiring the Bitcoin signatures proves they control the channel.

The blacklisting of conflicting nodes means that we disallow multiple different announcements: no node should ever do this, as it implies that keys have leaked.

While channels shouldn't be advertised before they are sufficiently deep, the requirement against rebroadcasting only applies if the transaction hasn't moved to a different block.

To avoid having to store excessive-sized messages, yet allow reasonable expansion in future, nodes are allowed to restrict rebroadcasting (perhaps statistically).

New channel features are possible in future; backwards compatible (or optional) ones will have odd feature bits, incompatible ones will have even feature bits ("It's OK to be odd!". These will be propagated by nodes even if they can't use the announcements themselves.

The node_announcement message

This allows a node to indicate extra data associated with it, in addition to its public key. To avoid trivial denial of service attacks, nodes for which a channel is not already known are ignored.

1. type: 257 (node_announcement)
2. data:
   • [64:signature]
   • [4:timestamp]
   • [33:node-id]
   • [3:rgb-color]
   • [32:alias]
   • [2:flen]
   • [flen:features]
   • [2:addrlen]
   • [addrlen:addresses]

The timestamp allows ordering in the case of multiple announcements; the rgb-color and alias allow intelligence services to give their nodes cool monikers like IRATEMONK and WISTFULTOLL and use the color black.

addresses allows the node to announce its willingness to accept incoming network connections: it contains series of address descriptors for connecting to the node. The first byte describes the address type, followed by the appropriate number of bytes for that type.

The following address descriptor types are defined:

1. 0: padding. data = none (length 0).
2. 1: IPv4. data = [4:ipv4-addr][2:port] (length 6)
3. 2: IPv6. data = [16:ipv6-addr][2:port] (length 18)

Requirements

The creating node MUST set timestamp to be greater than any previous node_announcement it has created. It MAY base it on a UNIX timestamp. It MUST set signature to the signature of the double-SHA256 of the entire remaining packet after signature using the key given by node-id. It MAY set alias and rgb-color to customize their node's appearance in maps and graphs, where the first byte of rgb is the red value, the second byte is the green value and the last byte is the blue value. It MUST set alias to a valid UTF-8 string, with any alias bytes following equal to zero.

The creating node SHOULD fill addresses with an address descriptor for each public network address which expects incoming connections, and MUST set addrlen to the number of bytes in addresses. Non-zero typed address descriptors MUST be placed in ascending order; any number of zero-typed address descriptors MAY be placed anywhere, but SHOULD only be used for aligning fields following addresses.

The creating node MUST NOT create a type 1 or type 2 address descriptor with port equal to zero, and SHOULD ensure ipv4-addr and ipv6-addr are routable addresses. The creating node MUST NOT include more than one address descriptor of the same type.

The creating node SHOULD set flen to the minimum length required to hold the features bits it sets.

The receiving node SHOULD fail the connection if node-id is not a valid compressed public key, and MUST NOT further process the message.

The receiving node SHOULD fail the connection if signature is not a valid signature using node-id of the double-SHA256 of the entire message following the signature field (including unknown fields following alias), and MUST NOT further process the message.

The receiving node SHOULD ignore the first address descriptor which does not match the types defined above. The receiving node SHOULD fail the connection if addrlen is insufficient to hold the address descriptors of the known types.

The receiving node SHOULD ignore the message if node-id is not previously known from a channel_announcement message, or if timestamp is not greater than the last-received node_announcement from this node-id. Otherwise, if the timestamp is greater than the last-received node_announcement from this node-id the receiving node SHOULD queue the message for rebroadcasting, but MAY choose not to for messages longer than the minimum expected length.

The receiving node SHOULD NOT connect to a node which has an unknown feature bit set in the node_announcement which is even.

The receiving node MAY use rgb and alias to reference nodes in interfaces, but SHOULD insinuate their self-signed origin.

Rationale

New node features are possible in future; backwards compatible (or optional) ones will have odd feature bits, incompatible ones will have even feature bits. These will be propagated by nodes even if they can't use the announcements themselves.

New address types can be added in future; as address descriptors have to be ordered in ascending order so they will be safely ignored. Future fields beyond addresses can still be added, optionally with padding within addresses if they require certain alignment.

The channel_update message

After a channel has been initially announced, each side independently announces its fees and minimum expiry for HTLCs. It uses the 8-byte channel shortid which matches the channel_announcement and one byte to indicate which end this is. It can do this multiple times, if it wants to change fees.

1. type: 258 (channel_update)
2. data:
   • [64:signature]
   • [8:short-channel-id]
   • [4:timestamp]
   • [2:flags]
   • [2:cltv-expiry-delta]
   • [4:htlc-minimum-msat]
   • [4:fee-base-msat]
   • [4:fee-proportional-millionths]

Requirements

The creating node MUST set signature to the signature of the double-SHA256 of the entire remaining packet after signature using its own node-id.

The creating node MUST set short-channel-id to match those in the already-sent channel_announcement message, and MUST set the least-significant bit of flags to 0 if the creating node is node-id-1 in that message, otherwise 1. It MUST set other bits of flags to zero.

The creating node MUST set timestamp to greater than zero, and MUST set it to greater than any previously-sent channel_update for this channel.

It MUST set cltv-expiry-delta to the number of blocks it will subtract from an incoming HTLC's cltv-expiry. It MUST set htlc-minimum-msat to the minimum HTLC value it will accept, in millisatoshi. It MUST set fee-base-msat to the base fee it will charge for any HTLC, in millisatoshi, and fee-proportional-millionths to the amount it will charge per millionth of a satoshi.

The receiving node MUST ignore flags other than the least significant bit. The receiving node SHOULD ignore ipv6 if port is zero. It SHOULD ignore the message if short-channel-id does not correspond to a previously known, unspent channel from channel_announcement, otherwise the node-id is taken from the channel_announcement node-id-1 if least-significant bit of flags is 0 or node-id-2 otherwise. The receiving node SHOULD fail the connection if signature is not a valid signature using node-id of the double-SHA256 of the entire message following the signature field (including unknown fields following fee-proportional-millionths), and MUST NOT further process the message.

The receiving node SHOULD ignore the message if timestamp is not greater than than the last-received channel_announcement for this channel and node-id. Otherwise, if the timestamp is equal to the last-received channel_announcement and the fields other than signature differ, the node MAY blacklist this node-id and forget all channels associated with it. Otherwise the receiving node SHOULD queue the message for rebroadcasting, but MAY choose not to for messages longer than the minimum expected length.

Initial Sync

Upon establishing a connection, the two endpoints negotiate whether to perform an initial sync by setting the initial_routing_sync flags in the init message. The endpoint SHOULD set the initial_routing_sync flag if it requires a full copy of the other endpoint's routing state. Upon receiving an init message with the initial_routing_sync flag set the node sends channel_announcements, channel_updates and node_announcements for all known channels and nodes as if they were just received.

If the initial_routing_sync flag is not set, or initial sync was completed, then the node resumes normal operation, see the Rebroadcasting section for details.

Rebroadcasting

Nodes receiving a new channel_announcement or a channel_update or node_update with an updated timestamp update their local view of the network's topology accordingly.

Once the announcement has been processed it is added to a list of outgoing announcements (perhaps replacing older updates) to the processing node's peers, which will be flushed at regular intervals. This store and delayed forward broadcast is called a staggered broadcast

If, after applying the changes from the announcement, there are no channels associated with the announcing node, then the receiving node MAY purge the announcing node from the set of known nodes. Otherwise the receiving node updates the metadata and stores the signature associated with the announcement. This will later allow the receiving node to rebuild the announcement for its peers.

After processing the announcement the receiving node adds the announcement to a list of outgoing announcements.

Requirements

Each node SHOULD flush outgoing announcements once every 60 seconds, independently of the arrival times of announcements, resulting in a staggered announcement and deduplication of announcements.

Nodes MAY re-announce their channels regularly, however this is discouraged in order to keep the resource requirements low.

Nodes SHOULD send all channel_announcement messages followed by the latest node_announcement and channel_update messages upon connection establishment.

Rationale

Batching announcements form a natural ratelimit with low overhead.

The sending of all announcements on reconnection is naive, but simple, and allows bootstrap for new nodes as well as updating for nodes which have been offline for some time.

HTLC Fees

The node creating channel_update SHOULD accept HTLCs which pay a fee equal or greater than:

fee-base-msat + htlc-amount-msat * fee-proportional-millionths / 1000000

The node creating channel_update SHOULD accept HTLCs which pay an older fee for some time after sending channel_update to allow for propagation delay.

Recommendations for Routing

As the fee is proportional, it must be calculated backwards from the destination to the source: only the amount required at the final destination is known initially.

When calculating a route for an HTLC, the cltv-expiry-delta and the fee both need to be considered: the cltv-expiry-delta contributes to the time that funds will be unavailable on worst-case failure. The tradeoff between these two is unclear, as it depends on the reliability of nodes.

Other more advanced considerations involve diversity of routes to avoid single points of failure and detection, and channel balance of local channels.

References

This work is licensed under a Creative Commons Attribution 4.0 International License.