lightning-bolts/07-routing-gossip.md

BOLT #7: P2P Node and Channel Discovery

This specification describes simple node discovery, channel discovery, and channel update mechanisms that do not rely on a third-party to disseminate the information.

Node and channel discovery serve two different purposes:

• Channel discovery allows the creation and maintenance of a local view of the network's topology, so that a node can discover routes to desired destinations.
• Node discovery allows nodes to broadcast their ID, host, and port, so that other nodes can open connections and establish payment channels with them.

To support channel discovery, peers in the network exchange channel_announcement messages containing information regarding new channels between the two nodes. They can also exchange channel_update messages, which update information about a channel. There can only be one valid channel_announcement for any channel, but at least two channel_update messages are expected.

To support node discovery, peers exchange node_announcement messages, which supply additional information about the nodes. There may be multiple node_announcement messages, in order to update the node information.

Table of Contents

• The announcement_signatures Message
• The channel_announcement Message
• The node_announcement Message
• The channel_update Message
• Initial Sync
• Rebroadcasting
• HTLC Fees
• Pruning the Network View
• Recommendations for Routing
• References

The announcement_signatures Message

This is a direct message between the two endpoints of a channel and serves as an opt-in mechanism to allow the announcement of the channel to the rest of the network. It contains the necessary signatures, by the sender, to construct the channel_announcement message.

1. type: 259 (announcement_signatures)
2. data:
   • [32:channel_id]
   • [8:short_channel_id]
   • [64:node_signature]
   • [64:bitcoin_signature]

The willingness of the initiating node to announce the channel is signaled during channel opening by setting the announce_channel bit in channel_flags (see BOLT #2).

Requirements

The announcement_signatures message is created by constructing a channel_announcement message, corresponding to the newly established channel, and signing it with the secrets matching an endpoint's node_id and bitcoin_key. The message is then sent using an announcement_signatures message.

The short_channel_id is the unique description of the funding transaction. It is constructed as follows:

1. the most significant 3 bytes: indicating the block height
2. the next 3 bytes: indicating the transaction index within the block
3. the least significant 2 bytes: indicating the output index that pays to the channel.

A node:

• if the open_channel message has the announce_channel bit set:
  • MUST send the announcement_signatures message.
    • MUST NOT send announcement_signatures messages until funding_locked has been sent AND the funding transaction has at least six confirmations.
• otherwise:
  • MUST NOT send the announcement_signatures message.
• upon reconnection:
  • MUST respond to the first announcement_signatures message with its own announcement_signatures message.
  • if it has NOT received an announcement_signatures message:
    • SHOULD retransmit the announcement_signatures message.

A recipient node:

• if the node_signature OR the bitcoin_signature is NOT correct:
  • MAY fail the channel.
• if it has sent AND received a valid announcement_signatures message:
  • SHOULD queue the channel_announcement message for its peers.

The channel_announcement Message

This message contains ownership information regarding a channel. It ties each on-chain Bitcoin key to the associated Lightning node key, and vice-versa. The channel is not practically usable until at least one side has announced its fee levels and expiry, using channel_update.

Proving the existence of a channel between node_1 and node_2 requires:

1. proving that the funding transaction pays to bitcoin_key_1 and bitcoin_key_2
2. proving that node_1 owns bitcoin_key_1
3. proving that node_2 owns bitcoin_key_2

Assuming that all nodes know the unspent transaction outputs, the first proof is accomplished by a node finding the output given by the short_channel_id and verifying that it is indeed a P2WSH funding transaction output for those keys specified in BOLT #3.

The last two proofs are accomplished through explicit signatures: bitcoin_signature_1 and bitcoin_signature_2 are generated for each bitcoin_key and each of the corresponding node_ids are signed.

It's also necessary to prove that node_1 and node_2 both agree on the announcement message: this is accomplished by having a signature from each node_id (node_signature_1 and node_signature_2) signing the message.

1. type: 256 (channel_announcement)
2. data:
   • [64:node_signature_1]
   • [64:node_signature_2]
   • [64:bitcoin_signature_1]
   • [64:bitcoin_signature_2]
   • [2:len]
   • [len:features]
   • [32:chain_hash]
   • [8:short_channel_id]
   • [33:node_id_1]
   • [33:node_id_2]
   • [33:bitcoin_key_1]
   • [33:bitcoin_key_2]

Requirements

The creating node:

• MUST set chain_hash to the 32-byte hash that uniquely identifies the chain that the channel was opened within:
  • MUST set the Bitcoin blockchain chain_hash value (encoded in hex) to: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f.
• MUST set short_channel_id to refer to the confirmed funding transaction, as specified in BOLT #2.
  • Note: the corresponding output MUST be a P2WSH, as described in BOLT #3.
• MUST set node_id_1 and node_id_2 to the public keys of the two nodes operating the channel, such that node_id_1 is the numerically-lesser of the two DER-encoded keys sorted in ascending numerical order.
• MUST set bitcoin_key_1 and bitcoin_key_2 to node_id_1 and node_id_2's respective funding_pubkeys.
• MUST compute the double-SHA256 hash h of the message, beginning at offset 256, up to the end of the message.
  • Note: the hash skips the 4 signatures but hashes the rest of the message, including any future fields appended to the end.
• MUST verify that node_signature_1 and node_signature_2 are valid signatures of the hash h (using node_id_1 and node_id_2's respective secrets).
• MUST verify that bitcoin_signature_1 and bitcoin_signature_2 are valid signatures of the hash h (using bitcoin_key_1 and bitcoin_key_2's respective secrets).
• SHOULD set len to the minimum length required to hold the features bits it sets.

The receiving node:

• MUST verify the integrity AND authenticity of the message by verifying the signatures.
• if there is an unknown even bit in the features field:
  • MUST NOT parse the remainder of the message.
  • MUST NOT add the channel to its local network view.
  • SHOULD NOT forward the announcement.
• if the short_channel_id's output does NOT correspond to a P2WSH (using bitcoin_key_1 and bitcoin_key_2, as specified in BOLT #3) OR the output is spent:
  • MUST ignore the message.
• if the specified chain_hash is unknown to the receiver:
  • MUST ignore the message.
• otherwise:
  • if bitcoin_signature_1, bitcoin_signature_2, node_signature_1 OR node_signature_2 are invalid OR NOT correct:
    • SHOULD fail the connection.
  • otherwise:
    • if node_id_1 OR node_id_2 are blacklisted:
      • SHOULD ignore the message.
    • otherwise:
      • if the transaction referred to was NOT previously announced as a channel:
        • SHOULD queue the message for rebroadcasting.
        • MAY choose NOT to for messages longer than the minimum expected length.
    • if it has previously received a valid channel_announcement, for the same transaction, in the same block, but for a different node_id_1 or node_id_2:
      • SHOULD blacklist the previous message's node_id_1 and node_id_2, as well as this node_id_1 and node_id_2 AND forget any channels connected to them.
    • otherwise:
      • SHOULD store this channel_announcement.
• once its funding output has been spent OR reorganized out:
  • SHOULD forget a channel.

Rationale

Both nodes are required to sign to indicate they are willing to route other payments via this channel (i.e. be part of the public network); requiring their Bitcoin signatures proves that they control the channel.

The blacklisting of conflicting nodes disallows multiple different announcements. Such conflicting announcements should never be broadcast by any node, as this implies that keys have leaked.

While channels should not be advertised before they are sufficiently deep, the requirement against rebroadcasting only applies if the transaction has not moved to a different block.

In order to avoid storing excessively-large messages, yet still allow for reasonable future expansion, nodes are permitted to restrict rebroadcasting (perhaps statistically).

New channel features are possible in the future: backwards compatible (or optional) features will have odd feature bits, while incompatible features will have even feature bits ("It's OK to be odd!"). Incompatible features will result in the announcement not being forwarded by nodes that do not understand them.

The node_announcement Message

This message allows a node to indicate extra data associated with it, in addition to its public key. To avoid trivial denial of service attacks, nodes not associated with an already known channel are ignored.

1. type: 257 (node_announcement)
2. data:
   • [64:signature]
   • [2:flen]
   • [flen:features]
   • [4:timestamp]
   • [33:node_id]
   • [3:rgb_color]
   • [32:alias]
   • [2:addrlen]
   • [addrlen:addresses]

The timestamp allows ordering of messages, in the case of multiple announcements; the rgb_color and alias allow intelligence services to give their nodes cool monikers like 'IRATEMONK' and 'WISTFULTOLL' and to use the color black.

addresses allows a node to announce its willingness to accept incoming network connections: it contains a series of address descriptors for connecting to the node. The first byte describes the address type and is followed by the appropriate number of bytes for that type.

The following address descriptor types are defined:

• 0: padding; data = none (length 0)
• 1: ipv4; data = [4:ipv4_addr][2:port] (length 6)
• 2: ipv6; data = [16:ipv6_addr][2:port] (length 18)
• 3: tor v2 onion service; data = [10:onion_addr][2:port] (length 12)
  • version 2 onion service addresses; Encodes an 80-bit, truncated SHA-1 hash of a 1024-bit RSA public key for the onion service (a.k.a. Tor hidden service).
• 4: tor v3 onion service; data [35:onion_addr][2:port] (length 37)
  • version 3 (prop224) onion service addresses; Encodes: [32:32_byte_ed25519_pubkey] || [2:checksum] || [1:version], where checksum = sha3(".onion checksum" | pubkey || version)[:2].

Requirements

The creating node:

• MUST set timestamp to be greater than that of any previous node_announcement it has previously created.
  • MAY base it on a UNIX timestamp.
• MUST set signature to the signature of the double-SHA256 of the entire remaining packet after signature (using the key given by node_id).
• MAY set alias AND rgb_color to customize its appearance in maps and graphs.
  • Note: the first byte of rgb is the red value, the second byte is the green value, and the last byte is the blue value.
• MUST set alias to a valid UTF-8 string, with any alias trailing-bytes equal to zero.
• SHOULD fill addresses with an address descriptor for each public network address that expects incoming connections.
• MUST set addrlen to the number of bytes in addresses.
• MUST place non-zero typed address descriptors in ascending order.
• MAY place any number of zero-typed address descriptors anywhere.
• SHOULD use placement only for aligning fields that follow addresses.
• MUST NOT create a type 1 OR type 2 address descriptor with port equal to zero.
• SHOULD ensure ipv4_addr AND ipv6_addr are routable addresses.
• MUST NOT include more than one address descriptor of the same type.
• SHOULD set flen to the minimum length required to hold the features bits it sets.

The receiving node:

• if node_id is NOT a valid compressed public key:
  • SHOULD fail the connection.
  • MUST NOT further process the message.
• if signature is NOT a valid signature (using node_id of the double-SHA256 of the entire message following the signature field, including unknown fields following alias):
  • SHOULD fail the connection.
  • MUST NOT further process the message.
• if features field contains unknown even bits:
  • MUST NOT parse the remainder of the message.
  • MAY discard the message altogether.
• MAY forward node_announcements that contain an unknown features bit, regardless of if it has parsed the announcement or not.
• SHOULD ignore the first address descriptor that does NOT match the types defined above.
• if addrlen is insufficient to hold the address descriptors of the known types:
  • SHOULD fail the connection.
• if port is equal to zero:
  • SHOULD ignore ipv6_addr OR ipv4_addr.
• if node_id is NOT previously known from a channel_announcement message, OR if timestamp is NOT greater than the last-received node_announcement from this node_id:
  • SHOULD ignore the message.
• otherwise:
  • if timestamp is greater than the last-received node_announcement from this node_id:
    • SHOULD queue the message for rebroadcasting.
    • MAY choose NOT to queue messages longer than the minimum expected length.
• SHOULD NOT connect to a node which has an unknown, even features bit set in the node_announcement.
• MAY use rgb_color AND alias to reference nodes in interfaces.
  • SHOULD insinuate their self-signed origin.

Rationale

New node features are possible in the future; backwards compatible (or optional) ones will have odd feature bits, incompatible ones will have even feature bits. These may be propagated by nodes even if they can't use the announcements themselves.

New address types can be added in the future; as address descriptors have to be ordered in ascending order, unknown ones can be safely ignored. Future fields beyond addresses can still be added, optionally with padding within addresses if they require certain alignment.

The channel_update Message

After a channel has been initially announced, each side independently announces the fees and minimum expiry delta it requires to relay HTLCs through this channel. Each uses the 8-byte channel shortid that matches the channel_announcement and 1 bit in the flags field to indicate which end this is. A node can do this multiple times, if it wants to change fees.

Note that the channel_update message is only useful in the context of relaying payments, not sending payments. When making a payment A -> B -> C -> D, only the channel_updates related to channels B -> C (announced by B) and C -> D (announced by C) will come into play. When building the route, amounts and expiries for HTLCs need to be calculated backward from the destination to the source. The initial exact value for amount_msat and minimal value for cltv_expiry, which are to be used for the last HTLC in the route, are provided in the payment request (see BOLT #11).

A node MAY still create a channel_update to communicate the channel parameters to the other endpoint, even though the channel has not been announced, e.g., because the announce_channel bit was not set.

For further privacy such a channel_update MUST NOT be forwarded to other peers. Note that such a channel_update that is not preceded by a channel_announcement is invalid to any other peer and would be discarded.

1. type: 258 (channel_update)
2. data:
   • [64:signature]
   • [32:chain_hash]
   • [8:short_channel_id]
   • [4:timestamp]
   • [2:flags]
   • [2:cltv_expiry_delta]
   • [8:htlc_minimum_msat]
   • [4:fee_base_msat]
   • [4:fee_proportional_millionths]

The flags bitfield is used to indicate the direction of the channel this update concerns: it identifies the node that this update originated from and signals various options concerning the channel. The following table specifies the meaning of the individual bits:

Bit Position	Name	Meaning
0	direction	Direction this update refers to.
1	disable	Disable the channel.

Requirements

The creating node MUST set signature to the signature of the double-SHA256 of the entire remaining packet after signature, using its own node_id.

The creating node MUST set chain_hash and short_channel_id to match the 32-byte hash and 8-byte channel ID that uniquely identifies the channel within the channel_announcement message.

The creating node MUST set the direction bit of flags to 0 if the creating node is node_id_1 in that message, otherwise 1. Bits which are not assigned a meaning must be set to 0.

A node MAY create and send a channel_update with the disable bit set to signal the temporary unavailability of a channel, e.g., due to loss of connectivity, or the permanent unavailability, e.g., ahead of an on-chain settlement. A subsequent channel_update with the disable bit unset MAY re-enable the channel.

The creating node MUST set timestamp to greater than zero, and MUST set it to greater than any previously-sent channel_update for this short_channel_id, and SHOULD base it on a UNIX timestamp. It MUST set cltv_expiry_delta to the number of blocks it will subtract from an incoming HTLCs cltv_expiry. It MUST set htlc_minimum_msat to the minimum HTLC value the other end of the channel will accept, in millisatoshi. It MUST set fee_base_msat to the base fee it will charge for any HTLC, in millisatoshi, and fee_proportional_millionths to the amount it will charge per transferred satoshi in millionths of a satoshi.

The receiving nodes MUST ignore the channel_update if it does not correspond to one of its own channels, if the short_channel_id does not match a previous channel_announcement, or if the channel has been closed in the meantime. It SHOULD accept channel_updates for its own channels in order to learn the other end's forwarding parameters, even for non-public channels.

The node_id for the signature verification is taken from the corresponding channel_announcement: node_id_1 if the least-significant bit of flags is 0 or node_id_2 otherwise. The receiving node SHOULD fail the connection if signature is not a valid signature using node_id of the double-SHA256 of the entire message following the signature field (including unknown fields following fee_proportional_millionths), and MUST NOT further process the message.

The receiving node MUST ignore the channel update if the specified chain_hash value is unknown, meaning it isn't active on the specified chain.

The receiving node SHOULD ignore the message if timestamp is not greater than that of the last-received channel_announcement for this short_channel_id and node_id. Otherwise, if the timestamp is equal to the last-received channel_announcement and the fields other than signature differ, the node MAY blacklist this node_id and forget all channels associated with it. The receiving node MAY discard the channel_announcement if the timestamp is far in the future. Otherwise the receiving node SHOULD queue the message for rebroadcasting, but MAY choose not to for messages longer than the minimum expected length.

Rationale

The timestamp field is used by nodes for pruning (either if it's too far in the future, or if it's been two weeks with no update), so it makes sense to have it be a UNIX timestamp (ie. seconds since UTC 1970-01-01). It can't be a hard requirement, however, given the possible case of two channel_updates within a second.

Initial Sync

Upon establishing a connection, the two endpoints negotiate whether to perform an initial sync by setting the initial_routing_sync flags in the init message. The endpoint SHOULD set the initial_routing_sync flag if it requires a full copy of the other endpoint's routing state. Upon receiving an init message with the initial_routing_sync flag set, the node sends channel_announcements, channel_updates and node_announcements for all known channels and nodes as if they were just received.

If the initial_routing_sync flag is not set, or initial sync was completed, then the node resumes normal operation: see the Rebroadcasting section for details.

Rebroadcasting

Nodes receiving a new channel_announcement or a channel_update or node_announcement with an updated timestamp SHOULD update their local view of the network's topology accordingly.

If, after applying the changes from the announcement, there are no channels associated with the announcing node, then the receiving node MAY purge the announcing node from the set of known nodes. Otherwise the receiving node updates the metadata and stores the signature associated with the announcement. This will later allow the receiving node to rebuild the announcement for its peers.

Once the announcement has been processed, it is added to a list of outgoing announcements for the processing node's peers, perhaps replacing older updates. This list will be flushed at regular intervals. This store-and-delayed-forward broadcast is called a staggered broadcast

Requirements

Each node SHOULD flush outgoing announcements once every 60 seconds, independently of the arrival times of announcements, resulting in a staggered announcement and deduplication of announcements.

Nodes MAY re-announce their channels regularly, however this is discouraged in order to keep the resource requirements low.

Nodes SHOULD send all channel_announcement messages followed by the latest node_announcement and channel_update messages upon connection establishment.

Rationale

Batching announcements forms a natural rate limit with low overhead.

The sending of all announcements on reconnection is naive, but simple, and allows bootstrapping for new nodes as well as updating for nodes that have been offline for some time.

HTLC Fees

The node creating channel_update SHOULD accept HTLCs that pay a fee equal or greater than:

fee_base_msat + ( amount_msat * fee_proportional_millionths / 1000000 )

The node creating channel_update SHOULD accept HTLCs that pay an older fee for some time after sending channel_update, to allow for propagation delay.

Pruning the Network View

Nodes SHOULD monitor the funding transactions in the blockchain to identify channels that are being closed. If the funding output of a channel is being spent, then the channel is to be considered closed and SHOULD be removed from the local network view.

Nodes MAY prune nodes added through node_announcement messages from their local view if the announced node no longer has any associated open channels. This is a direct result from the dependency of a node_announcement being preceded by a channel_announcement.

Recommendation on Pruning Stale Entries

Several scenarios may result in channels becoming unusable and the endpoints unable to send updates for these channels. For example, this happens in the case where both endpoints lose access to their private keys and cannot sign a channel_update nor close the channel on-chain. These channels are unlikely to be part of a computed route since they would be partitioned off from the rest of the network, however they would remain in the local network view and information on them would be forwarded to other nodes forever. For this reason, nodes MAY prune channels should the timestamp of the latest channel_update be older than 2 weeks (1209600 seconds). In addition nodes MAY ignore channels with a timestamp older than 2 weeks. Notice that this is a node policy and MUST NOT be enforced by peers, e.g., by closing channels when receiving outdated gossip messages.

Recommendations for Routing

When calculating a route for an HTLC, the cltv_expiry_delta and the fee both need to be considered: the cltv_expiry_delta contributes to the time that funds will be unavailable on worst-case failure. The tradeoff between these two is unclear, as it depends on the reliability of nodes.

If a route is computed by simply routing to the intended recipient, summing up the cltv_expiry_deltas, then nodes along the route may guess their position in the route. Knowing the CLTV of the HTLC and the surrounding topology with the cltv_expiry_deltas gives an attacker a way to guess the intended recipient. Therefore it is highly suggested to add a random offset to the CLTV that the intended recipient will receive, bumping all CLTVs along the route. In order to create a plausible offset the sender MAY start a limited random walk on the graph, starting from the intended recipient, sum the cltv_expiry_deltas, and then use the sum as the offset. This effectively creates a shadow route extension to the actual route, providing better protection against this kind of attack than simply picking a random offset.

Other more advanced considerations involve diversity of routes to avoid single points of failure and detection and channel balance of local channels.

Routing Example

Consider four nodes:

   B
  / \
 /   \
A     C
 \   /
  \ /
   D

Each advertises the following cltv_expiry_delta on its end of every channel:

1. A: 10 blocks
2. B: 20 blocks
3. C: 30 blocks
4. D: 40 blocks

C also uses amin_final_cltv_expiry of 9 (the default) when requesting payments.

Also, each node has a set fee scheme that it uses for each of its channels:

1. A: 100 base + 1000 millionths
2. B: 200 base + 2000 millionths
3. C: 300 base + 3000 millionths
4. D: 400 base + 4000 millionths

The network will see eight channel_update messages:

1. A->B: cltv_expiry_delta = 10, fee_base_msat = 100, fee_proportional_millionths = 1000
2. A->D: cltv_expiry_delta = 10, fee_base_msat = 100, fee_proportional_millionths = 1000
3. B->A: cltv_expiry_delta = 20, fee_base_msat = 200, fee_proportional_millionths = 2000
4. D->A: cltv_expiry_delta = 40, fee_base_msat = 400, fee_proportional_millionths = 4000
5. B->C: cltv_expiry_delta = 20, fee_base_msat = 200, fee_proportional_millionths = 2000
6. D->C: cltv_expiry_delta = 40, fee_base_msat = 400, fee_proportional_millionths = 4000
7. C->B: cltv_expiry_delta = 30, fee_base_msat = 300, fee_proportional_millionths = 3000
8. C->D: cltv_expiry_delta = 30, fee_base_msat = 300, fee_proportional_millionths = 3000

B->C. If B were to send 4,999,999 millisatoshi directly to C, it wouldn't charge itself a fee nor add its own cltv_expiry_delta, so it would use C's requested min_final_cltv_expiry of 9. Assume it also adds a "shadow route" to give an extra CLTV of 42. It could additionally add extra CLTV deltas at other hops, as these values are a minimum, but it doesn't here for simplicity:

• amount_msat: 4999999
• cltv_expiry: current-block-height + 9 + 42
• onion_routing_packet:
  • amt_to_forward = 4999999
  • outgoing_cltv_value = current-block-height + 9 + 42

A->B->C. If A were to send an 4,999,999 millisatoshi to C via B, it needs to pay B the fee it specified in the B->C channel_update, calculated as per HTLC Fees:

    fee_base_msat + ( amount_msat * fee_proportional_millionths / 1000000 )

200 + ( 4999999 * 2000 / 1000000 ) = 10199

Similarly, it would need to add the cltv_expiry from B->C's channel_update (20), plus C's requested min_final_cltv_expiry (9), plus 42 for the "shadow route". Thus the update_add_htlc message from A to B would be:

• amount_msat: 5010198
• cltv_expiry: current-block-height + 20 + 9 + 42
• onion_routing_packet:
  • amt_to_forward = 4999999
  • outgoing_cltv_value = current-block-height + 9 + 42

The update_add_htlc from B to C would be the same as the B->C direct payment above.

A->D->C. Finally, if for some reason A chose the more expensive route via D, it would send the following update_add_htlc to D:

• amount_msat: 5020398
• cltv_expiry: current-block-height + 40 + 9 + 42
• onion_routing_packet:
  • amt_to_forward = 4999999
  • outgoing_cltv_value = current-block-height + 9 + 42

And the update_add_htlc from D to C would be the same as the B->C direct payment again.

References

This work is licensed under a Creative Commons Attribution 4.0 International License.