The Extension Protocol in BitTorrent is a mechanism used to extend and enhance the functionality of the BitTorrent download protocol. It allows clients to communicate by sending and responding to messages that are different from the standard protocol. These messages can implement new features such as Distributed Hash Table (DHT) exchange, Peer Exchange (PEX), and support for higher-level encryption.
The Extension Protocol plays a crucial role in the BitTorrent protocol as it enables clients to perform more efficient and reliable data exchanges, and provides better privacy and security. For example, DHT supports finding other peers connected to the network without a tracker, while PEX allows clients to communicate directly with other clients, making data exchanges more stable and reliable.
Protocol Introduction#
The purpose of this protocol is to provide a simple and lightweight way to extend the BitTorrent protocol. Supporting this protocol makes it easy to add new extension features without interfering with the standard BitTorrent protocol or clients that do not support the extension features you want to add. In other words, this protocol provides a flexible and scalable mechanism that allows developers to personalize and customize the BitTorrent protocol without affecting other users.
In the BitTorrent protocol, one byte from the reserved bytes is used to propagate itself to other supporting clients. The byte used for the extension protocol is the 20th byte counting from the right (starting from 0). Therefore, the expression (reserved_byte[5] & 0x10) can be used to check if the client supports extension message transmission. This expression means that the 5th byte of the reserved bytes is bitwise ANDed with 0x10, and if the result is not 0, it means that the client supports extension messages.
After establishing support for the protocol, the client should support one new message:
| name | id |
|---|---|
| extended | 20 |
This message is sent like other BitTorrent messages, with a 4-byte length prefix and a single-byte identifier (which is 20 in this case). At the beginning of the message payload, there is a single-byte message identifier. This identifier can refer to different extension messages, but only one ID is specified, which is 0. If the ID is 0, it means that the message is a handshake message, which will be described below. The general layout of extension messages is as follows (including the message header used by the BitTorrent protocol):
| size | description |
|---|---|
| uint32_t | Length prefix. Specifies the number of bytes for the entire message. (Big-endian 1) |
| uint8_t | BitTorrent message ID, = 20 |
| uint8_t | Extension message ID. 0 = handshake, >0 = handshake-specific extension message. |
Handshake Message#
The payload of the handshake message is a bencoded dictionary. All items in the dictionary are optional. The client should ignore any unknown names in the dictionary. All parts of the dictionary are case-sensitive. The following items are defined in the dictionary:
| name | description |
|---|---|
| m | A dictionary of supported extension messages that maps the name of each extension message to an extension message ID. The only requirement for these IDs is that no two extension messages can share the same ID. Setting the extension number to zero means that the extension is not supported/disabled. The client should ignore any extension names it does not recognize. The extension message ID is used to send the extension message to the peer sending this handshake. In other words, these IDs are local IDs specific to this particular peer. |
Here are some items that implementations may choose to support:
| name | description |
|---|---|
| p | The local TCP listening port. Allows each party to know the TCP port number of the other party. Note that the receiving party does not need to send this extension message as its port number is known. |
| v | The client name and version (as a UTF-8 string). This is a more reliable way to identify the client than relying on the peer ID encoding. |
| yourip | A string containing a compact representation of an IP address to indicate the IP address that this node sees you as. In other words, this is the external IP address of the receiving party (excluding the port). It can be an IPv4 (4 bytes) or IPv6 (16 bytes) address. |
| ipv6 | If this node has an IPv6 interface, this is a compact representation of that address (16 bytes). The client may prefer to connect via an IPv6 address. |
| ipv4 | If this node has an IPv4 interface, this is a compact representation of that address (4 bytes). The client may prefer to connect via an IPv4 address. |
| reqq | An integer representing the number of outstanding request messages that this client supports without dropping any messages. In libtorrent, the default value is 250. |
The handshake dictionary can also include additional handshake information, such as support for encrypted headers or any other possible content.
Here is an example content of a handshake message payload:
| Dictionary | |
|---|---|
| m | Dictionary / LT_metadata······1 / ut_pex··················2 |
| p | 6881 |
| v | “µTorrent 1.2” |
And in encoded form:
d1:md11:LT_metadatai1e6:µT_PEXi2ee1:pi6881e1:v13:\xc2\xb5Torrent 1.2e
To avoid accidental conflicts with extension names, one or two characters of code should be added before the extension name to identify the client that introduces the extension. This applies to both the names of extension messages and any other information placed in the top-level dictionary. Unless defined by this specification, all one-byte and two-byte identifiers are invalid.
This message should be immediately sent to any node that supports this extension protocol after the standard BitTorrent handshake. Handshake messages can be sent multiple times throughout the connection, and the sender should not disconnect. Implementations may choose to ignore subsequent handshake messages (or parts of them).
Subsequent handshake messages can be used to enable/disable extensions without restarting the connection. If a node wants to disable support for LT_metadata without affecting any other extensions, it should send this message: d11. As mentioned above, the value 0 is used to turn off the extension.
The extension ID must be stored for each node, as each node may have different IDs for the same extension.
This specification intentionally does not specify any extensions, such as node exchange or metadata exchange. The protocol is only a transport for actual extensions to the BitTorrent protocol, and the extensions named in the example above (such as p) are just examples of possible extensions.
Rational Basis#
The reason the ID for extension messages is defined during the handshake process instead of using a global message ID registry is to ensure that extension message names have uniqueness, which is easier than using a global registry. The convention is to use a two-letter prefix on the extension message name, which identifies the client that first implemented the extension message. For example, LT_metadata is implemented by libtorrent, so it has the LT prefix.
If the client supporting the extension can decide which numbers its received messages will have, then those numbers are constants within that client. In other words, they can be used in a switch statement. For the other end, storing an array of IDs expected for each message and using that array for lookup when sending extension messages is straightforward.
The reason for using a dictionary instead of an array (which would implicitly allocate index numbers for extensions) is that if a client wants to disable certain extensions, the ID numbers would change, and it would not be possible to use constants (thus, cannot be used in a switch). If the message ID directly mapped to the BitTorrent message ID, the extensions in the handshake could also be mapped to existing extensions with fixed message IDs.
The reason for using a single-byte identifier as the extension message identifier is to follow the single-byte message identifier in the BitTorrent specification. This is considered sufficient. It does not limit the total number of extensions, only the number of extensions used simultaneously.
The reason for using a single-byte identifier as the standardized handshake identifier is: 1) the mainline DHT uses a single-byte identifier; 2) bandwidth savings. The only advantage of longer messages is to make the protocol more readable to humans, but the BT protocol is not designed to be a human-readable protocol, so why bother.
Overall, the Extension Protocol is an important part of the BitTorrent protocol, providing users with more features and a better user experience.