A  BitTorrent Private Tracker Client
"Accounting Error Message" Protocol Alert

Abstract
Private BitTorrent tracker websites have to obsess over what clients they choose to connect with.

One of the primary reasons those that run private BitTorrent protocol tracker websites must obsess over what clients they choose to support is this : so many BT protocol clients don't correctly provide proper upload or download information to the tracker.

Without proper accounting data from the client application proper upload and download ratios can be provided to their (paying) users.

A protocol fix to this unpleasant situation is long overdue.

Scope
This protocol modification is not designed to be used with public trackers -- nor should it ever be used with public trackers, as public trackers are open to anyone to use and only engage in limited accounting of the activity of each Torrent. Public trackers most importantly do not store any user specific upload or download accounting data.

Public vs Private FAQ
Public torrent hosting sites such as The Pirate Bay allow users to search and download from their collection of torrent files. Users can typically also upload torrent files for content they wish to distribute. Often, these sites also run BitTorrent trackers for their hosted torrent files, but these two functions are not mutually dependent : a torrent file could be hosted on one site and tracked by another, unrelated site.

Private host/tracker sites such as Demonoid, TheBox.bz, etc ... operate like public ones except that they restrict access to registered users and keep track of the amount of data each user uploads and downloads, in an attempt to reduce leeching.

Both private and public trackers provide a way to browse the files (in list form) contained on the site. This list can often be sorted with respect to several criteria, being relevance (seeders-leechers ratio) one of the most popular and useful (due to the way the protocol behaves, the download bandwidth achievable is very sensitive to this value).
The leech problem
A BitTorrent user may often choose to leave the swarm as soon as they have a complete copy of the file they are downloading, freeing up their outbound bandwidth for other uses. If enough users follow this pattern, torrent swarms gradually die out, meaning a lower possibility of obtaining older torrents (see content unavailability above). Some BitTorrent websites have attempted to address this by recording each user's download and upload ratio for all or just the user to see, as well as the provision of access to newer torrent files to people with better ratios. Users who have low upload ratios may see slower download speeds until they upload more.

This abstraction prevents (statistical) leeching, since after a while they become unable to download at even a fraction of the theoretical bandwidth of their connection. Some trackers exempt dial-up users from this policy, because their uploading capabilities are limited. The BitTorrent protocol also attempts to minimize the damages of leeches by using only a portion of their bandwidth for one-directional trades and using the majority for two-directional trades that tend to help the swarm as a whole.

The cheater problem
There are "cheating" clients like BitThief which claim to be able to download without uploading. Such exploitation negatively affects the cooperative nature of the BitTorrent protocol, although it might prove useful for people in countries where uploading copyrighted material is illegal, but downloading is not.

Some countries, such as South Africa, have relatively high bandwidth prices, on average costing around 10 USD a gigabyte.

The undersea cables which link South Africa (and many developed nations) to the internet can carry only a low amount of bandwidth compared to what is required. This issue has been addressed in South Africa's case with the new SEACOM undersea cable which promises to bring cheaper bandwidth to Africa.

In bandwidth constrained countries, torrent users tend to minimize sharing.

Examples of forbidden clients on private trackers

Some Commonly Banned Clients


The fix
The protocol fix for this annoying private tracker problem should ideally take up as little bandwidth as possible and be agnostic to (IP4 or IP6) as well as agnostic to (TCP or UDP).

This fix should apply to the granular level of individual torrents at a host. It is my understanding that BitTorrent clients make global announcements to their tracker hosts as per their global upload or download ratios, so there there must be some support for local torrent vs local client misbehavior.

The message protocol fix should be a one way and one way only message. However, obliging the client to respond to the tracker with a lower complexity reply message must also be considered. Reply messages should not be part of the protocol, but a reply message has been created here if clients or trackers want to implement it.

Data structures the tracker should maintain
Structure Name
 Recommended Data Type
 Use
User_max_up_speed
 Unsigned Short, kilobytes
 Track user upload speed
User_max_down_speed
 Unsigned Short, kilobytes Track user download speed
User_24h_max_up_kilobytes
 Unsigned Int, kilobytes Track 24h MB Traffic Uploads
User_24h_max_down_kilobytes Unsigned Int, kilobytes Track 24h MB Traffic Uploads
User_maximum_24h_traffic
 Unsigned Int, kilobytes Maximum possible observed traffic
User_24h_traffic_STDEV_up Unsigned Byte,
Standard Deviation Units
 Should be within 0.15 to 0.05 strictly
based on upload speeds. "0.XX"
User_24h_traffic_STDEV_down Unsigned Byte,
Standard Deviation Units		 Should be within 0.15 to 0.05 strictly
based on upload speeds. "0.XX"
Update_Interval
 Already existant
 Hourly updates a good idea
Last_Updated
 Signed Long, Unix Time
 64 bit POSTIX Time
Record_Expires
 Unsigned Byte, hours
 every 10.6 days, delete record if inactive

Notes

Tracker Response

Currently the tracker responds to the client with "text/plain" document consisting of a bencoded dictionary with the following important keys (all non-applicable keys will be ignored, only the applicable ones for the protocol described here) :

Tracker 'scrape' Convention (an alternate delivery method)

By convention most trackers support another form of request, which queries the state of a given torrent (or all torrents) that the tracker is managing. This is referred to as the "scrape page" because it automates the otherwise tedious process of "screen scraping" the tracker's stats page.

The scrape URL is also a HTTP GET method, similar to the one described above. However the base URL is different. To derive the scrape URL use the following steps: Begin with the announce URL. Find the last '/' in it. If the text immediately following that '/' isn't 'announce' it will be taken as a sign that that tracker doesn't support the scrape convention. If it does, substitute 'scrape' for 'announce' to find the scrape page.

Unofficial extensions to scrape that could be used for this protocol

Below are the response keys are being unofficially used. Since they are unofficial, they are all optional.


Suggested data structures, Private Tracker Accounting Error Message

Data structure
 Typical output
 Byte Cost
 Total Bytes
version : unsigned byte; 0000x for version 0001, always Binary not ASCII
 1
 1
message-number : unsigned int; (32 bits)
 Message number should always go up in sequence by some fixed rule (+3 for one private tracker vs +11 for another). The message number must restart at zero after each tracker reboot, after that it is up to the tracker to determine its own sequence for this accounting message.
4
 5

file-or-client : char[2];		 "LL" ('local' file statistical error); "LR" forces reply message.
or
"GG" ('global' client statistical error); "GR" forces reply message.
 2
 7
mode-accounting-failure : char[2]; "UU" (upload) or "DD" (download) or "ST" (global client statistics)
 2
 9
affected-object-hash : unsigned byte[12]; The first (for uploads) or last (for downloads) 12 bytes of the SHA-1 hash of the affected file. Always Binary not ASCII. Must be XORed over "101010101..." (uploads) or "11001100..." (downloads). Sending the file SHA-1 hash would decrease the privacy to the protocol.
or
For global client failures, the Trim[(SHA-1(peer_id)), 12] hash of the client's peer_id, of 20 characters.
 12
 21
message-crc16-CCITT : short unsigned int;
 Mandatory. Two bytes. Applies to the whole message. This field is necessary for the client program to make a reply.
CRC-16-CCITT : x16 + x12 + x5 + 1		 2
 23

Sample messages, "/" separates data fields and is not part of the protocol -- for your visual amusement, all representations are in the hex domain unless otherwise indicated by bold (for decimal numbers) or underline (for ASCII chars).
As you can see these error messages are compact and meaningful and secure at the same time.

This protocol could greatly help in the debugging of BT clients that are designed to work with private trackers.


Messages
REPLY MESSAGE ONLY :  All of the remaining messages in the protocol take the form of <length prefix><message ID><payload>. The length prefix is a four byte big-endian value. The message ID is a single decimal byte. The payload is message dependent. It would be ideal for the reply protocol could be worked into this message area, but for now this work will have to be postponed pending more research.

Suggested parameters for the reply message should be {TBA}.

Suggested data structures, Private Tracker Accounting Error Message Reply

Data structure
 Typical output
 Total
Byte Cost
version : unsigned byte; 0000x for version 0001, always Binary not ASCII
 1
answerback :  unsigned int;
 Message number should always go up or down in sequence by some fixed rule unique to the private tracker to avoid protocol attacks. By only sending back the CRC of the message number, acknowledgment is possible and some possible other measures could be taken by either program or the user or both.

Replies should work like this:

answerback = crc32K(message-number + message-crc16-CCITT), essentially a 32 bit checksum of a total string of 48 bits the "+" concatenates not adds. The answerback should be computed and stored by the tracker when the accounting error message is issued.

CRC-32-Koopan should be used here :
x32 + x30 + x29 + x28 + x26 + x20 + x19 + x17 + x16 + x15 + x11 + x10 + x7 + x6 + x4 + x2 + x + 1  		4
message-crc8-CCITT :  unsigned byte;
 Mandatory. One byte. Applies to the whole message. This should only be paid attention to if the user is communicating a low bit rates -- as with dialup modems.
CRC-8-CCITT : x8 + x2 + x + 1		 5

Sample reply messages
As long as any BT client can be emulated by another client, this accounting message will be needed.


References

General Topic
BitTorrent protocol
Technologies used by the protocol

Created by :
Max Power

Document Created :
02 February 2010

Last modified :
11  February 2010