A  BitTorrent Private Tracker Client
"Accounting Error Message" Protocol Alert

Abstract
Private BitTorrent tracker websites have to obsess over what clients they choose to connect with.

One of the primary reasons those that run private BitTorrent protocol tracker websites must obsess over what clients they choose to support is this : so many BT protocol clients don't correctly provide proper upload or download information to the tracker.

Without proper accounting data from the client application proper upload and download ratios can be provided to their (paying) users.

A protocol fix to this unpleasant situation is long overdue.

Scope
This protocol modification is not designed to be used with public trackers -- nor should it ever be used with public trackers, as public trackers are open to anyone to use and only engage in limited accounting of the activity of each Torrent. Public trackers most importantly do not store any user specific upload or download accounting data.

Public vs Private FAQ
Public torrent hosting sites such as The Pirate Bay allow users to search and download from their collection of torrent files. Users can typically also upload torrent files for content they wish to distribute. Often, these sites also run BitTorrent trackers for their hosted torrent files, but these two functions are not mutually dependent : a torrent file could be hosted on one site and tracked by another, unrelated site.

Private host/tracker sites such as Demonoid, TheBox.bz, etc ... operate like public ones except that they restrict access to registered users and keep track of the amount of data each user uploads and downloads, in an attempt to reduce leeching.

Both private and public trackers provide a way to browse the files (in list form) contained on the site. This list can often be sorted with respect to several criteria, being relevance (seeders-leechers ratio) one of the most popular and useful (due to the way the protocol behaves, the download bandwidth achievable is very sensitive to this value).
The leech problem
A BitTorrent user may often choose to leave the swarm as soon as they have a complete copy of the file they are downloading, freeing up their outbound bandwidth for other uses. If enough users follow this pattern, torrent swarms gradually die out, meaning a lower possibility of obtaining older torrents (see content unavailability above). Some BitTorrent websites have attempted to address this by recording each user's download and upload ratio for all or just the user to see, as well as the provision of access to newer torrent files to people with better ratios. Users who have low upload ratios may see slower download speeds until they upload more.

This abstraction prevents (statistical) leeching, since after a while they become unable to download at even a fraction of the theoretical bandwidth of their connection. Some trackers exempt dial-up users from this policy, because their uploading capabilities are limited. The BitTorrent protocol also attempts to minimize the damages of leeches by using only a portion of their bandwidth for one-directional trades and using the majority for two-directional trades that tend to help the swarm as a whole.

The cheater problem
There are "cheating" clients like BitThief which claim to be able to download without uploading. Such exploitation negatively affects the cooperative nature of the BitTorrent protocol, although it might prove useful for people in countries where uploading copyrighted material is illegal, but downloading is not.

Some countries, such as South Africa, have relatively high bandwidth prices, on average costing around 10 USD a gigabyte.

The undersea cables which link South Africa (and many developed nations) to the internet can carry only a low amount of bandwidth compared to what is required. This issue has been addressed in South Africa's case with the new SEACOM undersea cable which promises to bring cheaper bandwidth to Africa.

In bandwidth constrained countries, torrent users tend to minimize sharing.

Examples of forbidden clients on private trackers

Some Commonly Banned Clients (not authoritative)


The fix
The protocol fix for this annoying private tracker problem should ideally take up as little bandwidth as possible and be agnostic to (IP4 or IP6) as well as agnostic to (TCP or UDP).

This fix should apply to the granular level of individual torrents at a host. It is my understanding that BitTorrent clients make global announcements to their tracker hosts as per their global upload or download ratios, so there there must be some support for local torrent vs local client misbehavior.

The message protocol fix should be a one way and one way only message. However, obliging the client to respond to the tracker with a lower complexity reply message must also be considered. Reply messages should not be part of the protocol, but a reply message has been created here if clients or trackers want to implement it.

Data structures the tracker should maintain
Structure Name
 Recommended Data Type
 Use



User_max_up_speed_kb
 Unsigned Int, kilobytes
 Track user upload speed
User_max_down_speed_kb
 Unsigned Int, kilobytes Track user download speed



User_24h_max_up_kb
 Unsigned Int, kilobytes Track 24h MB Traffic Uploads (volitile)
User_24h_max_down_kb Unsigned Int, kilobytes Track 24h MB Traffic Downloads (volitile)



User_maximum_24h_traffic
 Unsigned Int, kilobytes Maximum possible observed traffic



User_24h_traffic_STDEV_up Unsigned Byte,
Standard Deviation Units
 Should be within 0.10 to 0.05 strictly
based on upload speeds. "0.XX"
User_24h_traffic_STDEV_down Unsigned Byte,
Standard Deviation Units		 Should be within 0.10 to 0.05 strictly
based on upload speeds. "0.XX"



User_update_interval
 Already existant
 Hourly updates a good idea
User_last_updated
 Signed Long, Unix Time
 64 bit POSTIX Time, NOT BCD time!



User_record_expires
 Unsigned Byte, hours
 Delete record if user inactive more than  ~7 days

Notes

Tracker Response

Currently the tracker responds to the client with "text/plain" document consisting of a bencoded dictionary with the following important keys (all non-applicable keys will be ignored, only the applicable ones for the protocol described here) :

Tracker 'scrape' Convention (an alternate delivery method)

By convention most trackers support another form of request, which queries the state of a given torrent (or all torrents) that the tracker is managing. This is referred to as the "scrape page" because it automates the otherwise tedious process of "screen scraping" the tracker's stats page.

The scrape URL is also a HTTP GET method, similar to the one described above. However the base URL is different. To derive the scrape URL use the following steps: Begin with the announce URL. Find the last '/' in it. If the text immediately following that '/' isn't 'announce' it will be taken as a sign that that tracker doesn't support the scrape convention. If it does, substitute 'scrape' for 'announce' to find the scrape page.

Unofficial extensions to scrape that could be used for this protocol

Below are the response keys are being unofficially used. Since they are unofficial, they are all optional.


Suggested data structures, Private Tracker Accounting Error Message

Data structure
 Typical output
 Bytes
Total
version : unsigned byte; 00x for version 01, always Binary not ASCII
 1
 1
message-number : unsigned int; (32 bits)
 Message number should always go up in sequence by some fixed rule (+3 for one private tracker vs +11 for another). The message number must restart at zero after each tracker reboot, after that it is up to the tracker to determine its own sequence for this accounting message.
4
 5
force-reply : char[1]; "Y" forces immediate reply (6s), "R" forces prognostic reply (67s), "N" forces no reply 1
 6
file-or-client : char[1];
 "L" ('local' file statistical error) or "G" ('global' client statistical error)
 1
 7
mode-accounting-failure : char[1]; "U" (upload) or "D" (download); or "S" (global client statistics)
 1
 8
affected-object-hash : unsigned byte[12]; The {first (for uploads) or last (for downloads)} 12 bytes of the SHA-1 hash of the affected shared object. Always Binary not ASCII. Must be XORed over "101010101..." (uploads) or "11001100..." (downloads).

Sending the 'cleartext' of the file SHA-1 hash would decrease the privacy to the protocol.

ALSO
For global client failures this should be filled with binary zeros.
 12
 20
message-crc16-CCITT : short unsigned int;
 Mandatory. Two bytes. Applies to the whole message. This field is necessary for the client program to make a reply.
CRC-16-CCITT : x16 + x12 + x5 + 1		 2
 22

The accounting error messages must be compact and meaningful as well as secure -- and all at the same time.

This protocol could greatly help in the debugging of BT clients that are designed to work with private trackers.


Messages
REPLY MESSAGE ONLY :  All of the remaining messages in the protocol take the form of <length prefix><message ID><payload>. The length prefix is a four byte big-endian value. The message ID is a single decimal byte. The payload is message dependent. It would be ideal for the reply protocol could be worked into this message area, but for now this work will have to be postponed pending more research.

Suggested parameters for the reply message should be {TBA}.

Suggested data structures, Private Tracker Accounting Error Message Reply

Data structure
 Typical output
 Bytes
 Total
version : unsigned byte; 0000x for version 0001, always Binary not ASCII. Mandatory.
 1
 1
message-processing-status : unsigned byte;
 01x = success : no syntax errors, no checksum errors;
05x = success : syntax errors, no checksum errors;
09x = failure : syntax and checksum errors ... did you send this?
 1
 2
reply :  unsigned int;
 Mandatory. Message number should always go up or down in sequence by some fixed rule unique to the private tracker to avoid protocol attacks. By only sending back the CRC of the message number, acknowledgment is possible and some possible other measures could be taken by either program or the user or both.

Replies should work like this :

Essentially a 32 bit checksum of the accounting error message should be computed by the client and transmitted back to the tracker. The answerback should be computed and stored by the tracker when the accounting error message is issued.

Syntax ::
reply = crc32K(private-tracker-error-message + md5(private-tracker-error-message))
"+" designates binary domain bit string concatenation

This checksum should not include the CRC16 transmitted to the client in the original accounting error message, as the CRC16 is only there for adverse network conditions. The goal is to eliminate the CRC16 as a source of spoofing or attack.

CRC-32-Koopan (aka crc32K) should be used here :
x32 + x30 + x29 + x28 + x26 + x20 + x19 + x17 + x16 + x15 + x11 + x10 + x7 + x6 + x4 + x2 + x + 1  		4
 5
message-crc8-CCITT :  unsigned byte;
 Mandatory. One byte. Applies to the whole message. This should only be paid attention to if the user is communicating a low bit rates -- as with dialup modems.
CRC-8-CCITT : x8 + x2 + x + 1

If this checksum is deemed unworkable, CRC-16-CCITT : x16 + x12 + x5 + 1 should be used as a replacement. Asymmetry of checksums here is a protection mechanism to make spoofing more difficult. Hashsums are only being avoided to limit bandwidth.
 1
 6

As long as any BT client can be emulated by another client, this accounting message will be needed.

As a matter of industrial practice, the reply back should be in UDP to aviod any "man-in-the-middle" attacks common with TCP. Sending this message twice (1 x UPD, 1 x TCP) may provide some nominal network diagnostic utility. The UDP message should be sent first, as sending the TCP version first may permit the UDP version to be filtered out at the router level.



References

General Topic
BitTorrent protocol
BitTorrent Protocol standards
Technologies used by the protocol








Created by : Max Power // Document Created : 02 February 2010 // Last modified : 15  August 2010