The Emscher
Protocol
Specification
version: 0.14
(Version-Byte
in Header: 1)
Date: 2008-05-22 14:00 UTC
Valid for: Emscher v0.6.3b
1.
Introduction
The
Emscher-Protocol is designed for anonymous file sharing. It supports
data-encryption and indirect data-transfer. The Emscher-Network is based on
Gnutella. Because of the indirect data-transfer, each Emscher-Client gets an
individual 21 Byte Emscher-Address.
2. How
it works
2.1
Structure
The
Emscher-Network has a Hub/Leaf structure. Every Client has to connect to a Hub
for becoming part of the Emscher-Network. After successful connecting the
Client gets an Emscher-Address
(red=Hub2Hub blue=Hub2Leaf)
2.2
Connecting
When
connecting, the Client sends an /E_HI-Signal to a Hub. The Hub answers with an /E_HELLO-Signal. The
next Signals are RSA-1024-Encrypted, because they contain the AES-256-Key for
data-transfer and the new Emscher-Address (when Client is a Leaf). Then the Client sends a
RSA-1024-Encrypted /E_REGISTER to the Hub for registering there. If the Client
is a Hub, it adds to that signal its own Emscher-Address. When the Hub allows
the connection (e.g. no rejection, Hub not full) the Hub sends an /E_WELCOME
Signal with the AES-256-Key and the Emscher-Address. Otherwise the Hub sends an
/E_DENIED-Signal. All signals following these ones are AES-256 encrypted.
2.3
Disconnecting
For
disconnecting the Client sends an /E_BYE-Signal to the Hub connected to.
2.4
Emscher-Data Packages
2.4.1
Parts of a Data Package
An Emscher
Package is made of three parts:
|
Part |
Position(Byte) |
|
“ENP”
-> Emscher-Network Protocol (Beginning
of every signal) |
0-3 |
|
Header |
4-68 |
|
Data(depends
on header) |
> 68 |
Every
Emscher-Package is AES (Rijndael) encrypted.
Hint: The Size
of Every encrypted Emscher-Package must not be greater than 1450 Bytes!
2.4.2
The Header
The header
consists of these parts:
Part
Pos(Byte) Pos(Hex)
--------------------------------------------
Version
0 00
Data
Offset
1 01
ID
2-21 02-15
HTL
22 16
Checksum
23-26 17-1A
Data
Length
27-30 1B-1E
Source
Address
31-51 1F-33
Destination
Address 52-72 34-48
Reserved
73-76 49-4C
Description:
- Version
(1 Byte): Protocol version
- Data
Offset (1 Byte): The position of the first data byte
- ID (20
Byte): ID of the package
- HTL (Hops
to Live)(1 Byte): Is decremented by every Client redirecting the signal. If it
is zero (0), the signal will be destroyed.
- Checksum
(4 Byte): CRC32 checksum of the data
- Data
Length(4 Byte): Length of the data beginning on the data offset
- Source
Address (21 Byte): The Clients own Emscher Address
-
Destination Address (21 Byte): Emscher Address of the recipient
-
Reserved(4 Byte): Regardless/For later use
2.4.3
Emscher-Addresses
Every
Client has its own, individual Emscher-Address. It is used to identify a Client
in the network. Emscher-Addresses are 21-Byte long and consists of a 20-Byte
“Local Network ID (LNID)” (Every Hub and its Leaves are forming a local
network) and a 1-Byte “Client ID (CID)”. When a Hub starts, it generates a
20-Byte LNID(see 2.4.3.2); the last Byte is 0x00. Every Hub can feed 253 Leaves
(CID 0x01 until 0xFE).
2.4.3.1
Special Addresses
There are
three types of Special Emscher-Addresses: The local and global
Broadcast-Address and the Zero-Address. The local Broadcast-Address ends with
0xFF; the global Broadcast-Address consists of 20x 0xFF. They are used to send
a signal to all clients it can reach in the local or global network (e.g. file
search). Every Client starting in Leaf-Mode has as initial Address the
Zero-Address (20x 0xFF). When connecting to a hub, it gets a new
Emscher-Address from the Hub. These addresses can neither be a source address
nor a client address!
2.4.3.2 How to generate an LNID
To generate
a 20 Byte LNID the following data are used: Date/Time, Current value of the
high-performance counter, high-performance-counter-frequency
They will
be stored in this struct:
TIDData =
packed record
thetime :
TDateTime;
qpc : int64;
qpf : int64;
end;
Now a 20
Byte SHA1 sum of the TIDData struct will be calculated. This SHA1 sum is the
new LNID.
2.4.4
Sending & Routing
Before
sending a package, the Client generates an ID and the CRC32-Checksum of the
data. Now the Client uses its Emscher-Address as source address and the targets
address as destination address. If the Client is a Leaf and the source address
not equal to its address, the hub will destroy the package. A package will also
be destroyed, when HTL is 0. After that the package will be redirected randomly
through the network through several hubs. If the package comes to a hub having
the same LNID as the target, it will be send to the target. If the package and
the hub have the same destination address, the hub will not send this package
again.
2.5
Searching
For searching
a file or for looking for peers, the Clint sends an /E_SEARCH-Signal allocated
to a broadcast address. Each search has a query and a query type. The type can
be “Q”, “QA” and “H”. “Q” and “QA” are standard queries with a small
difference: Lets take the query “A B C”. Q searches for “A or B or
C”, but QA for “A and B and C”. The third type, “H”, is for
searching a special file having the hash code in Query. It is used for looking
for peers in an Emscher ring. Does a Client has got an applicable file it sends
an /E_SEARCHRESPONSE to the search starter. This signal contains the ID of the
package with the search query and a list of results (FILE:HASH).
(Animated;
blue=/E_SEARCH red=/E_SEARCHRESPONSE;
H=Hub L=Leaf; Leaf with green circle-> Asker)
2.6
2.7
Emscher rings/files
2.7.1
What are Emscher rings?
Emscher
rings are comparable to the temporary networks in BitTorrent. Every Emscher
ring represents one file. In Emscher rings there are only “peers”, that means,
it does not matter if a peer is a hub or a leaf.
2.7.2
Looking for peers
For finding
peers the Client uses the search with type “H” (see 2.5) using the hash code of
the file as query
2.7.3
Bitfields
A bitfield
represents the parts of the shared file which the peer owns. Every Bit in the
bitfield represents one part. A part is 64KB(64 Blocks á 1KB) long.
2.7.4
Part requesting
To request
a part of the shared file, the peer sends an /E_PARTREQUEST-Signal to the owner
of the part. This signal contains the file hash and the number of the part. The
owner answers with an /E_PARTRESPONSE-Signal containing a success or failure
message (e.g. “DH” (Don’t have)).
2.7.5
Data transfer
After the
successful requesting of a part the asker will send an /E_ACKNOWLEDGE to
initiate the transfer. Then the owner will send an /E_BLOCK-Signal containing
the first 1KB of the part. Continuing the peer sends an /E_ACKNOWLEDGE-Signal
to the owner acknowledging the receiving. Then the owner sends an
/E_BLOCK-Signal again. This continues as long as there is data to send. If the
owner has send the last 1KB of the part and has got an acknowledge for that, it
will send an /E_FINISH-Signal to the peer signaling the finish of the data
transfer.
2.8 Hashing
|
What to hash |
Hash algorithm |
|
Files |
SHA-1 |
|
Parts |
MD5 |
|
The data of a package |
CRC32 |
2.9 Encryption
/E_HI and
/E_HELLO are not encrypted. /E_REGISTER, /E_WELCOME and /E_DENIED are
RSA-Encryped.
The rest is
AES(Rijndael) encrypted.
3. Signal reference
|
Signal |
Description |
|
/E_HI/<RSA-Key>/<Port>/<User-Agent>/<Mode>
[/<Leaves>/<MaxLeaves>] |
The first signal send to a hub for connecting to the network. It
contains the public RSA-Key of the client, its port, the clients
name and the mode (Hub(“1”) or Leaf(“0”)). |
|
/E_HELLO/<RSA-Key>/<User-Agent>/[/<Leaves>/
<MaxLeaves>] |
This signal is an answer to an /E_HI. For the parameters see E_HI. |
|
/E_REGISTER[/<Emscher-Address>] |
Sent after receiving an /E_HELLO-Signal to register at the hub,
Emscher-Address is included if the sender is a hub(Hub-EA) |
|
/E_WELCOME/<AES-Key>/<Emscher address>/<Leaves>/
<MaxLeaves> |
An answer to an /E_REGISTER-Signal after successful connecting. The
whole signal is RSA-Encrypted and contains the AES-Key used for transferring
data between the hub and the client, the new Emscher address for the client
and the count of leaves. |
|
/E_DENIED/FULL|REJECT|NOHUB |
An answer to an /E_REGISTER-Signal after connecting failue. This
signal is as well as the E_WELCOME-Signal RSA-Encrypted and contains the
reason for the failure |
|
/E_BYE |
Sent for disconnecting from a hub |
|
/E_PING |
This signal is sent for checking if a Hub is still alive |
|
/E_PONG |
Answer to an /E_PING |
|
/E_BLOCK/<HASH>/ <NUMBER>/<DATA> |
This signal is for data transfer between two peers, it contains the
file hash, the part number and the data |
|
/E_ACKNOWLEDGE/ <HASH>/<NUMBER> |
Acknowledge for an
/E_BLOCK-Signal |
|
/E_FINISH/<HASH>/<NUMBER> |
Sent past an acknowledge to signal the finish of the data transfer |
|
/E_SEARCH/Q|QA|H/<QUERY> |
A search query. This signal is used for searching files or looking for
peers. The second parameter is the search type(Query, Query-And,Hash) and the
third the query(search string or hash code) |
|
/E_SEARCHRESPONSE/<PID>/<User-Agent>/<LIST> |
Answer to a search query. This signal contains the package ID(package
containing the search query) , the User-Agent and the list of results: /FILENAME:SIZE:HASH:METADATA [/FILENAME:SIZE:HASH:METADATA[/…]] |
|
/E_PARTREQUEST/<HASH>/<NUMBER> |
Used for requesting a part in an Emscher ring. <HASH> is the
hash code of the shared file and number the number of the part. |
|
/E_PARTRESPONSE/<HASH>/<NUMBER>/OK|FULL|DH |
Response after an part request. Contains the file hash, the number of
the requested part and a success message. |
|
/E_GETBITFIELD/<HASH> |
Asking a Peer for the Bitfield of a file |
|
/E_NEWBITFIELD/<HASH>/<BITFIELD> |
A Peer sends to an asking Peer
its (new) Bitfield |
4. EmscherCaches
EmscherCaches
are used to get the IP-Addresses of hubs from the internet (Similar to the
Gnutella WebCaches).
4.1 How they work
The Clients
use the POST method to get data from the webcaches. The main parameter “action”
controls if a client wants for example to add itself or to get all hubs. Here
all possible values of “action”:
|
Value of action |
Further parameters |
Output message(s) |
Description |
|
add |
port
-> Port of client client
-> User-Agent of client |
“DONE” or
“ERROR/message” |
Adds the
client to the cache |
|
isdead |
ip ->
IP of hub |
“DONE” or
“ERROR/message” |
Removes
one hub from the cache |
|
getall |
|
“DONE/[IP/[IP/[…]]]END“ |
Returns
all listed hubs from the cache |
|
addcache |
url ->
url of cache |
“DONE” or
“ERROR/message” |
Adds one
cache to the cache |
|
cacheisdead |
url ->
url of cache |
“DONE” or
“ERROR/message” |
Deletes
one listed cache from the cache |
|
getallcaches |
|
“DONE·[Cache·[Cache·[…]]]END” (“·”
means Space!) |
Returns all
listed caches |
Hint: All
output messages begin with {EMSCHERCACHE_RESPONSE_BEGIN} and end with
{EMSCHERCACHE_RESPONSE_END}
4.2 Further ideas
5. Emscher links
...starts always
with „emscher://“ (like „http://“). They are used for locating a special file
in the network from a web page. There are three types of Emscher links:
|
emscher://file|<filehash>
|
Adds a
file to the download list |
|
emscher://hublist|<IP>[|<IP>]
etc |
Adds one
or more hubs |
|
emscher://cache|<CACHE> |
Adds a Webcache |