Peer-to-Peer and Overlay Networks

Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Peer-to-Peer and Overlay Networks
The eMule Protocol1
Ricardo Lopes Pereira
ricardo.pereira@inesc-id.pt
IST
17-9-2014
1
Images from “The eMule Protocol Specification”, Yoram Kulbak and
Danny Bickson
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
1
Characteristics
High level
Granularity
2
Protocol
Server connection
File Search
File download
3
Algorithms
Queue management
Part selection
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
High level
Granularity
eMule
eMule is the most popular implementation of the eDonkey
2000 protocol, which includes a few extensions
The network contains a set of independent servers
There isn’t a single P2P network but a set of them, which
may intercept
Clients use both TCP and UDP according to the task being
executed
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
High level
Granularity
eMule
Servers:
Are independent, do not communicate among themselves
Are the network entry point
Supply clients with a list of other servers. This list is defined
by the server operators
Make available a file search service
Supply information about peers connected to the same server
sharing the same file
Make possible communicating with peers behind firewalls
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
High level
Granularity
eMule
Clients:
Share files among themselves
Maintain a connection to one server using TCP
May query other servers using UDP
Transfer files using TCP
May query other peers using UDP
Share information about servers, peers and files among
themselves
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
High level
Granularity
Network example
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
High level
Granularity
File transmission
Files:
Are divided into parts (9,28MB)
The sharing unit is the full part
Peers share file parts, even if they don’t yet have the full file
Parts are divided into chunks (180KB)
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
TCP connection to server and back!
Clients connect to a single
server, using TCP
Login states GUID, TCP
listening port, protocol version,
user nick ...
Server attempts to establish
back a new connection, in order
to test peer’s ability to receive
connections
If successful, server responds
with an High ID (peer IP)
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Clients unable to receive connections
Server responds with low
id.
Server message contains
warning text, stating
limitations
Servers present two limits,
(soft and hard) for the
number of supported
clients. After reaching the
soft limit, no more low ID
peers are accepted
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Session start
Client supplies list of
shared files, which it may
have to update during the
session
Client requests list of
alternative servers
Server provides status
information (number of
clients and servers)
Server sends welcome text
message
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Session start
Server supplies list of
servers
Server states name,
version and comments
Client requests peers
sharing the files it wants
to download
Server supplies partial list
of the peers it knows
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
File description
For each of the shared files, the client states:
ID
Name
Size
Type (audio, video, images, ...) (optional)
Length (opcional)
Bitrate (opcional)
Codec (opcional)
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Other servers - UDP server state
Clients communicate with
other servers in order to learn
their state/existence. UDP
messages limited to:
10 mgs/s in total
1 msg each 5s per server
Servers which do not respond
x times in a row are discarded.
Reply returns random value
sent in the request
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
File search
Client sends search
request
Server sends reply
After finding the intended
file, user starts download.
Client requests sources
Server sends stats
(number of clients and
files)
Server sends set of
sources
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
File search
When performing a search, the user may state:
Search string (with logical operators). Sent using prefix
notation
File type (optional)
Minimum and maximum file size (optional)
Availability (minimum number of known sources) (optional)
File extension (optional)
When it receives a set of sources, the client adds them to its list.
Peers are contacted in the order they are learned
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
UDP Enhanced file search
Client may extend the file search to several servers, using UDP
May search for files or sources
Sent simultaneously to several servers
Server only answers if it has data to send (UDP)
File search decided by user
Source search automatic when known sources are inferior to
100
No more than 1 msg/s
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Connecting to another peer
Client identifies itself and
indicates to which server
it is connected
eMule extension indicates
support for: UDP, secure
ID, peer exchange,
compression
Only one connection
between two peers
Connection closed after
40s of inactivity
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Connecting to low ID peers
A high ID peer may, using the
server, request a low ID peer
to connect back. This allows
NAT and firewalls to be
circumvented.
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Verifying a file existence
Client which started
connection indicates the
file it wants
Other responds whether it
has the file and which
parts it has
May request list of known
sources (peer exchange)
Files are identified using
16B file hash (MD4)
Peer may request root
hash (part hashes)
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Download request
After confirming existence
of file, peer requests
download
Uploading peers assigns
score to request
Request queued
If it’s not its turn to be
served, it will have to wait
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Download start
Download may start when
request reaches the top
positions of the queue
If peer isn’t connected,
uploader will connect
back
Client pipelines requests
(up to 3 outstanding
chunks)
If downloader already has
the file, it will cancel the
request
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
Transmitting file parts
Sending part messages
carry 5 to 15KB
Messages may be
compressed
Requests to a peer are all
for the same part
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Server connection
File Search
File download
UDP Periodic Download request
Sent every 20 minutes
Reply may be queue
position
Reply may be queue full
Reply may be file not
found
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Queue management
Part selection
Credit system
Secure identification
Client uses a random 16B value as ID. Each has a RSA
private/public key. Public keys are exchanged on the first
connection between two peers. Secure signatures allow peers to
trust each others and remember past file exchanges. Credits are
accounted individually between each pair of peers
Credit
Value from 1 to 10, calculated as the minimum of (units of MB):
uploaded total∗2
downloaded total
√
uploaded total + 2
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Queue management
Part selection
Upload queue
Number of upload slots limited to ensure a minimum of
2,4KB/s/slot
Slots assigned to the top positions of the queue
Clients sorted by score
score = (rating ∗ queued time seconds)/100
rating = initial rating ∗ credit ∗ file priority
initial rating is 0 for banned users
initial rating is ∞ for friends
initial rating is usually 100
initial rating is 200 during the first 15 minutes after being
assigned an upload slot
file priority is user defined (from 0.3 to 1.8)
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Queue management
Part selection
Part selection
The requested part is chosen according to the following criteria
(decreasing importance):
Part rarity (discrete: very rare, rare and common)
Parts which allow previewing the file (first and last)
Parts which aren’t being downloaded from another source
Parts whose download is closer to terminate
After downloading a part, its checksum is verified. Should it fail, the
peer will attempt to replace chunks sequentially until the checksum
becomes valid. Chucksum is a SHA1 hash over each file part. It is
called root hash
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Characteristics
Protocol
Algorithms
Queue management
Part selection
Question
Any doubts?
Ricardo Lopes Pereira
Peer-to-Peer and Overlay Networks
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising