Degree 3 Suffices: A Large-Scale
Overlay for P2P Networks ?
Marcin Bienkowski??1 , André Brinkmann? ? ?2 , and Miroslaw Korzeniowski†3
1
University of Wroclaw, Poland
University of Paderborn, Germany
Wroclaw University of Technology, Poland
2
3
Abstract. Most peer-to-peer (P2P) networks proposed until now have
either logarithmic degree and logarithmic dilation or constant degree and
logarithmic dilation. In the latter case (which is optimal up to constant
factors), the constant degree is achieved either in expectation or with
high probability. We propose the first overlay network, called SkewCCC,
with a maximum degree of 3 (minimum possible) and logarithmic dilation. Our approach can be viewed as a decentralized and distorted version
of a Cube Connected Cycles network. Additionally, basic network operations such as join and leave take logarithmic time and are very simple to
implement, which makes our construction viable in fields other than P2P
networks. A very good example is scatternet construction for Bluetooth
devices, in which case it is crucial to keep the degree at most 7.
1
Introduction
Peer-to-peer networks have become an established paradigm of distributed computing and data storage. One of the main issues tackled in this research area is
building an overlay network that provides a sparse set of connections for communication between all node pairs. The aim is to build the network in a way
that an underlying routing scheme is able to quickly reach any node from any
other, without maintaining a complete graph of connections. In this paper, we
investigate such networks suitable not only for peer-to-peer networks but also
for Bluetooth scatternet formation and one-hop radio networks.
An important property of the investigated networks is their scalability. We
introduce a scalable and dynamic network structure which we call SkewCCC.
The maximum in- and out-degree of a node inside the SkewCCC network is 3
and routing and lookup times are logarithmic in the current number of nodes.
Naturally, it is impossible to decrease the degree to 2 while preserving any network topology besides a ring. Our routing scheme is name-driven, i.e. packet
?
??
???
†
This work has been partially supported by EU Commission COST 295 Action “DYNAMO — Foundations and Algorithms for Dynamic Networks”.
Supported by MNiSW grant number N206 001 31/0436, 2006-2008.
Partially supported by the EU within the 6th Framework Programme under contract
IST-2005-034891 Hydra
Supported by MNiSW grant number PBZ/MNiSW/07/2006/46.
routes can be calculated based on their destination address without requiring
extensive routing tables [8].
The construction of the SkewCCC network is fully distributed and remains
very simple from the computational and communication perspective. This is
crucial when designing algorithms for weak devices like small embedded systems,
which are often found in sensor networks. To minimize the production costs such
devices have very limited computational power and have to reduce their energy
usage [6].
It is widely known in the area of P2P networks that if we use a ring-based
network such as Chord [12] as a basis for a distributed hash table (DHT), then
the load balance is not even, unless special algorithms are employed in order to
smoothen the load. In particular, if nodes choose random places for themselves,
the expected ratio of the highest loaded to the lowest loaded node is Ω(n log n).
Our design applied as a topology for a DHT is perfectly balanced, i.e. without
applying any additional load balancing schemes each node has an in- and outdegree independent of the network size.
Besides being used as an overlay sensor network or as a peer-to-peer network,
the proposed SkewCCC architecture can also be applied for underlay networks.
In particular, for the construction of Bluetooth scatternets and one-hop radio
networks, it is not sufficient to ensure that the (expected) network degree is
bounded by some constant, but it is absolutely necessary to keep the degree
smaller than 7.
2
Related Work
The current research in the area of P2P networks concentrates on providing
dynamic overlay networks with good properties (small diameter, small in- and
out-degree). The basic task of peer-to-peer applications, i.e. efficiently locating
the peer that stores a sought data item, is performed by means of consistent
hashing [7]. The main requirement of this technique is the ability to perform
a name-driven routing in the network; the parameters of this routing, like dilation
(max. path length) influences directly the performance of the whole system. In
this section, we compare our solution with existing dynamic networks. By n we
denote the number of nodes currently in the system.
The first proposed distributed hash table (DHT) solutions were ring topologies: Chord [12] and Chord-like structures: Tapestry [13] and Pastry [11]. By
introducing shortcuts inside the ring, these DHTs achieve good properties: their
dilation is O(log n) and also the runtime of joining and leaving the network is
logarithmic. On the other hand, each node has to keep Θ(log n) pointers to other
nodes and thus the out-degree is large.
SkipNets [5] and Skip-Graphs [1] as well their deterministic versions such as
Hyperrings [2] are based on a hierarchical structure. In this approach the network
is organized into levels, where the first level is just a ring of all nodes and higher
levels are built of independent rings being refinements of rings from previous
levels. The refinement continues until each node itself is a ring. The approach
yields good properties concerning storage of sorted or unevenly distributed data
but does not improve the degree/distance factor of Chord. In all mentioned
designs, the degree of each node is logarithmic (as there are logarithmic number
of levels and each node has degree two in each level) and logarithmic dilation
(one step has to be done in each ring).
The first overlay network with constant out-degree and logarithmic diameter
was Viceroy [9]. The network is based on the randomized approximation of a
butterfly network [8] that adds a constant number of outgoing links for long
range communications. Furthermore, join or leave operations inside Viceroy require only a constant number of local updates on expectation and a logarithmic
number with high probability. Nevertheless, the in-degree of at least one node
inside the network becomes with high probability Ω(log n/ log log n) if each node
has only a single choice for its address during the join process. The in-degree
can also be bounded with high probability if each node has Θ(log n) random
choices for its address during the join operation. However this leads to Θ(log2 n)
updates for each join operation.
The approach of using multiple choices is also used inside the Distance Halving DHT network [10]. The structure is based on a dynamic decomposition of
a continuous space into cells, which are assigned to nodes. The underlying deBruijn structure ensures a degree of d for each node and leads to a path length
of O(logd n) for key lookups.
Besides providing solutions for general networks, there has been some research on overlay networks which consider the special properties of Bluetooth
networks. The first scalable overlay for Bluetooth (a network of constant degree
and poly-logarithmic diameter) has been presented in [3]. The network is based
on a backbone that enables routing based on virtual labeling of nodes without
large routing tables or complicated path-discovery methods. The scheme is fully
distributed, but still poses high demands on the computational abilities of the
underlying devices.
In [4], we have presented an overlay topology with special support for Bluetooth networks which is based on Cube Connected Cycles networks [8]. The resulting network has constant in- and out-degree as well as a dilation of O(log n).
The main drawback of the approach is that the scheme is centralized (in particular, each node has to know the current number of nodes in the system) and
hence not scalable. Although the scheme we present in this paper is also based
on the CCC network, it is a big step forward as it is completely distributed and
self-balancing.
In Fig. 1, we summarize the parameters of different distributed solutions,
most of them holding with high probability.
3
SkewCCC
As we base our approach on the hypercube and the CCC (Cube Connected
Cycles) networks, we shortly review their construction. These networks were
extensively studied and have good properties concerning maintenance, diameter,
Network
out-degree in-degree
Chord-like networks
Θ(log n)
Θ(log n)
Viceroy (single choice)
O(1)
Ω( logloglogn n )
Viceroy (multiple choice)
O(1)
O(1)
Distance Halving (m.c.)
O(d)
O(d)
SkewCCC
3
3
dilation
join runtime
Θ(log n)
Θ(log n)
Θ(log n)
O(log n)
Θ(log n)
Ω(log2 n)
Θ(logd n) Ω(log n · logd n)
Θ(log n)
Θ(log n)
Fig. 1. Summary of old and new results
a)
101
111
001
011
100
000
b)
110
010
Fig. 2. a) 3-dimensional hypercube, b) 3-dimensional CCC
degree of nodes and routing speed. On the other hand, they are meant exclusively
for static networks. For a thorough introduction to these kinds of networks, we
refer the reader to [8].
In this paper, we use the following notation. By a string we always mean
a binary string, whose bits are numbered from 0. For any two strings a and b,
we write a v b to denote that a is a (not necessarily proper) prefix of b. We
denote an empty string by and use · to denote a concatenation of two strings;
⊕ denotes the bitwise xor operation. We also identify strings of fixed lengths
with binary numbers they represent.
Definition 1. The d-dimensional hypercube network has n = 2d nodes. Each
node is represented by a number 0 ≤ i < n. Two nodes i and j are connected if
and only if i ⊕ 2k = j for an integer 0 ≤ k < d.
A d-dimensional Cube Connected Cycles (CCC) network is essentially a
d-dimensional hypercube in which each node is replaced with a ring of length d
and each of its d connections is assigned to one of the ring nodes. This way the
degree of the network is reduced from d to 3, whereas almost all of the network
properties (e.g. diameter) are changed only slightly.
Definition 2. The d-dimensional CCC network has d · 2d nodes. Each node is
represented by a pair (i, j), where 0 ≤ i < d and 0 ≤ j < 2d . Each such node is
connected to three neighbors: two cycle ones with indices ((i ± 1) mod d, j) and
a hypercubic one (i, j ⊕ 2i ).
Examples of a 3-dimensional hypercube and CCC network are given in Fig. 2.
We present our network in two stages. First, we describe the network and its
properties. Second, we describe algorithms, which assure proper structure when
nodes join and leave the network.
3.1
Network Structure
When we compare the CCC network with the hypercube, we may think that the
d-dimensional hypercube is a skeleton for the d-dimensional CCC. In other words,
if we replace each node of the hypercube (to which we refer as a corner) by a cycle
of nodes, then the resulting network is a CCC. In the following description we
start from a description of such a skeleton for our network, called SkewHypercube,
then we show how to replace each of its corners by a ring of real network nodes,
finally creating a structure, which we call a SkewCCC network.
SkewHypercube In the following we describe a skeleton network called skewHypercube. Each node of this network will correspond to a group of real nodes.
To avoid ambiguity, we refer to a skeleton node as a corner.
First, we define the set of corners. Each corner i has an identifier, which is
a string si of length di . The number di is called the dimension of the corner.
We require that the set of corner identifiers C = {si } is prefix-free and complete.
Prefix-freeness means that for any two strings si , sj , neither si v sj nor sj v si .
Completeness means that for any infinite string s, there exists an identifier si ∈ C,
s.t. si v s. The description above implies that (i) a single corner with empty
name s = constitutes a correct set C and (ii) any correct set of corners C can be
obtained by multiple use of the following operation (starting from the set {}):
take a corner i and replace it with two corners j and k with identifiers sj = si · 0
and sk = si · 1.
Second, we define the set of edges in a SkewHypercube.
Definition 3. Two SkewHypercube corners i and j are connected iff
(i) there exists 0 ≤ ki < di , s.t. si ⊕ 2ki v sj or
(ii) there exists 0 ≤ kj < dj , s.t. sj ⊕ 2kj v si .
We note that if all identifiers of corners have the same length d, then our
SkewHypercube is just a regular d-dimensional hypercube. On the other hand,
the definition above allows the following situation to occur. It may happen that
a single corner s has identifier 0 (dimension 1) and there are 2k corners with
dimension k + 1 with identifiers starting with 1. This results in corner s having
the degree of 2k . We will explicitly forbid such situations in the construction of
our network and require that the dimensions of neighboring corners can differ
at most by 1. This ensures that each corner of dimension d has at most 2d
neighbors. An example of a SkewHypercube is presented in Fig. 3.
Identifiers To specify which nodes are stored in particular corners of the SkewHypercube, we have to give nodes unique identifiers. These identifiers are
infinite strings, chosen randomly upon joining the network, where each bit is
equiprobably 0 or 1. To avoid the burden of handling infinite strings, one may
follow the approach of SkipGraphs [1], i.e. a node chooses for an identifier a string
of a fixed length, and when two nodes with the same identifier meet, they choose
a)
101
b)
111
011
00
100
010
1101
1100
Fig. 3. a) SkewHypercube skeleton, b) SkewCCC, only core nodes are depicted
additional random bits until their identifiers differ on at least one bit. Moreover, it will follow from our construction that such an identifier conflict will be
detected right after a node joins the network. In practical applications, it is sufficient to choose identifiers of length 160 bits, as in such case the probability of
a conflict is overwhelmingly low.
The identifier of a node decides where the node should be placed in the
network and constitutes its address. Namely, a node with identifier x is stored
in a corner s, s.t. s v x. It remains to show how nodes are connected within
a corner and how neighbor relations between corners are represented by real
edges between nodes.
From Skeleton to SkewCCC As mentioned previously, a corner identified
by a string si contains a group of nodes whose identifiers have prefix si . Nodes
within a corner are managed in a centralized fashion.
As each skeleton node of degree d can have up to 2d neighbors, we require that
each corner has at least 2d nodes, called core ones. These nodes are connected in
a ring and each of them is responsible for a (potential) connection to a different
corner. For efficiency of routing, we demand that core nodes are sorted in the ring
in the same way as in the original CCC network. It means that a node in corner
s responsible for a connection to s ⊕ 2k+1 follows on the ring a node responsible
for a connection to s ⊕ 2k , whereas a node responsible for a connection to s ⊕ 20
follows the one responsible for a connection to s ⊕ 2d−1 .
It might happen that a corner contains not only its core nodes, but also has
to manage additional ones, which are called spare. When they join the corner,
we put them on the ring between core nodes and we balance the path lengths
between two consecutive core nodes. It means that if there are m spare nodes
in the corner of dimension d, then there are between bm/2dc and dm/2de spare
nodes between any two consecutive core nodes. Due to this construction, each
node in the network has degree at most 3. An example of such network is given
in Fig. 3.
3.2
Network Maintenance
In this section, we show how to maintain the shape of the network in a distributed
way in a dynamic setting, where nodes may join or leave the system. We start
with showing a name-driven routing scheme, in which it is sufficient to know
the destination identifier to reach the corresponding node in a greedy manner in
a logarithmic number of steps (in the total number of nodes in the network).
We introduce a constant ` ≥ 2, which is a parameter of our algorithms. The
runtime of our operations increase linearly with `, and the probability that the
network is not balanced (the dimensions of corners vary) decreases exponentially
with `.
Routing and Searching When we search for a node with name s, we want
to find a corner with a name being prefix of s. Routing is performed using the
bit-fixing algorithm in which we fix bits of s one by one starting from the lowest
bit. When we want to fix the k-th bit of s, and s differs from the current corner
name in this bit, we go to a node in the current corner which has a connection
to a corner differing exactly on this bit and traverse this connection. When we
reach a destination corner (whose name is a prefix of s), we forward the message
through all nodes in the corner. The message either reaches its destination or
reaches some node for the second time. If the latter happens, this node answers
the request with a negative (not found) answer.
Joining We assume that when a node joins a network, it knows (or can discover
and contact) another node which is already a member of the network. After
choosing an identifier, the joining node asks its contact to find this identifier
in the network. As the identifiers are supposed to be different, the result of
the search will be negative, but a node with the longest prefix matching the
identifier will be returned as a new contact. The new node joins (as a spare
node) the corner to which its new contact belongs. At this point, it is checked if
a split of the corner is necessary.
Splitting A corner s of dimension d could be allowed to split when there are
enough resources to form two corners of dimension d + 1, i.e. if there are at
least 2(d + 1) nodes with prefix s · 0 and with prefix s · 1. However, for efficiency
in a dynamic system, we split a corner when the number of nodes with both
prefixes exceeds 12 · ` · (d + 1), where ` ≥ 2 is a constant parameter described
above. Additionally, such a corner is allowed to split only if it has no neighbors
of dimension d − 1.
After splitting, we create two corners: s · 0 and s · 1 and assign nodes to
them according to their names. A connection is established between the two
new corners: a core node is responsible for this connection in each of them. Each
connection from s is assigned to a proper one of the two new corners. If the
connection has been to a corner of dimension d, both corners connect to the
neighboring corner and the latter has to assign an additional core node to serve
this connection. If the connection has been to two corners of dimension d + 1,
then s · 0 connects to the corner with a 0 on position d and s · 1 to the corner
with a 1 on position d.
Leaving When a node i wants to leave the network, it searches for a special
spare node j in its corner, tells j to take its place, and leaves. The special spare
node j is chosen so that after removing i and migrating j into its place, spare
nodes are distributed evenly in the corner. After i leaves and j migrates, a check
is performed if the number of nodes in the corner has decreased sufficiently to
call a merging operation.
Merging When the number of nodes in a corner s of dimension d drops below
2d, we have to merge s with its neighbor s0 = s ⊕ 2d−1 , i.e. with the one differing
from s on the last bit. Actually, we do it already when the number of nodes in
such a corner drops to (6` + 7) · d. First, we send a message to all neighboring
corners of dimension d + 1 (possibly including the neighbors across bit d), telling
them to merge first. They merge recursively and after all neighbors of s are of
dimension d or d − 1, we merge the two corners. Naturally, whenever a corner
receives a message from one of its neighbors (of lower dimension) telling it to
merge, it starts the merge procedure too.
3.3
Analysis
Before we bound the runtime of all operations on the system, we prove that with
high probability the system is balanced, i.e. the dimensions of each corner are
roughly the same.
To formally define this notion, we introduce (just for the analysis) a parameter du , which would be the current dimension of the network if it would be a
CCC. This means that if n is the current number of nodes in the network, then
du · 2du ≤ n < (du + 1) · 2du +1 . For du > 2, it holds that du /2 ≤ ln n ≤ 2du .
Additionally, we introduce a parameter dl differing from du by a constant:
dl := du −log `−5. For simplicity of notation, we assume that all nodes’ identifiers
are infinite.
Definition 4. A skew CCC network is balanced if all corners’ dimension are
between dl and du .
Now we show that the network is balanced with high probability. We note
that even in case of bad luck (happening with polynomially small probability),
the system still works — it might just work slower.
Lemma 1. If the network is stable, i.e. no nodes are currently joining or leaving
and no split or merge operations are currently being executed or pending, then
with probability 1 − 2 · n−` the network is balanced.
Proof. We prove two claims:
(i) the probability that there exists a corner with dimension du or greater is
at most n−` ;
(ii) the probability that there exists a corner with dimension dl or smaller is
at most n−` .
For proving (i), we take a closer look at the set S of all node identifiers.
For any string s, let Ss = {si ∈ S : s v si }, i.e. Ss consists of all identifiers
starting with s. We say that S is well separated if for each du -bit string s,
|Ss | ≤ (6` + 7) · du .
First, we observe that if S is well separated, then there is no corner of dimension du or higher. Assume the contrary and choose the corner si with highest
dimension (at least du ). Then there are at most (6` + 7) · du identifiers starting with si , i.e. remaining in this corner. As all the neighbors of corner si have
smaller or equal dimension, this corner should be merged, which contradicts the
assumption that the network is stable.
Second, we show that the probability that S is not well separated is at
most n−` . Fix any string s of length du . For each node i with identifier si let an
s v si . Since s is a du -bit string,
indicator random variable Xis be equal
Pnto 1 if
s
we have P
E[Xis ] = 2−du . Let X s =
X
;
by the linearity of expectation,
i
i=1
n
E[X s ] = i=1 E[Xis ] = n · 2−du ≥ du . Using the Chernoff bound, we obtain that
Pr [X s ≥ (6` + 7)du ] ≤ Pr [X s − E[X s ] ≥ (6` + 6) · E[X s ]]
(6` + 6) · E[X s ]
≤ exp −
3
≤ e−(2`+2)·du
≤ n−(`+1) .
There are 2du ≤ n possible du -bit strings, and thus (by the sum argument) the
probability that S is not well separated is at most n · n−(`+1) = n−` .
Proving (ii) is analogous. We say that S is well glued if for each dl -bit string s,
|Ss | ≥ 12` · dl .
Again, we prove that if S is well glued, then there is no corner of dimension dl
or lower. Assume the contrary and let si be the corner with lowest dimension.
There are at least 12`·dl nodes in corner si . As all the neighbors of corner si have
greater or equal dimension, this corner should be splitted, which contradicts the
assumption that the network is stable.
Again, we show that the probability that S is not well glued is at most n−` .
Let t be
random variables Xit denote if t v si and
Pnany dtl -bit string, tindicator−(d
t
X = i=1 Xi . Then E[X ] = n · 2 u −5−log `) ≥ 32` · du . Using the Chernoff
bound, we get that
Pr X t ≤ 12` · dl ≤ Pr X t ≤ (1 − 1/2) · E[X t ]
≤ e−E[X
t
]/8
≤ e−4`·du
≤ n−2` .
There are 2dl ≤ n possible dl -bit strings, and thus the probability that S is not
well glued is at most n · n−2` ≤ n−` .
t
u
According to Lemma 1, the system is balanced with high probability. Now,
we will show that all basic operations are performed in a time that is logarith-
mic in the current number of nodes. In the following, we assume that the high
probability event of the network being balanced actually happens.
Lemma 2. If the network is balanced, then each search operation is performed
in logarithmic time.
Proof. Since each corner is of dimension Θ(log n), we have to fix Θ(log n) bits
in order to reach the destination corner. As the number of nodes in each corner
is within a constant factor from its dimension, there are Θ(log n) nodes in each
corner. In particular, there is a constant number of spare nodes between any two
consecutive core nodes. Thus, in order to fix the i-th bit after fixing the (i − 1)-st
bit we have to traverse only a constant number of edges. In order to fix the first
bit and to reach the destination node after reaching the destination corner, we
have to travel at most through all the nodes of two corners. Hence, the total
number of traversed edges is O(log n).
t
u
Before we prove upper bounds for join and leave operations we bound the
time of split and merge.
Lemma 3. If the network is balanced, then each split operation is performed in
logarithmic time.
Proof. When we split a corner s of dimension d into corners s · 0 and s · 1 of
dimension d + 1, then there are more than sufficient nodes for each corner and
we know that each neighboring corner is of dimension d or d + 1.
Since the corner s currently has to split and ` ≥ 1, there are at least 12(d+1)
nodes of each type in s. Starting in any node, we traverse the ring a constant
number of times and do the following. In the first pass, we make sure that
there are two connections to neighbor corners across every bit. If there are two
connections already, there is nothing to do and if there is only one, we take any
spare node in the corner we are currently splitting and make a connection to
a spare node in the neighboring corner. From now on, these two spare nodes
(one in each corner) are core nodes. Finally, we add two core nodes without an
outside connection to s; they will be responsible for connecting the two corners
into which we split s.
In the second pass, we use all spare nodes to create two additional rings: one
built of nodes with the d-th bit equal to 0 and the other built of nodes with the
d-th bit equal to 1. Since each ring has at most 2(d + 1) core nodes and at least
12(d + 1) nodes in total, each of the newly created rings has at least 10(d + 1)
nodes. In the next pass, we can go along each of the three rings in parallel and
pass the responsibility for a connection to another corner from the old ring to
one of the new ones. This means that the ring with nodes with d-th bit equal
to 0 takes responsibility for the connections to corners (s ⊕ 2k ) · 0 (analogously
if the last bit is equal to 1).
In the last traversal of all rings, we delete nodes from the old ring and make
them join one of the new rings as spare nodes. Again, we move nodes with the
d-th bit equal to 0 to the newly created corner s · 0 and nodes with the d-th bit
equal to 1 to the newly created corner s · 1.
As we have used only a constant number of traversals of rings of length
O(log n), the whole split operation needs time O(log n).
t
u
We note that in case of search and split operations, the system can be balanced in a weaker sense than we defined, i.e. we just need that the corner dimension is O(log n). The merge operation is the only operation which depends on
the property that the difference between dimension of the corners is constant. It
is also the most time and work consuming function, as a single merge operation
might need executing other merge operations before it can start.
Lemma 4. If the network is balanced, then a merge operation is performed in
time O(polylog(n)), including recursive execution of other needed merge operations, whereas the amortized cost per merge operation is O(log n).
Proof. Since the total number of different dimensions of all corners in the network is bounded by a constant (log ` + 5), the recursive execution of other possibly necessary merge operations for neighboring corners of higher dimensions
has only constant depth. As on each level a corner can have d + O(1) = O(log n)
neighbors, the total number of involved corners is logO(1) n. Below we prove that
a single merge operation of corners s·0 and s·1 of dimension d+1 into a corner s
of dimension d has cost O(d) = O(log n), if all neighbors have been reduced to
equal or lower dimension.
Similarly to the split operation, we can traverse in parallel both rings s·0 and
s · 1 which we want to merge. We denote their dimension by d. As no neighbor
of s · 0 and s · 1 is of dimension d + 1, only d connections of 2d core nodes are
actually used in each of them. We first build a ring composed of these used core
nodes of both old rings, whereas we zip them into one ring interleaving the core
nodes of s · 0 and the core nodes of s · 1. When we remove core nodes from the old
rings, we glue the holes so that we get two rings composed of spare nodes. Next,
we remove two core nodes which have been responsible for the connection across
bit d − 1 (they connected s · 0 to s · 1) and move them to one of the old rings as
spare nodes. In the next traversal of the old rings we calculate how many spare
nodes they contain and then, in the last traversal, we evenly distribute the spare
nodes in the newly created corner s.
Notice that there is no need to add any connections to neighboring corners
— all necessary connections already exist. On the other hand, if our new (d − 1)dimensional corner is a neighbor to another (d − 1)-dimensional one, we have
a double connection with this corner. We should remove one of the connections,
namely the one which originates from s · 1.
Since the cost of merging s·0 and s·1 into s (not including recursive merging of
neighbors) is Θ(d), and the cost of splitting s into s·0 and s·1 has also been Θ(d),
we can amortize the cost of merging into s against the cost of splitting s. This
shows that the amortized cost of a merge operation together with its symmetric
split operation of a corner of dimension d is Θ(d) = Θ(log nm ) = Θ(log ns ),
where nm is the number of nodes in the system at the moment when we perform
the merge operation and ns is the number of nodes at the moment when we have
performed the split operation.
t
u
Lemma 5. The join operation can be performed in logarithmic time.
Proof. Each time a node joins the network, it has to search its position inside the
network and to take its position inside the ring. Based on the previous lemmas,
the search operation can be performed in logarithmic time and the update of
the ring structure involves the creation of two new edges and the removal of
one existing edge. Furthermore, it might happen that the corner has to split its
dimension, resulting in additional O(log n) operations.
t
u
Finally, in the following lemma, we analyze the cost of a leave operation.
Lemma 6. The leave operation can be performed in polylogarithmic and amortized logarithmic time.
Proof. A typical leave operation only triggers a few connection updates. If the
node has been a spare node in its corner, the leave operation involves two connection updates, if the node has been a core node, one additional update has to
be performed to re-connect to the neighboring corner.
Besides the connection updates, a node leaving the network might also trigger
a merge operation of the corner. Based on the previous lemmas, each merge
operation costs at most polylogarithmic time (and logarithmic amortized time)
and majorizes the cost of a leave operation.
t
u
4
Conclusion and Outlook
We have shown a fully distributed but simple scheme which joins a potentially
very large set of computationally weak nodes into an organized network with
minimal possible degree of 3, logarithmic dilation and name-driven routing.
Based on the properties and the structure of the SkewCCC network, it is
possible to further investigate aspects of heterogeneity and locality. The former
means allowing the existence of network nodes which can have a higher degree
and potentially also greater computational power. The latter aspect would incorporate distances of the underlying network.
References
1. J. Aspnes and G. Shah. Skip graphs. ACM Transactions on Algorithms, 3(4),
2007. Also appeared in Proc. of the 14th SODA, pages 384–393, 2003.
2. B. Awerbuch and C. Scheideler. The hyperring: a low-congestion deterministic data
structure for distributed environments. In Proc. of the 15th ACM-SIAM Symp. on
Discrete Algorithms (SODA), pages 318–327, 2004.
3. L. Barrière, P. Fraigniaud, L. Narayanan, and J. Opatrny. Dynamic construction
of bluetooth scatternets of fixed degree and low diameter. In Proc. of the 14th
ACM-SIAM Symp. on Discrete Algorithms (SODA), pages 781–790, 2003.
4. M. Bienkowski, A. Brinkmann, M. Korzeniowski, and O. Orhan. Cube connected
cycles based bluetooth scatternet formation. In Proc. of the 4th International
Conference on Networking, pages 413–420, 2005.
5. N. J. A. Harvey, M. B. Jones, S. Saroiu, M. Theimer, and A. Wolman. Skipnet:
a scalable overlay network with practical locality properties. In Proc. of the 4th
USENIX Symposium on Internet Technologies and Systems, 2003.
6. X. Jiang, J. Polastre, and D. Culler. Perpetual environmentally powered sensor
networks. In Proc. of the 4th Int. Symp. on Information Processing in Sensor
Networks (IPSN), pages 463–468, 2005.
7. D. Karger, E. Lehman, T. Leighton, M. Levine, D. Lewin, and R. Panigrahy.
Consistent hashing and random trees: Distributed caching protocols for relieving
hot spots on the world wide web. In Proc. of the 29th ACM Symp. on Theory of
Computing (STOC), pages 654–663, 1997.
8. F. T. Leighton. Introduction to parallel algorithms and architectures: array, trees,
hypercubes. Morgan Kaufmann Publishers, 1992.
9. D. Malkhi, M. Naor, and D. Ratajczak. Viceroy: A scalable and dynamic emulation
of the butterfly. In Proc. of the 21st ACM Symp. on Principles of Distributed
Computing (PODC), pages 183–192, 2002.
10. M. Naor and U. Wieder. Novel architectures for P2P applications: The continuousdiscrete approach. ACM Transactions on Algorithms, 3(3), 2007. Also appeared
in Proc. of the 15th SPAA, pages 50–59, 2003.
11. A. I. T. Rowstron and P. Druschel. Pastry: Scalable, decentralized object location,
and routing for large-scale peer-to-peer systems. In Middleware, pages 329–350,
2001.
12. I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek,
and H. Balakrishnan. Chord: a scalable peer-to-peer lookup protocol for internet
applications. IEEE/ACM Transactions on Networking, 11(1):17–32, 2003. Also
appeared in Proc. of the ACM SIGCOMM, pages 149–160, 2001.
13. B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, and J. Kubiatowicz.
Tapestry: A resilient global-scale overlay for service deployment. IEEE Journal on
Selected Areas in Communications, 22(1):41–53, 2004.