An Efficient Remote Routing Mechanism in Space-Parallel
Network Simulation
Zhiyu Hao, Xiaochun Yun, Hongli Zhang
(Corresponding author: Zhiyu Hao)
School of Computer Science and Technology, Harbin Institute of Technology
Harbin 150001, China
(Email: [email protected])
Abstract: Simulation is an efficient tool to analyze the events of network security, such as worm propagation and
DDoS attacks. Remote routing mechanism is one of the key problems for the space-parallel simulations. This
paper presents an abstract model of remote routing mechanism, and proposes border-router based remote routing
mechanism, which computes remote routing states based on the border routers, and looks up the appropriate
routing states by comparing the length of different remote routes. Experimental results show that, our approach
reduces 90% of the size of the remote routing table, about 20% of the storage space during the initialization of the
simulation, and over 25% of the simulation time.
Keywords: network simulation, space-parallelization, remote routing mechanism
Corresponding author:
Zhiyu Hao
Email: [email protected]
Addr: Room A502, Jia No. 3, Yumin Road, Chaoyang Distr. Beijing City, Beijing, China, 100029
Tel: +86-010-82990826
Fax: +86-010-82990501
An Efficient Remote Routing Mechanism in Space-Parallel
Network Simulation
Zhiyu Hao, Xiaochun Yun, Hongli Zhang
(Corresponding author: Zhiyu Hao)
School of Computer Science and Technology, Harbin Institute of Technology
Harbin 150001, China
(Email: [email protected])
Abstract
Simulation is widely recognized as an
essential tool to analyze large-scale networks,
especially for the behavior of network security
events such as worm propagation or DDoS
attacks. One approach to creating simulation
models for large-scale topologies in network
simulations is to use a space-parallel
simulation. Remote routing mechanism is one
of the key problems for the space-parallel
simulations. Most of the current space-parallel
simulators compute and store the remote
routing states between each pair of nodes
based on the complete topology, which suffers
from the excessive storage space requirement
and CPU time usage. This paper presents an
abstract model of remote routing mechanism,
and proposes a new approach to remote
routing state calculation, storage and lookup,
named border-router based remote routing
mechanism, which computes remote routing
states based on the border routers, and looks
up the appropriate routing states by comparing
the length of different remote routes.
Experimental results show that, our approach
reduces 90% of the size of the remote routing
table, about 20% of the storage space during
the initialization of the simulation, and over
25% of the simulation time.
Keywords: network simulation, space
parallelization, remote routing mechanism
1
Introduction
The last decade has seen the rapid growth
of Internet, and security of Internet is greatly
threatened by the malicious events such as
worms and DDoS attacks. Due to the
complexity of networks, simulation plays a
vital role in attempting to characterize both the
behavior of the Internet and the research on
the network security. The use of simulation is
becoming increasingly prevalent in the
networking research community. And some
network simulators are developed and widely
used, such as NS-2[1], SSFNet[2, 3], and
REAL[4].
However, Simulation tools have not been
able to keep up with the rapid increases in the
scale, complexity and speed of modern
networks. A packet level simulation of a
network containing a few hundred nodes can
be expected to require hours, perhaps even
days, of CPU time to simulate only a few
minutes of network operation using a
contemporary workstation. While some
applications will require models containing
millions to billions of network nodes for
scalability studies. Other simulations require
long periods of network operation to be
simulated to capture statistics concerning
infrequent events. So it is clear that
space-parallel network simulation is one of the
approaches to achieve the performance needed
to simulate these networks. The space
parallelization of network simulations has
been studied for a long period, and there are
some well-known systems such as PDNS[5,
6], GTNetS[7], DaSSF[8]. Most of them adopt
Parallel Discrete Event Simulation (PDES)
mechanism [9] to achieve the parallelization.
Space-parallel network simulation greatly
releases the limitation of the hardware
resources, and remarkably enhances the
simulation
scale.
However,
space
parallelization also introduces some additional
problems. Remote routing mechanism is one
of the most important among them. Since in a
space-parallel network simulation, the model
for the entire simulated network is divided
logically into k sub-models, where k is the
number of federates in the distributed
simulation. With this approach, each
simulation instance running on a federate is
responsible for approximately 1/k of the entire
topology model, and instantiates simulation
objects to represent its own portion of the
network elements in the complete topology.
Since a given instance has no responsibility
for the remaining (k-1)/k portion of network
elements, no simulation objects are created
and thus the instance has no knowledge about
the remaining topology. In order to provide
the correct packet forwarding, it is needed of
the remote routing states information and the
corresponding method of looking up routing
states for each instance.
In most of the current space-parallel
network simulators, the remote routing states
are computed based on the complete topology
information. The shortest-path routes are
calculated for each pair of nodes in the
topology, and then remote routing states are
computed according to the partition
information. However, this method suffers
from the excessive CPU time usage because of
the high computation complexity. Besides, the
large amount of the remote routing states will
cause pretty large storage and time
requirements for constructing the remote
routing table for each simulation instance, and
also low efficiency of remote routing state
lookup. This paper presents a border-router
based remote routing mechanism, which
computes remote routing states for each
border router on an instance according to the
partition information. When a remote routing
state lookup is needed, the simulation instance
chooses the appropriate routing state by
comparing the total length of both the local
and remote paths. Since only the border-router
relative remote routing information is
considered, border-router based remote
routing mechanism reduces both the
computation time and the amount of the
remote routing states. Moreover, it reduces the
initialization time, storage space for the
remote routing tables, and the lookup time
requirement for the remote routing states.
Therefore the efficiency of the space-parallel
simulation is highly enhanced.
The remainder of this paper is organized
as follows. Section 2 gives an abstract model
of remote routing mechanism. Section 3
describes the design and implementation
details of our border-router based remote
routing mechanism. Section 4 gives the
experimental results. Finally, section 5
describes conclusions and future directions of
our research.
2
Abstract model of remote routing
mechanism
Assume that the simulated topology
graph is denoted by G  (V , E ) , and G is
divided
into
k
sub-graphs,
which
are
G0  (V0 , E0 ) , G1  (V1, E1) , …, Gk 1  (Vk 1, Ek 1) .
Erlink is the set of all the edges that are cut by
the partition algorithm. Then there should be
V
k 1
i 0
E
k 1
i 0
Vi
Ei  Erlink .
Assume that m is current router node that
a packet has just been forwarded to, m Vi . n
is the destination node of the packet, n V j ,
and i  j . Then the objective of the remote
arrives at the destination.
P1
routing mechanism is to provide an algorithm
rLookup, by which the current simulation
r1
instance related to Gi can get a proper border
r2
c0
1:
assume
r3
r4
r7
router r, which satisfies the condition below.
Condition
P3
r8
that
r5
pathm, r ,Gi  (m,..., r ) is one of the shortest paths
r9
r11
between m and r in Gi, pathm,n,G  (m,..., n) is
r10
a shortest path between m and n in G, which
satisfies pathm, r ,Gi  pathm, n,G .
Take the simple topology in Figure 1 as
an example. The topology graph is divided
into 4 sub-graphs, each is simulated by a
separate instance, say P1 to P4. When a packet
is forwarded from a client node c0 in P1 to a
server node s0 in P4, the shortest path in the
complete topology r1 to r6 is used. A dashed
line in Figure 1 denotes a local path in a
certain instance. Since P1 doesn’t have the
topology knowledge of the other instances, in
s0
r6
P2
P4
Fig. 1 Example of the space-parallel
network simulation
According to the description of the
abstract model of remote routing mechanism,
there are three key problems to solve: 1)
computation algorithm of remote routing
states, 2) storage method of the remote routing
state information, and 3) algorithm of
rLookup.
3
Border-router
based
routing mechanism
remote
order to forward the packet correctly, it needs
the remote routing information that a packet
can be transmitted to r6 through the link (r2,
r3). For the same reason, P3 needs the remote
routing information that a packet can be
transmitted to r6 through the link (r4, r5).
While in order for the transmission of the
reverse packets from s0 to c0, the remote
routing information that a packet can be
forwarded to r1 through border router r5 and r3
is required too. When a packet on node c0
needs the route computation, the method
rLookup is used on P1 to get the proper border
router r2. Then the packet is forwarded to r2 by
using the local routing mechanism, and
transmitted to P3 through the remote link (r2,
r3). This procedure continues until the packet
Here are some definitions used for the
description of the remote routing mechanism.
Definition 1 (Remote link). A link that
belongs to Erlink , in other words, a link that is
across different sub-graphs is called a remote
link. In Figure 1, the links (r2, r3), (r4, r5), (r7,
r11), (r8, r9) and (r10, r11) are all remote links.
Definition 2 (Border router). The end
nodes of any remote link are called border
routers. The node r2, r3, r4, r5, r7, r8, r9, r10 and
r11 in Figure 1 are border routers. It should be
noted that, a border router may have more
than one remote links. For example in Figure
1, border router r11 has two remote links
which are (r7, r11) and (r10, r11).
Definition 3 (Terminal router). The
router nodes that are directly connected with
at least one leaf node in the complete
topology. In Figure 1, router node r1 is
connected with leaf node c0, thus r1 is a
terminal router, and so is r6.
Definition 4 (Locally reachable). We
say that two nodes in the same sub-graph are
locally reachable, if there is at least one local
path between them in this sub-graph. The two
nodes r1 and r2 in Figure 1 are locally
reachable since there is a local path denoted
by the dashed line between them.
First, we give an overview on the simple
complete-topology based remote routing
mechanism. In order to calculate all the
remote routing states that might be used
during the simulation, the shortest paths
between any pair of the terminal routers
should be considered in the complete-topology
based remote routing mechanism. The process
procedure for each pair of terminal routers is
as follows:
i) Compute the shortest path between
them according to the complete topology.
ii) Travel through this path to find all the
remote links in it.
iii) For each end node of a remote link,
add a remote routing state for it in the proper
sub-graph.
The computation is finished when the
shortest paths between any pair of terminal
routers have been handled as is described
above. The format of a remote routing state is
BorderRouterID rLinkIP SrcIP SrcMask
DestIP DestMas
where BorderRouterID is the ID of the current
border router, rLinkIP is the remote link that a
packet should be forwarded through, SrcIP is
the address of the source node, SrcMask is the
net mask of the source address, DestIP is the
address of the destination node, and DestMask
is the net mask of the destination address. All
the remote routing states compose the remote
routing table in each sub-graph. When a
remote routing state lookup is required, the
simulation instance looks up its remote
routing table, finds all the appropriate remote
routing states by comparing the source and
destination addresses of the packet with the
SrcIP，SrcMask，DestIP，DestMask in each
routing state, and then chooses a proper
remote routing state whose BorderRouterID is
locally reachable to the current router node.
After forwarding the packet to this border
router represented by BorderRouterID, the
instance uses the remote link represented by
rLinkIP to forward the packet to another
instance. And this procedure continues until
the packet reaches its destination.
According to the description above, in
the complete-topology based remote routing
mechanism, the simulator should store a few
remote routing states for each pair of terminal
routers. The number of the states is
determined by the number of remote links
included in the path between the terminal
routers. So the size of the remote routing table
is O(l*n2), where l is the average number of
remote links that a shortest path between a
pair of terminal routers includes, and n is the
total number of terminal routers. The high
complexity will greatly reduce the efficiency
of constructing and looking up the remote
routing table. For example, in a topology
graph of 1,000 nodes, the topology is divided
into 8 sub-graphs. Assume that there are
totally 500 terminal routers, and the average
number of remote links in each shortest path
l=4. Then after the computation of the
complete-topology based remote routing
mechanism, the total number of remote
routing states is 1,000,000, and the remote
routing table size in each instance is about
120,000, which is absolutely a big problem for
the simulation efficiency.
In order to reduce the size of the remote
routing table, in each remote routing state,
only the destination address is stored, with the
source address not included. Therefore the
remote routing state computation changes to
the computation procedure based on the
border routers, and the mechanism cares only
for the paths through which remote link from
a certain border router, a packet can be
transmitted to its destination. But there may be
multiple border routers through which a
packet can reach the same destination in a
sub-graph, and when a lookup is required,
multiple remote routing states will be found
and the algorithm should choose a proper one
according to some reasonable rules. Therefore
additional information should be included in
each remote routing state, which is the length
of the shortest path from this border router to
the destination node in the complete topology
graph. So the format of a remote routing state
is revised as follows:
BorderRouterID rLinkIP DestIP DestMask
PathLength
where PathLength is the length of the shortest
path from BorderRouterID to DestIP in the
complete topology, and the meaning of the
other parameters are the same as in the
complete-topology based remote routing
mechanism.
With the remote routing table computed
by the procedure mentioned above, the
algorithm of finding a remote routing state is
as follows. First， the simulation instance
looks up its remote routing table to find all the
remote routing states whose DestIP matches
the destination address of the packet. For each
of these remote routing states, if the border
router of it is locally reachable for the current
node, the length of the local path from the
current node to this border router is denoted as
localpathlen. Then the instance chooses a
remote routing state which satisfies the
conditions of 1) its BorderRouterID is locally
reachable for the current node, and 2) it has
the smallest value of the localpathlen plus the
PathLength.
In the border-router based remote routing
mechanism, the simulator should compute the
shortest paths for each pair of a border router
and a terminal router. For the worst case, each
of the shortest paths starts with a remote link.
Therefore the size of remote routing table is
O(m  n) , where m is the total number of the
border routers, and n is still the number of
terminal routers. Since in most simulation
applications, each instance is responsible for
thousands of nodes, and if the partition
algorithm is reasonable enough to create
remote links as few as possible, the border
routers in each instance will be normally in
the magnitude of ten. Therefore the space
complexity of the border-router based
mechanism is much lower than the
complete-topology based one.
Algorithms of border-router based remote
routing mechanism are given below.
1. Algorithm of remote routing state
computation.
Suppose V is the set of border routers, T
is the set of terminal routers, T  N , R is the
set of remote links, the shortest paths from a
certain border route n to all the terminal
routers in T are stored in Path[N], each record
in Path[N], say Path[m], is a list of the node
ID along the path from n to m.
procedure CalRemoteRouting
for all n  V do
Path[ N ]  Dijkstra(n);
for all m  T do
if FirstHop( Path[m])  R then
add a remote routing record;
end if
end for
end for
end procedure
It should be noted that in the algorithm,
when computing the remote routing states, all
the terminal routers are considered, not only
the ones that are not in the same instance as
the border router n. That is because there may
be some shortest paths between a local
terminal router and n that are across different
instances.
2. Algorithm of remote routing lookup.
The algorithm of remote routing state
lookup is shown in Figure 2.
Suppose that the packet is now at node n, and the packet header information is
stored in p, RRouterTable is the remote routing table in this instance.
procedure HandlePacket(n, p)
( Length, BorderRouter )  LookupRRouterTable(n, p);
if localhops  Length and localhops  0 then
//It is a local route
Transfer p through local route；
else //the destination of p needs a remote route
Transfer p through BorderRouter;
end if
end procedure
procedure LookupRRouteTable(n, p)
Length  MAXLENGTH ;
BorderRouter  NULL;
for all record  RRouterTable do
if destination address of p matches record.dest
and n and record.borderrouter are locally reachable then
Length  min( Length, localhops  record .PathLength);
if Length  localhops  record .PathLength then
BorderRouter  record .BorderRouter;
end if
end if
end for
return (Length, BorderRouter);
end procedure
Fig. 2 Algorithm of remote routing state lookup
For any remote routing state computed
the procedure LookupRRouterTable, the
by the algorithm, it should be one portion of a
procedure HandlePacket chooses the local
shortest path for the complete simulated
path to forward the packet p, as is described in
topology. Otherwise the algorithm won’t work
the 4th line in algorithm 2.
even though it has high efficiency and small
2) If it is a path with at least one remote
space requirement. The proof is as follows.
link, since in algorithm 1, the RRouterTable
Conclusion 1. The paths found by the
includes all the remote routing states for the
border-router
based
remote
routing
current node to the destination, the packet p
mechanism are the shortest paths for the
should choose a state whose DestIP matches
complete topology.
the destination to forward p. Using reduction
Proof: The shortest paths from the
to absurdity. Assume that algorithm 2 finds the
current node to the destination may either be
remote routing state with the border router
just a local path, or include one or more
BR0, and the path through it is not a shortest
remote links.
path for the complete topology. Then there
1) If it is a local path, then in algorithm 2,
must be another remote routing state with the
after the comparison of the length of the local
border router BR, through which the length of
path with the shortest remote path found by
the path is the shortest for the complete
topology. So there must be
localhopsBR  PathLengthBR 
localhopsBR0  PathLengthBR0
.
While according to algorithm 2, there must be
localhopsBR  PathLengthBR 
localhopsBR0  PathLengthBR0
,
which leads to a contradiction. Hence the path
found by the algorithm is the shortest path for
the complete topology.□
According to the description above, the
border-router
based
remote
routing
mechanism can greatly reduce the size of the
remote routing table and therefore highly
improve the simulation efficiency, with the
shortest paths for the complete topology being
correctly chosen.
4
Performance
To investigate how our approach
performs in a real simulator, we integrated
border-router
based
remote
routing
mechanism
into
well-known
PDNS,
performed benchmarks, and compared the
results with the complete-topology based
remote routing mechanism. In this section, we
describe details of the experiments and
analyze the experimental results.
We ran our experiments on 8 PCs, each
with a Pentium IV 1.0GHz CPU, 2 GB RAM,
and running RedHat Linux operating system.
The router topologies for the simulations were
built using a synthetic topology generator
NEM[10], which can generate topologies with
different parameters such as topology size and
average node degree. We repeated each
experiment 5 times, and use the average as the
experimental results.
We generated different size of topologies
from 500 to 4,000 nodes as the router
topologies, with an average node degree of
2.6. Each router whose degree is 1 is
connected with 32 leaf nodes. Each leaf node
randomly chooses another to create an FTP
connection, and transmit a 10,000-byte file.
The simulated time is 300 seconds.
The following parameters are used to
compare the efficiency of the two approaches.
 Computation time for the remote routing
states, which specifies the time needed to
compute all remote routing states.
 Size of the remote routing table, which is
the total number of the remote routing
states.
 Initialization time, which specifies the
time to initialize the simulation instances
before the application is actually run. The
time to initialize the remote routing table
for each instance is included in this
parameter.
 Memory usage after initialization, which
specifies the average memory usage for
each instance after the initialization. For
the two approaches, the memory used for
the topology and application information
is identical, so the difference of the
memory usage after initialization between
them is caused by the storage space for
the remote routing table.
 Simulation time, which specifies the time
needed to simulate the FTP applications.
This parameter is computed from the end
of the initialization.
The relationship between the three
time-related parameters is shown in Figure 3.
Computation time for the
remote routing states
Pre-computation
phase
Initialization time
Simulation time
Simulation
phses
Fig. 3 Relationship between the time-related
parameters
The results of the experiments are shown
in Figure 4 to Figure 8. We can see from
Figure 4 and 5 that, the computation time for
the remote routing states of the border-router
based remote routing mechanism is about 75%
less than that of the complete-topology based
mechanism, and the size of the remote routing
table is 90% less, which shows obviously
great enhancements.
Computation time for the remote routing table
700
Complete-topology based
Border-router based
500
400
300
Initialization time
200
1400
100
1200
0
0
500 1000 1500 2000 2500 3000 3500 4000 4500
Topology size
Fig. 4 Computation time
Remote routing table size
Initialization time(s)
Computation time(s)
600
(actually about 10% of the total number of
simulation events for most applications), the
improvements are pretty large. And as is
shown in Figure 7, the memory usage after the
initialization is over 20% less than that of the
complete-topology based mechanism, which
leads to the improvement for the simulation
scale.
Complete-topology based
Border-router based
1000
800
600
400
200
0
500 1000 1500 2000 2500 3000 3500 4000 4500
Topology size
6000000
4000000
2000000
0
0
500 1000 1500 2000 2500 3000 3500 4000 4500
Topology size
Fig. 5 Size of the remote routing table
As is described in Section 2, the great
reduction of the remote routing tables will
lead to the reduction of the time and memory
usage for the initialization of the simulation
instances, and also the simulation time. This is
proved by the results shown in Figure 6-8. We
can see from Figure 6 that the initialization
time of the border-router based mechanism
reduces about 25% compared with that of the
complete-topology based mechanism. And the
simulation time of our approach is also about
25% less than that of the complete-topology
based one, as is shown in Figure 8. Since the
number of the events of remote routing state
lookup is probably only a small portion of the
total events during the simulation process
Memory usage after initialization(MB)
Fig. 6 Initialization time
Memory usage after initialization
700
Complete-topology based
Border-router based
600
500
400
300
200
100
0
0
500 1000 1500 2000 2500 3000 3500 4000 4500
Topology size
Fig. 7 Memory usage after initialization
Simulation time
Complete-topology based
Border-router based
6000
Simulation time(s)
Remote routing table size
0
Complete-topology based
Border-router based
8000000
5000
4000
3000
2000
1000
0
0
500 1000 1500 2000 2500 3000 3500 4000 4500
Topology size
Fig. 8 Simulation time
According to the experimental results
shown above, the space and time requirements
for the border-router based remote routing
mechanism are both improved compared with
that of the traditional complete-topology based
mechanism, and so does the simulation
efficiency.
Techniques and Applications, June 1999.
[4]
Keshav S. REAL : A Network Simulator.
Computer Science Department Technical Report
88/472, UC Berkeley, December 1988.
[5]
Riley G, Fujimoto R, and Ammar M. A Generic
Framework
for
Parallelization
of
Network
Simulations. In Proceedings of 7th International
Symposium on Modeling, Analysis and Simulation of
5
Conclusions and future work
Computer
and
Telecommunication
Systems
(MASCOTS’99), October 1999.
This paper presents a new approach of
border-router
based
remote
routing
mechanism for the space-parallel network
simulation, which computes the remote
routing table based on the border routers, and
looks up the remote routing states according to
the length of both the local and remote paths.
Experimental results show that, compared
with the traditional complete-topology based
remote routing mechanism, our approach
reduces the size of the remote routing table for
about 90%, and the memory usage for the
simulator initialization decreases for over
20%, and the simulation time reduces for
about 25%.
Our implementation of the border-router
based remote routing mechanism in PDNS is
somewhat quick and dirty. In the future we
will propose more efficient algorithm of
remote routing state lookup. Besides, we will
also focus on the other factors that affect the
efficiency
of
space-parallel
network
simulation, such as the partition algorithms
and so on.
References
[1]
McCanne R and Floyd S. The LBNL network
simulator. Lawrence Berkeley Laboratory, 1997.
[2]
Cowie J, Nicol D, and Ogielski T. Modeling the
global
Internet.
Computing
in
Science
and
Engineering, January 1999.
[3]
Cowie J, Liu H, Liu J et al. Towards realistic
million-node internet simulations. in International
Conference on Parallel and Distributed Processing
[6]
Riley
G,
Fujimoto
Parallel/Distributed
R,
NS.
and
Ammar
Software
www.cc.gatech.edu/computing/
M.
on-line:
compass/
pdns/
index.html, 2000. Georgia Institute of Technology.
[7]
Riley G. The Georgia Tech Network Simulator.
In Proceedings of the ACM SIGCOMM workshop on
Models, methods and tools for reproducible network
research, pp. 5–12, ACM Press, 2003.
[8]
Liu J and Nicol D. DaSSF 3.1 user’s manual.
April 2001.
[9]
Fujimoto R. Parallel Discrete Event Simulation.
Communications of ACM, 1990, 33(10): 30~53.
[10] Magoni D and Pansiot J. Internet Topology
Modeler Based on Map Sampling. In Proceedings of
7th
IEEE
Symposium
on
Computers
Communications, pp. 1021-1027, July 2002.
and