A Case for Virtualizing Nodes on
Network Experimentation
Testbeds
Konrad Lorincz
Harvard University
July 29, 2017
Motivation
In the beginning …. 1960s, 1970s


hardware was expensive => commodity
computer systems shared by multiple users
Today



hardware is relatively cheap
connectivity and location are a commodity
Network experimentation testbeds are expensive => shared
»
ex: PlanetLab, Globus,
Centurion
PlanetLab
July 29, 2017
2
Sharing Scenario

Current mode of sharing
1.
2.

Each user has an account on the testbed
To perform experiment – user logs on one node, runs script
which starts jobs on a subset of the nodes
Unfortunate Scenarios
1.
Inconsiderate user: Node N is performing critical
experimental computation for user A
»
2.
Considerate user: Node N is running A’s experiment
»
July 29, 2017
User B runs his/her script that starts computationally intensive
job on node N
User B checks load on node N, determines it’s very low, starts a
moderately computation intensive job on node N
3
Current Way to Solve This

In a Small Lab

John posts a message on the bulletin board …
Please don’t use machines X, Y, and Z between
5/2/02 – 5/5/02. I am running my experiments.
Thanks,
-John
PS. If you absolutely must log on one of these machines during this
time, please don’t run any computationally intensive jobs.

Analysis


July 29, 2017
May work OK in a small lab => clearly does not scale
Inefficient use of resources
4
Desired Sharing Properties
Need a good mechanism for sharing so that users don’t get in each other’s way

Ideal Scenario: Researcher A specifies to the scheduler that it needs N number of
machines/nodes, for H number of hours, each with W amount of RAM, X amount of
disk size, CPU equivalent to Y MHz, and bandwidth of Z Mbps.
Desired Properties of Mechanism for Sharing


Efficient multiplexing of resources amongst users
Control over resources
»
»

Isolation
»
»

Reservation of resources
Enforcement
Security
Performance
Customization
July 29, 2017
5
Solution
Claims
1.
2.
July 29, 2017
Sharing problem can be solved by
=> virtualizing the testbed nodes
Good way to virtualize nodes
=> use a VMM
6
Advantages of Virtualization

Efficient Multiplexing of Resources


Takes advantage of statistical multiplexing (demand-based)
Resources requested, if granted, are guaranteed for the
duration of the experiment
»
July 29, 2017
Only need to request resources needed => may be very few
7
Advantages of Virtualization

Control Over Resources




July 29, 2017
Can be fine-grained and precise
Allows for reservation/allocation of resources
Enforcement of reservations and policies
Commercialization? – if it can be controlled and accounted
for => service could be sold
8
Advantages of Virtualization

Isolation

Security
»
»
»
»
Almost perfect security from other VMs (i.e. users)
Namespaces of each VM is private => cannot even name a
resource on another VM
Even if a VM is compromised, the rest are not
Downside:


Performance – aim for soft guarantees
»
Actions of other users on the node don’t affect my availability to
resources (guarantees)

July 29, 2017
Resources don’t span multiple protection domains
=> can’t share a common service across VMs
If Bob overloads his VM, it won’t affect my VM
9
Advantages of Virtualization

Customization


User may ask for exact machine configuration he/she
requires (RAM, disk size, CPU, network bandwidth)
May load different OSes or modified/customized ones
»
»
»

Easy management
»
July 29, 2017
User may require a particular OS (FreeBSD)
Denali kernel can be loaded and run on
User has superuser privileges
user configures once a customized VM image
=> transfers it to required nodes or other testbeds
10
Arguments Against Virtualization

Overhead of Running a VMM


July 29, 2017
Relatively small! VMware server, Denali + others report
about 10%
Hardware is getting faster => overhead will be even less of a
factor
11
Arguments Against Virtualization

Use Dedicated Physical Machines

Reservation too-coarse grained
»

must dedicate an entire physical machine to a user => may
require the equivalent of 1/8 of the machine
More expensive
»
More expensive to buy 6-12 machines then one more powerful
one + VMM
"We're running 20 virtual machines on one four-way
system, and it's handling everything from CRM
applications and security to application development
and testing, all of which has saved us huge amounts
of time and money in hardware costs."
-- Alan Thomas
Senior Technical Consultant, National Gypsum
Company
July 29, 2017
12
Arguments Against Virtualization

Do/Implement the sharing at the OS level



July 29, 2017
VMMs are much simpler than conventional OSes
Other mechanisms at the OS level impose particular software
interfaces or models to the user => with a VMM it’s clear, you get a
“raw” machine
Cannot provide strong performance isolation => the need to support
high-level abstractions prevents OSes from providing strong
performance guarantees
» Precise resource accounting is difficult because resources
intertwined in the implementation of abstractions. ex: file buffer
cache and TCP\IP socket buffers consume memory resources
that aren’t charged to any particular app. [Denali paper]
13
Can It Be Implemented?

Not only feasible, but practical

Several off the shelf products exist:
»
»
»


For us, VMMs only needs to support half-doze to dozen VMs,
can easily handle (VMware ESX supports up to 64 VMs)
If a particular VMM doesn’t provide guarantees for a
resource, it can be easily extended with well known
scheduling algs.
»
»
»
July 29, 2017
VMware ESX server
IBM z/VM
Disco (for NUMA)
Fair queuing => network bandwidth
Stride scheduling => CPU
Cello framework => disk bandwidth allocation
14
Conclusion


Network experimentation testbeds are a commodity
Users have different needs + require guarantees.
Currently no formal way to handle this

Virtualizing testbed nodes by running a VMM is a
good solution.
1.
2.
3.
4.
July 29, 2017
Efficient multiplexing of resources
Fine-grained control over physical resources
Isolation
Customization
15
App 1
App 1
App 2
App 1
…
OS 1
OS 2
HW
HW
…
OS 5
HW
VMM
Hardware (HW)
July 29, 2017
16