Gaining Control of
Virtualized Server
Environments
By Jim Metzler, Ashton Metzler & Associates
Distinguished Research Fellow and Co-Founder,
Webtorials Editorial/Analyst Division
Introduction
The traditional data center network was based on an architecture and design tailored to satisfy the
connectivity requirements of a relatively simple server infrastructure based on the following set of
assumptions:

Each server was dedicated to a single application and had a single IP address and a single MAC
address.

Nearly all of the traffic within the data center was client/server related. This type of traffic is often
referred to as “north-south” traffic.

The relatively small amount of traffic that went between servers was readily visible on physical
switch ports. This type of traffic is often referred to as “east-west” traffic.

The physical separation of servers satisfied most of the requirements for security and for the
delineation of departmental resources.

The static nature of the application environment lent itself to manual processes for server
deployment or redeployment.

VLANs tended to be used primarily for traffic isolation and to provide high availability between the
access and distribution tiers. As a result, VLANs tended to be localized within application pods.

There were clear lines of demarcation between the server and application management domain and
the network management domain.
Interestingly enough, the implementation of server virtualization has negated the validity of each of the
assumptions listed above. As a result, the traditional approach to designing and managing data center
networks is no longer valid.
The goal of this white paper is to discuss what IT organizations must do to successfully meet the
networking challenges associated with a highly dynamic, virtualized data center. In order to achieve
this goal, three IT professionals were interviewed. One of the interviewees was the senior manager of
global network design for a major health care organization. His organization doesn’t allow him to be
explicitly named in a document such as this, so he will be referred to in this white paper as The Senior
Gaining Control of Virtualized Server Environments
June 2011
Page 1
Manager. The other two interviewees were Aaron Larkins, the CEO of Profitability.net and Charles
Hollingsworth, the manager of IT Architectural Planning at Georgia State University.
The Adoption of Server Virtualization
There are numerous benefits that can be derived from server virtualization, including:

Server Consolidation
Once it has been virtualized, a single physical server can support multiple virtual machines (VMs)
and as a result, applications that would normally require a dedicated server can now share a single
physical server. This enables IT organizations to reduce the number of servers in their data centers
which yields a significant savings in both CAPEX (i.e., costs of server hardware, SAN Host bus
adapters, and Ethernet NICs) and OPEX; i.e., server management labor expense, plus facility costs
such as power, cooling, and floor space.

Flexible Server and Application Provisioning
A production VM can be transferred without service interruption from a given physical server to a
different physical server, either within the same data center or between different data centers. This
capability enables workload management and optimization across an IT organization’s virtualized
data center(s). This capability also helps IT organizations meet business objectives by:
o
Streamlining the provisioning of new applications.
o
Improving backup and restoration operations.
o
Enabling zero-downtime maintenance.
o
Enable Other Initiatives
In order to more effectively manage and control user desktops, many IT organizations are implementing
virtual desktops. Virtualized servers are a prerequisite for virtualized desktops because a virtual
desktop implementation requires that a VM in a data center server hosts a complete user desktop
including all its applications, configurations, and privileges.
A recent market research report1 contained the results of a survey that asked IT professionals about
their organization’s adoption of server virtualization and the impact of that adoption on networking and
management. Throughout this white paper that report will be referred to as The Research Report.
The Research Report presented the responses to a survey question in which the respondents were
asked to indicate the percentage of their company’s data center servers that have either already been
virtualized or that they expected would be virtualized within the next year. Their responses are shown
in Table 1.
1
http://www.webtorials.com/content/2010/11/2010-cloud-networking-report.html
Gaining Control of Virtualized Server Environments
June 2011
Page 2
None
1% to 25%
26% to 50%
51% to 75%
76% to 100%
Have already been
virtualized
21.6%
33.0%
18.9%
15.1%
11.3%
Expect to be virtualized
within a year
12.4%
25.6%
21.9%
21.9%
18.2%
Table 1: Deployment of Virtualized Servers2
One observation that can be drawn from the data in Table 1 is that over the next year the deployment
of server virtualization will continue to increase. For example, over the next year the percentage of IT
organizations that will have virtualized the majority of their data center servers is anticipated to almost
double.
Georgia State University is an example of an organization that has made only a moderate deployment
of server virtualization as they have virtualized roughly one third of their servers.
The company that The Senior Manager works for is an example of an organization that has been very
aggressive with implementing server virtualization. That company has already virtualized roughly
eighty percent of their five thousand servers and their goal is to get as close as they can to having one
hundred percent of their servers virtualized. The Senior Manager stated that they will not virtualize all
of their servers in the near term in part because in some instances porting the code from a physical
server to a VM is too expensive and in part because of the compliance concerns they have relative to
some of their healthcare applications.
As previously noted, once a server has been virtualized it can support multiple VMs. One measure of
the efficiency, and hence the cost savings of server virtualization, is the average number of VMs on a
single physical server. This number is typically referred to as the consolidation ratio. In the current
environment, organizations are typically implementing consolidation ratios between 5:1 and 10:1. The
consolidation ratio is expected to grow rapidly as IT departments become more familiar with
virtualization technology and as they deploy multi-core server processors that are optimized for
virtualization.
The company that The Senior Manager works for is an example of a company that is driving cost
savings by implementing a high concentration ratio. According to The Senior Manager they currently
average fifty VMs per physical server and that number is likely to increase in part due to the fact that
they are aggressively implementing 10 Gbps ports.
IT organizations, however, will only be able to appreciate these added cost savings if they redesign
their data center networks and change their approach to management. The Research Report indicated
that most IT organizations understood that and were making progress in those directions. For example,
The Research Report presented the responses to a survey question in which the respondents were
asked “Has your IT organization already redesigned, or within the next year will it redesign, its data
center LAN in order to support cloud computing in general, and virtualized servers in particular?”
Those responses are shown in Table 2.
2
http://www.webtorials.com/content/2009/12/cloud-computing-a-reality-check-guide-to-risk-mitigation.html
Gaining Control of Virtualized Server Environments
June 2011
Page 3
Already Have
Cloud Computing in
General
Virtualized Servers in
Particular
No Plans
28.6%
Will Within the
Next Year
42.9%
50.5%
30.7%
18.8%
28.6%
Table 2: Redesign of the Data Center LAN
The data in Table 2 indicates the significant impact that server virtualization is having on the design of
data center networks. As discussed in The Research Report, some of the primary design changes that
IT organizations are making to their data center networks in order to better support server virtualization
include implementing:

A 2-Tier network topology to improve traffic flows among servers.

Edge Virtual Bridging (EVB) to provide scalable I/O performance and full visibility of VM-VM traffic
on the same physical server.

The combination of multi-chassis LAGs and virtual LAN switching to improve performance and
availability via support for active-active Layer 2 switching and load balancing across redundant
source/destination paths.
The Research Report also indicated that most IT organizations understand the importance of modifying
how they manage in a virtualized environment. In particular, The Research Report presented the
responses to a survey question in which the survey respondents were asked to indicate how important
it is to their organization to get better over the next year at managing some key tasks related to server
virtualization. They were given the following five-point scale:
1.
2.
3.
4.
5.
Not at all important
Slightly important
Moderately important
Very important
Extremely important
Included in Table 3 are the tasks and the percentage of the survey respondents who indicated that the
task was either very or extremely important for their organization to get better at over the next year.
Gaining Control of Virtualized Server Environments
June 2011
Page 4
Server Virtualization Management Task
Perform traditional management tasks
such as troubleshooting and performance
management on a per VM basis
Keep track of VMs as they are moved
between physical servers
Dynamically move VMs, and all of the
supporting management functionality,
between physical servers
Discover VMs
Manage the traffic that goes between
VMs on a single physical server
Importance: Very or Extremely
49%
38%
37%
33%
31%
Table 3: Importance of Managing Server Virtualization
As shown in Table 3, the survey respondents indicated that getting better at each of the individual
challenges associated with server virtualization is important to their organization. In addition, it is
reasonable to look at the five challenges contained in Table 3 as being a single challenge - managing
server virtualization. When looked at that way, getting better at server virtualization is extremely
important to IT organizations.
Managing at the Virtual Machine Level
The Senior Manager commented that server virtualization is “great when it works” but that they struggle
when it doesn’t. When there is a problem caused by server virtualization, the approach that they take is
to get all of the appropriate personnel into a war room and “tear apart the problem”. He added that,
“this can take hours and can affect business”. Hollingsworth concurred and said that, “When looked at
overall, server virtualization is wonderful, but there are some legitimate issues.” He added that server
virtualization increases complexity and “makes it difficult to know what is running where and when.”
Below are some techniques and functionality that IT organizations can implement to respond to the
challenges that are associated with server virtualization.
Network Configuration at the Virtual Machine or Virtual Port Level
In the traditional data center, physical servers were typically dedicated to a single application.
Therefore, network policies could be implemented by configuring the physical port of the network
access switch with the Access Control Lists (ACLs), Quality of Service (QoS) classifications, and the
rate-limiting characteristics required for specific applications. With virtualization, multiple disparate
applications can be running on VMs sharing a single physical server. IT organizations need the ability
to manage network traffic using the same granular policies relating to security, QoS, and regulatory
compliance for applications running on VMs as it did previously for applications running on dedicated
physical servers. This can be accomplished if the network access switch is capable of mapping each
VM to a separate, individually configurable virtual switch port. The network access switch management
system can then maintain a profile for each VM, including the VM’s IP address and virtual MAC
address, VLAN memberships, ACLs and QoS classification.
Gaining Control of Virtualized Server Environments
June 2011
Page 5
The benefits of virtual ports can be maximized in those situations in which the physical network access
switch’s operating system includes support for switching all VM-to-VM traffic even when the source and
destination VMs are residing on the same physical server. With this form of virtualized switching at the
network edge, the physical access switch processes all VM traffic rather than having VM-to-VM traffic
within the same physical server being switched by a hypervisor’s software based virtual switch
(vSwitch). Virtual port profiles and edge virtualized switching are two subjects that are currently being
addressed by the IEEE 802.1Qbg working group which is expected to deliver ratified standards within
the next two years. The standards will include specifications for a Virtual Ethernet Port Aggregator
(VEPA) in external access switches and a Virtual Station Interface (VSI) Protocol to associate and deassociate VM MAC Addresses with a virtual port profile.
The way that VEPA will work is that it will offload networking tasks from the server’s vSwitch, which will
in turn maximize the server’s available CPU resources to support higher consolidation ratios. Virtual
port capability, together with VEPA, also allows existing traffic monitoring and troubleshooting tools to
gain visibility into all the traffic flows for each VM. This visibility makes it possible for IT organizations to
continue to manage the availability and performance of applications. Additional benefits of VEPA
include high performance, low latency, hardware-based switching for all VM traffic and the ability to
minimize the number of switching tiers in the data center by potentially eliminating the hypervisor and
blade server switching tiers. In heterogeneous hypervisor server environments, VEPA also eliminates
the need to support the multiple vSwitch architectures and management tools that result from using
multiple vendors, further reducing complexity. The use of VEPA can also eliminate the organizational
and economic stress that so many companies have experienced as both the network and the server
organizations claimed or disclaimed responsibility or were forced to increase staffing to manage the
multitude of vSwitches.
From the point of view of the network, VEPA functionality can be implemented on most switches as a
relatively simple firmware upgrade rather than requiring new ASICs or other hardware. This means that
data center networks can deploy pre-standard VEPA without running the risk of having to replace their
access switches when the standard is finalized. A number of vendors already support pre-standard
VEPA in their switches. However, for VEPA to work as intended, hypervisor vendors will need to
provide software support for the server side of VEPA. While this also appears to be a relatively easy
modification, the hypervisor vendors seem somewhat reluctant to embrace a pre-standard switching
solution that can bypass their software switches.
The Senior Manager stated that they are undergoing a proof of concept trial relative to using an
external switch to switch between VMs. He added that in order to reduce organizational stress that his
organization is discussing the creation of a data center group that is comprised of personnel who have
a background in networking, servers, applications, storage and load balancing.
Hollingsworth discussed some of the organizational challenges associated with server virtualization.
One example he gave is the coordination needed between the network engineering group and the
platform engineering group to make sure that vLANs are properly exposed on network equipment,
virtual switches and virtual servers. This requires a degree of coordination and synchronization of
group priorities that was not previously required. For example, it is necessary to make sure the
synchronization of vLANs is not configured just for the initial host the VM is placed on, but also for any
host where the VM might be moved. Hollingsworth added that debugging issues are also much more
challenging. For example, in a conventional network connection the network engineering group can
deploy packet capture and analysis tools between the host and the nearest switch. That is not always
possible within a virtualized server environment where communications between VMs may never leave
a physical server. In these cases, the network engineering group is dependent on the platform
engineering group either capturing and supplying the appropriate management data for analysis, or
Gaining Control of Virtualized Server Environments
June 2011
Page 6
moving the appropriate VM from one server to another so that the traffic flow is forced to route to the
physical network where network engineering can capture and analyze the data. The complexity of
these issues are leading both the platform engineering group and the network engineering group to
realize that the virtual switch provides an unnecessary layer of abstraction that doesn’t help either
engineering group.
Dynamic Policy
In addition to enabling server consolidation, another major benefit of server virtualization is derived from
the relative ease with which a VM can be provisioned or migrated among physical servers. This
enables the creation of a dynamic server environment in which the compute capacity devoted to an
application can expand and contract in real time to meet fluctuations in user demand, and virtual
servers can be easily relocated to balance load on physical server platforms or to ensure higher
application availability.
The Senior Manager stated that in order to support patch management and to ensure adequate
application performance that they move VMs between servers in the same data center on a weekly
basis. He stated that the majority of these moves are automated and that they would like to move VMs
between servers in disparate data centers but that they are constrained both by the requirement to not
cross a layer 3 network boundary as well as the difficulty associated with moving the associated
storage.
Larkins commented that they automatically move VMs between servers within their data center in
response to a server failure. They also move VMs manually in order to improve performance. He
added that he expects to more fully automate the movement of VMs in the near future as they support
more applications built for cloud computing which include APIs that enable appropriate resources to be
dynamically added or removed.
When a VM is provisioned or migrated to a new physical server, changes in the configuration of the
network are required to ensure appropriate policies are enforced for the application that is running in
the VM. VMs can be created or moved in a matter of seconds or minutes but it can take hours or even
days to implement the appropriate network configuration if manual processes are employed. The
delays associated with reconfiguring the network can result in performance shortfalls, security
vulnerabilities and gaps in regulatory compliance.
Taking full advantage of the dynamic server environment made possible by server virtualization
requires that the virtual port profile associated with a VM can be automatically created within the
network and migrated to the appropriate access switch whenever a VM is migrated among servers.
The dynamic creation and movement of port profiles can be achieved in those instances in which the
switch vendor provides the right level of integration between the hypervisor management system and
the switch element management system via APIs supported by both systems. This integration allows
the hypervisor manager to automatically notify the switch management system when a VM is created or
moved, so that the virtual port profile in the network is created or moved as required without human
intervention. Fully automated configuration of virtual port profiles not only enables a highly dynamic
server environment, but it also minimizes operational workloads, reduces the human errors that occur
during configuration and preserves the traditional responsibilities of the server and network
administrations. Deployment of multiple types of hypervisors within a single data center is becoming
increasingly common. This means that switch element management systems need to support the APIs
of each hypervisor management system employed within the data center.
Gaining Control of Virtualized Server Environments
June 2011
Page 7
Visibility
The dynamic nature of virtualized server environments can make it difficult for IT organizations to have
sufficient visibility to keep track of the status of each VM throughout its lifecycle. This lack of visibility
can result in VM Sprawl whereby obsolete or low activity VMs proliferate and waste system, storage
and network resources. This lack of visibility can also make it difficult for organizations to comply with
the large and growing list of regulatory requirements.
From a server management perspective, full visibility into a virtualized server environment requires
system management tools that can extract detailed life cycle data from the various hypervisors’
management systems. This capability enables the management system to catalog each VM in terms of
characteristics such as its time of deployment, the version of the OS it is running, its location, the
ownership of the VM, the allocated resources, the workload history and the patch status.
In a similar fashion, network managers also need a network management tool that can both gather
information about, and report on the network services provided to VMs. In those instances in which the
data center network supports virtual port profiles, much of the data required by network managers can
be collected via enhanced capabilities within the switches’ element management system. VM location
and inventory data can be stored in a network management database where it can be correlated with
both historical and real-time network performance statistics. This provides a valuable aid in
troubleshooting application performance issues and verifying the correctness of network configurations.
VM-aware network element management systems allow network managers to have direct access to the
information required to troubleshoot virtualized server networking issues without the time delays that
are typically involved when network managers must coordinate manually with the appropriate server
management personnel.
Larkins stated that driven in part by their regulatory requirements that his clients want to know where
their data is at all times. As a result, he needs detailed visibility into the creation and movement of VMs
in order to provide that insight. He added that his customers expect that he will be monitoring the
systems that he provides to them and that to do this, his company “Monitors as much as we can,
collects as much data as we can, and trends that data as much as we can.”
Gaining Control of Virtualized Server Environments
June 2011
Page 8
Summary
The traditional data center network was based on an architecture and design that was tailored to satisfy
the connectivity requirements of a relatively simple server infrastructure based on a variety of
assumptions. This included the assumption that each server was dedicated to a single application and
had a single IP address and a single MAC address and that the static nature of the application
environment lent itself to manual processes for server deployment or redeployment. While server
virtualization delivers efficiency, flexibility and agility to application provisioning and management, it has
negated the validity of most, if not all, of the assumptions that were used to design traditional data
center networks. As a result, IT organizations must develop a new architecture and design for their
data center networks.
As part of their new approach to architecting and designing data center networks, IT organizations need
to manage network traffic using the same granular policies relating to security, QoS, and regulatory
compliance for applications running on VMs as it did previously for applications running on dedicated
physical servers. This can be accomplished if the network access switch is capable of mapping each
VM to a separate, individually configurable virtual switch port. This topic is the subject of standards
currently being developed by the IEEE.
In order for IT organizations to take full advantage of the dynamic server environment made possible by
server virtualization requires that the virtual port profile associated with a VM can be automatically
created within the network and migrated to the appropriate access switch whenever a VM is migrated
among servers. The dynamic creation and movement of port profiles can be achieved in those
instances where the switch vendor provides the right level of integration between the hypervisor
management system and the switch element management system via APIs that are supported by both
systems.
Given the dynamic nature of virtualized server environments, in order for IT organizations to have full
visibility into these environments requires a system management tool that can extract detailed life cycle
data from the various hypervisors’ management systems. In a similar fashion, IT organizations also
need a network management tool that can both gather information about, and report on the network
services provided to VMs. In those instances in which the data center network supports virtual port
profiles, much of the data that is required by network managers can be collected via enhanced
capabilities within the switches’ element management system.
Gaining Control of Virtualized Server Environments
June 2011
Page 9
About the Author – Jim Metzler
Jim has a broad background in the IT industry. This includes being a software engineer, an engineering
manager for high-speed data services for a major network service provider, a product manager for
network hardware, a network manager at two Fortune 500 companies, and the principal of a consulting
organization. In addition, Jim has created software tools for designing customer networks for a major
network service provider and directed and performed market research at a major industry analyst firm.
Jim’s current interests include cloud networking and application delivery.
About the Webtorials® Editorial/Analyst Division
The Webtorials® Editorial/Analyst Division, a joint venture of industry veterans Steven Taylor and Jim
Metzler, is devoted to performing in-depth analysis and research in focused areas such as Metro Ethernet
and MPLS, as well as in areas that cross the traditional functional boundaries of IT, such as Unified
Communications and Application Delivery. The Editorial/Analyst Division’s focus is on providing
actionable insight through custom research with a forward looking viewpoint. Through reports that
examine industry dynamics from both a demand and a supply perspective, the firm educates the
marketplace both on emerging trends and the role that IT products, services and processes play in
responding to those trends.
®
For more information and for additional Webtorials Editorial/Analyst Division products, please contact
Jim Metzler at [email protected] or Steven Taylor at [email protected].
Published by Webtorials
Editorial/Analyst
Division
www.Webtorials.com
Division Cofounders:
Jim Metzler
[email protected]
Steven Taylor
[email protected]
Professional Opinions Disclaimer
All information presented and opinions expressed in this publication represent the
current opinions of the author(s) based on professional judgment and best available
information at the time of the presentation. Consequently, the information is subject
to change, and no liability for advice presented is assumed. Ultimate responsibility
for choice of appropriate solutions remains with the reader.
Copyright © 2011 Webtorials
For editorial and sponsorship information, contact Jim Metzler or Steven Taylor.
The Webtorials Editorial/Analyst Division is an analyst and consulting joint venture
of Steven Taylor and Jim Metzler.
Gaining Control of Virtualized Server Environments
June 2011
Page 10