Cisco IT Case Study – July 2012
Enterprise Messaging on Cisco UCS
How Cisco IT Deployed Enterprise Messaging on
Cisco UCS
Messaging platform upgrade and new servers reduce costs and improve
management, availability, and disaster recovery.
Challenge
EXECUTIVE SUMMARY
CHALLENGE
● Upgrade to a new messaging platform and
new server infrastructure at the same time
SOLUTION
● Microsoft Exchange 2010 messaging platform
on Cisco UCS® servers
RESULTS
● Consolidated infrastructure with 60% fewer
servers, located in three data centers instead
of five
● Larger mailbox storage and improved user
experience
● Improved service resilience and disaster
recovery
LESSONS LEARNED
● Analyze network capacity and evaluate
multiple hardware configurations
● Test custom applications
● Identify special needs for user
communications
NEXT STEPS
● Server virtualization to match actual
operational needs
In 2005, Cisco IT chose the Microsoft Exchange 2003 platform as
the corporate standard for email, calendaring, and contacts software.
By 2011, the need to upgrade to Microsoft Exchange 2010 was
indicated by:
●
User requests for more mailbox space
●
The need to provide a better experience for Apple Macintosh
and mobile device users
●
The end of Microsoft’s end support for Exchange 2003
An aging server infrastructure was also a factor. “By this time we'd
gone so long without an upgrade, the server infrastructure was
getting quite old and we were facing a higher risk of hardware
failures that could cause unacceptable service outages,” says Ken
Pauley, senior IT manager for communications and collaborations IT
at Cisco. “So we made the decision to upgrade to Exchange 2010
and replace the entire underlying server infrastructure at the same
time.”
Solution
For Exchange 2003, Cisco IT deployed three separate infrastructures, including one for employee user accounts
and one for vendor and external user accounts. The third infrastructure, for voicemail accounts, was replaced in
2011 by the Cisco Unity® Connection on Cisco UCS deployment. The 220 servers and associated storage
systems for these infrastructures were housed in five locations around the world.
The previous infrastructures were based on a modular pod design of servers and storage systems. Servers were
installed in racks and each rack had a dedicated storage pod, with an additional SAN associated with the rack for
backup purposes. Although simple to support and easily scaled by adding a new pod, this design was also very
expensive. Adding even just a small number of users required installing a complete pod and all of its associated
connectivity and other infrastructure.
© 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.
Page 1 of 6
For Microsoft Exchange 2010, Cisco has consolidated all user accounts onto a single, high-availability
infrastructure that is based on 113 Cisco Unified Computing System™ (Cisco UCS®) physical servers—a design
reflecting the Cisco IT service-oriented architecture for data center virtualization.
Replacing these servers allowed the IT team to define a standard hardware and software configuration and fully
implement and test the servers before starting the messaging platform upgrade. Although the upgrade could have
been installed on virtualized servers from the start, the team decided to gather operational data in order to
determine the right plan for server virtualization in the future.
The Cisco UCS servers and associated storage systems are configured into multiple clusters at three sites in the
United States. The primary clusters (with 30 servers each) are deployed in the Cisco data centers in Richardson
and Allen, Texas. These centers are connected by a Metro Virtual Data Center (MVDC) active-active design, which
provides for complete, real-time replication and failover between the clusters. The third site, in Research Triangle
Park, North Carolina, operates a 27-server cluster as a backup for disaster recovery if both Texas data centers
become unavailable.
As shown in Figure 1, this design allows for failover at both the hardware and application levels. The server layer
handles application failover because each host has eight active and eight passive databases. For example, if an
active database fails in Richardson, failover is automatic to the associated passive database in Allen. If the
application must fail-over to the disaster recovery site in Research Triangle Park, user accounts are back online
within seconds to support business continuity.
Load balancing of all client traffic across the two primary clusters is managed by Cisco Application Control Engine
(ACE) GSS 4400 Series Global Site Selector Appliances, which also handle failover to the third cluster. To manage
these elements, Cisco IT uses internal and vendor tools for the Cisco products, the messaging platform, and the
storage systems.
© 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.
Page 2 of 6
Figure 1.
Cluster Design for Cisco UCS Servers that Host the Messaging Platform
Cisco Exchange 2010
Applications
Cisco ACE GSS
Mobile
Devices
Firewall
Clients
VPN
Prod
uctio
n
tion
duc
Pr o
Production – Texas
Richardson DC
Dis
ast
er R
eco
ve r
y
DR – North Carolina
Allen DC
ACE – Round Robin
Research Triangle Park DC
ACE – Round Robin
ACE – Round Robin
AD Site – Exchange HA
AD Site – Exchange DR
CAS Array
Client Access Server
Hub Transport
Mailbox Copy 1
CAS Array
Client Access Server
Client Access Server
Hub Transport
Mailbox Copy 2
Hub Transport
Database Availability Group
(DAG)
Mailbox Copy 3
Storage of email messages, calendars, and contacts data is another significant element of this design. “With 5.8
petabytes of data storage allocated, the enterprise messaging infrastructure represents the single largest storage
deployment within Cisco and 10 percent of all storage deployed by the company,” says Pauley. “It is also 10 times
more than the amount of storage in the old infrastructure, reflecting our users’ needs to store more messages for a
longer time and Cisco IT's need to provide backup storage.” Table 1 shows key statistics for the old and new
messaging infrastructures.
Table 1.
Key Statistics about the Cisco IT Enterprise Messaging Infrastructures
Exchange 2003 Infrastructure at Cisco






Five sites worldwide
220 servers, 40 rack spaces
10% average CPU utilization
0.44 PB storage
112,000 user mailboxes, 400 MB each
No high-availability or disaster recovery
configuration
Exchange 2010 Infrastructure at Cisco






Three sites in the United States
113 servers, 30 rack spaces
Target of 35% average CPU utilization in normal
operation and 70% utilization in a highavailability/disaster recovery scenario
5.8 PB storage
120,000 user mailboxes, 2 GB each
Configuration supports high availability and disaster
recovery
Upgrade Process
“We made the decision to upgrade all users to the new messaging platform within the same timeframe instead of
doing a phased upgrade by location,” says Pauley. “However, this meant the complete infrastructure needed to be
© 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.
Page 3 of 6
up and running from day one, with all servers and storage systems deployed and
PRODUCT LIST
Data Center and Virtualization
● Cisco Unified Computing System™
ready to go at the same time.”
Before beginning the actual upgrade, Cisco IT used a small proof-of-concept project
to assess the planned upgrade process, its potential impact on users, and needs for
● Cisco® ACE Application Control Engine
user communication. This project included a small number of Cisco executives in
Borderless Networks
● Cisco Wide-Area Application Services
Software
● Cisco ACE GSS 4400 Series Global Site
Selector Appliances
order to verify access and user experience for calendars that are jointly managed
with their administrative assistants.
For the actual upgrade, the Cisco IT team used a scheduling tool to designate
upgrade dates for defined groups of users, automatically send e-mail and voicemail
notices about the planned move, then perform the actual upgrade of the user’s mailbox from the old to the new
platform.
The upgrade was initially installed on non-virtualized Cisco UCS servers in order to maintain high levels of
performance for the messaging applications. “I'd rather over-deploy servers and scale back than under-deploy and
have performance issues,” says Pauley.
User Features
Today Cisco users enjoy a common experience in the messaging applications, whether accessed from a Windows,
Macintosh, or Linux-based client or a web browser interface. The enterprise messaging applications are also
integrated with other popular tools such as Cisco TelePresence and the internal Cisco Integrated Workforce
Experience (IWE) social sharing site.
With 55,000 mobile devices already authorized on the Cisco network, users increasingly expect to access and
synchronize their e-mail, calendars, and contacts from their smartphone or tablet computer. Given the growing
number of mobile devices used by Cisco employees, the deployment team planned the new messaging
infrastructure to support access from three devices per user.
Results
With the mid-2012 completion of the upgrade to the messaging platform and its server infrastructure, Cisco is
realizing results in several areas.
Simpler infrastructure. Migrating to the Cisco UCS servers allows for a simpler infrastructure design, with 60
percent fewer servers and fewer cluster locations. This design reduces capital and operating costs, and allows for
improved resilience and disaster recovery.
“We were able to achieve this level of reduction because of the performance and integrated protocol cable design
of the Cisco UCS servers,” says John Kerruish, Cisco IT architect. “We were able to configure more users per
server, which meant we could use fewer servers even before we looked at virtualizing them.”
Rapid deployment. Cisco IT completed the upgrade of 88,000 user mailboxes in just six weeks. “If we had kept
the previous cluster design with multiple locations around the world, it would've taken a year to go to all of the
locations and finish the upgrade in a phased schedule,” says Pauley.
Simpler client interface for Macintosh users. Cisco IT now supports two native Macintosh email clients,
replacing the 45 different combinations of tools used previously.
© 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.
Page 4 of 6
More message storage. Users now enjoy a standard mailbox size of 2 GB, compared to the previous limit of 400
MB. To control storage costs, Cisco IT can easily reduce or increase mailbox sizes based on actual usage levels
and changes to message retention periods.
Improved recovery from cluster failures. The previous design did not allow for complete replication of user
accounts and data. In a cluster failure, Cisco IT was able to recover users’ mailboxes, but not their email
messages. Today, Cisco IT can perform full replication of all user messaging data, with automatic failover between
the two primary clusters. The disaster recovery site provides an additional cluster for recovering user accounts
only.
Lessons Learned
Analyze network capacity. Configuring a messaging platform for a large enterprise in a high-availability failover
design requires significant network bandwidth between the data centers. Verify adequate capacity to support both
high-volume traffic from remote users and replication traffic.
During periods of high network congestion, the bandwidth between the primary clusters can be configured to give
priority to real-time user traffic and allow the replication traffic to absorb any latency.
Evaluate multiple hardware configurations. Take the time to test various configurations of physical and
virtualized servers to identify the best hardware configuration for the specific messaging platform. It is also
important to allow sufficient time to accurately configure the storage systems for user messages and related data.
Allow time for testing custom applications. Cisco IT operates a lab that allowed internal developers to test their
specialized applications and also allowed the deployment team to perform functional and load testing before
finalizing the server configurations.
With this lab testing, Cisco IT was able to avoid conducting a pilot project, which accelerated the production
schedule.
Identify special needs for user communication. Cisco IT created special communications for Macintosh users
because they would see the most change in their access and experience with the new enterprise messaging
system.
Next Steps
The upgrade to the new enterprise messaging infrastructure was complete as of mid-2012. After monitoring
operations and utilization levels, Cisco IT will begin planning for partial server virtualization to further reduce costs.
Cisco IT will also plan upgrading the Wide-Area Application Services (WAAS) software to accelerate email traffic
for Windows devices, which use the native, encrypted Messaging Application Programming Interface (MAPI) to
retrieve messages.
© 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.
Page 5 of 6
For More Information
Cisco Unified Computing System: www.cisco.com/go/ucs
Cisco Application Control Engine: www.cisco.com/go/ace
To read additional Cisco IT case studies on a variety of business solutions, visit Cisco on Cisco: Inside Cisco IT:
www.cisco.com/go/ciscoit
Note
This publication describes how Cisco has benefited from the deployment of its own products. Many factors may
have contributed to the results and benefits described; Cisco does not guarantee comparable results elsewhere.
CISCO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
Some jurisdictions do not allow disclaimer of express or implied warranties, therefore this disclaimer may not apply
to you.
© 2012 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public.
Page 6 of 6