Over 45 miles of InfiniBand® double data rate cabling is required to connect Pleiades 11,776 nodes.
Inside a NASA Production
Supercomputing Center
It takes a super workload management tool
to power grid, cluster and on-demand computing
environments for computational modeling and
simulation applications at NASA.
By Cathleen Lambertson
Photo courtesy of NASA Ames Research Center
I
n the past, scientific and engineering advancements relied
primarily on theoretical studies and physical experiments.
Today, however, computational modeling and simulation
are equally valuable in such endeavors, especially for an
agency such a s t he National Aeronautic s a nd Space
Administration (NASA). With a mission “to pioneer the future
in space exploration, scientific discovery and aeronautics
research,” the use of high-end computing (HEC) for high-fidelity modeling and simulation has become integral to all four of
NASA’s mission directorates: aeronautics research, exploration
systems, science and space operations.
These HEC resources are provided at NASA’s Advanced
Supercomputing (NAS) Division at Ames Research Center,
M o f f e t t F i e l d , C a l i f. N A S o f f e r s p r o d u c t i o n a n d
development systems to U.S. scientists in government, industry
and at universities, with users currently numbering over 1,500.
Projects such as designing safe and efficient space exploration
vehicles, projecting the impact of human activity on weather
patterns and simulating space shuttle launches are studied using
the facility’s supercomputers. “We provide world-class HEC and
associated services to enable NASA scientists and engineers in
all mission directorates to broadly and productively employ largescale modeling, simulation and analysis for mission impact. We
pursue a future where these services empower ever greater NASA
mission successes,” says William Thigpen, the HEC capability
deputy project manager at the NAS Division.
The facility’s current HEC systems include four supercomputers, a 30-petabyte mass storage system for long-term data storage,
Reprinted from Concept To Reality magazine’s 2011 Summer/Fall issue.
An biannual publication
Free subscriptions at www.altair.com/c2r
two secure front-end systems requiring two-factor authentication
and two secure unattended proxy systems for remote operations.
Key system resources at NAS include: Pleiades, a 112,896-core,
1.34 petaFLOPS (Pflop/s) SGI® Altix® ICE cluster; Columbia, a
4,608-processor SGI Altix® (Itanium 2); Schirra, a 640-processor
IBM® Power5+; and hyperwall-2, a 1,024-core, 128-node GPU
cluster.
Since 300 to 400 jobs are typically running 24 hours a day,
seven days a week, the NAS staff works nonstop to meet the
demands for time on these machines. “Our mission is to accelerate and enhance NASA’s mission of space exploration,
scientific discovery and aeronautics research by continually
creating and ensuring optimal use of the most productive HEC
environment in the world,” says Thigpen. “Our viewpoint is
that we spend a lot of money getting hardware in here, but it
really makes sense that it is effectively exploited by our users
because the bottom line is we’re not about big hardware, we’re
about big science and engineering.”
Building a Supercomputer – Pleiades
Originally installed in the fall of 2008, the Pleiades supercomputer is an SGI Altix ICE 8200/8400 InfiniBand® cluster with
Intel® Xeon® quad and hex-core processors, running at 1.09
Pflop/s on the LINPACK benchmark – the industry standard for
measuring a system’s floating-point computing power. Considered
one of the most powerful general-purpose supercomputers ever
built, each of the Pleiades 185 racks (11,776 nodes) has 16
InfiniBand switches to provide the 11D dual-plane hypercube
that provides the interconnect for the cluster. The InfiniBand
fabric interconnecting Pleiades’ 11,776 nodes requires more than
45 miles of cabling. Pleiades is the largest (measured by number of
nodes) InfiniBand cluster in the world (see sidebar for full system
specifications).
Currently ranked seventh on the TOP500 list of the world’s
most powerful computers, the Pleiades supercomputer was built to
augment NASA’s current and future high-end computing requirements. “Pleiades is a general-purpose machine and provides for all
three components of supercomputers – [capability, capacity and
time critical],” says Thigpen. “We have users running jobs using
over 18,000 cores, providing new insights into the formation of
the universe. There are numerous users running parameter studies (often thousands) using from one to a few thousand cores.
Pleiades is also being utilized to answer time-critical questions
concerning the shuttle.”
The most recent upgrade to Pleiades occurred in May 2010
and included the incorporation of 32 SGI ICE 8400 racks with
the Intel Xeon X5670 (Westmere) processor, which effectively
doubled the science and engineering compute capability to the
NAS facility’s general user community. “Placing these racks into
service involves the connection of over 4,400 cables. While 1,024
Concept To Reality Summer/Fall 2011
were done in the factory, the rest had to be done on our floor,”
notes Thigpen. “We then performed an extensive series of tests at
first focused on the new racks and then on the total system. There
are no other systems with this many nodes connected via
InfiniBand in the world, and so we encounter and resolve issues
that haven’t been seen elsewhere.”
According to Alan Powers, the HEC technical director for
Computer Science Corp. at NAS, the goal of the upgrade was to
have an up-and-running production machine after the integration. “If you look at most other supercomputing sites, it is hard for
them to do this type of integration in their environment. We cut
in these racks in two weeks, and now they are in production. We
added 32 racks, which is considered a very large machine to begin
with at most sites, but we just think of it as part of our system.”
Choosing Components and Software
Pleiades was built to meet as many of the emerging NASA science and engineering mission requirements as possible while
remaining within the HEC budget. “The Pleiades architecture was
chosen because it provided the best performance/cost ratio of the
systems we looked at. Since its original installation in 2008, it has
undergone two expansions. We will continue to build it out as long
as the fundamental economics of the system remain sound, and the
science and engineering returns remain high,” states Thigpen.
To build Pleiades, NAS engineers began with the components
recommended by the vendor and those being used on other
systems. The result has been an easy transition to the new
environment for NASA users. “We want an environment where
the components complement each other, are an easy natural
transition for our users and provide a reliable environment,” says
Thigpen. For example, the SGI ICE 8200 and 8400 are standard
products that have been taken to an extreme size at the NAS
facility. Additionally, the InfiniBand network was expanded to
incorporate both the data analysis and visualization cluster, as
well as the storage system.
Another consideration is outlining and selecting a scalable
architecture. Powers explains, “We chose SGI because it had a
certain architecture that allowed us to build and grow it. [It also
had] the best price/performance based on our workload. Where
we are today, we’re near a petaflop capability, and it’s been built
over a couple of years; we’ve been adding to it slowly. The other
vendors’ price/performance wasn’t even close to this platform.”
Managing the Workload
When providing supercomputing resources to 1,500 users,
24/7, workload management is a top priority. Originally developed at NAS in the 1990s and then commercialized, PBS
Professional® workload management software has been used
since its inception. Commercially developed by Altair
Engineering, Inc., Troy, Mich., the PBS platform is designed to
T R E N D S
I N
T E C H N O L O G Y
The Pleiades – System
Specifics
• 185 racks (11,776 nodes)
Subsystems
• 1.34 Pflop/s peak cluster
• 14 front-end nodes
• 1.09 Pflop/s LINPACK rating (June 2011)
• 4 bridge nodes
• Total cores: 112,896
• 2 PBS servers
• Total memory: 191 TB
Interconnects
• Nodes
• Internode – InfiniBand, with all nodes connected in
a partial 11D hypercube topology
 2 quad-core processors per node
• Two independent InfiniBand fabrics
 Xeon E5472 (Harpertown) processors
• InfiniBand DDR, QDR
 Processor speed – 3 GHz
• Gigabit Ethernet management network
 Cache – 6 MB per pair of cores
Storage
 Memory Type - DDR2 FB-DIMMs
• SGI® InfiniteStorage NEXIS 9000 home filesystem
 1 GB per core, 8 GB per node
• 12 DDN RAIDs, 6.9 PB total
 1,280 Nehalem nodes
 2 quad-core processors per node
 Xeon X5570 (Nehalem) processors
• 6 Oracle Lustre cluster-wide filesystems
Operating Environment
• Operating system – SUSE® Linux®
 Processor speed - 2.93 GHz
• Job Scheduler – PBS Professional®
 Cache – 8 MB Intel® Smart Cache for 4
cores
• Compilers – Intel and GNU C, C++ and Fortran
 Memory Type - DDR3 FB-DIMMs
Computational fluid dynamics (CFD) calculation of the flow
around the Orion crew module, with wind tunnel sting.
 3 GB per core, 24 GB per node
• MPI – SGI MPT, MVAPICH2, Intel MPI
 4,608 Westmere nodes
 2 six-core processors per node
 Xeon X5670 (Westmere) processors
 Processor speed – 2.93GHz
 Cache – 12 MB Intel® Smart Cache for 6 cores
 Memory Type – DDR3 FB-DIMMs
 2 GB per core, 24 GB per node
power grid, cluster and on-demand computing environments.
PBS Professional is used to manage all HEC resources at NAS,
including Pleiades. “We have found Altair continues to be
receptive to enhancements that NAS needs to manage unique
systems. The choice for Pleiades was a natural progression
from years of a very effective working relationship with
Altair,” states Thigpen.
PBS Professional is a resource allocation tool that makes it
possible to create intelligent policies to manage distributed,
Photo courtesy of NASA Exploration Systems Mission
Directorate, Principal Investigator, Joseph Olejniczak
 5,824 Harpertown nodes
mixed-vendor computing assets as a single, unified system.
Based on a policy-driven architecture, it continually optimizes
how technical HEC resources are used, ensuring that they are
used effectively and efficiently. Simply put, the software looks at
the jobs that want to run, looks at the resources available for
them to run on and makes the best match based on a number of
criteria. “Those criteria can include the user that’s running and
how many jobs that user currently has running, or how many
cores his job is currently using. It can also be the queue that a
www.altair.com/c2r
I N
T E C H N O L O G Y
Photo courtesy of NASA Ames Research Center
T R E N D S
The Pleiades
supercomputer
at the NASA
Advanced
Supercomputing
(NAS) facility
at NASA Ames
Research Center.
user
submit s
t hei r jo b i n, a n d
those queues can have
things limiting them, like how
many jobs are running or how
many cores all of the jobs together are
using. It also can be the mission directorate
those users are in,” explains Thigpen.
Powers adds: “The ‘P’ in PBS stands for ‘portable,’
and it allows us to run this on any architecture. We’ve had
PBS on fat node architectures, on thin node clients and on IBM
architectures. PBS has been able to adapt to all those computing environments. This has allowed our users to have a
consistent set of batch scripts across these different environments. They only have to learn one thing. So one, it’s flexible;
two, we can use it on any architecture; and three, it’s easy for
users to learn.”
relatively small amount on a
system, they can run through
hundreds or thousands of alternatives before building a physical
prototype. This will allow for a better
product with lower production costs.”
However, there are many issues to address when
considering whether an HEC environment is the right
choice for an enterprise. “There has to be a balance between
the cost of the resources, the technology they enable, the
increased productivity of their staff, the potential return on their
investment and what their competition is doing,” Thigpen
concludes.
Cathleen Lambertson is Contributing Editor, Tech Briefs Media
Group.
Your Own HEC Environment
According to Thigpen, HEC is an enabling technology that
allows a company to build products that can meet their customers’ requirements in a cost-effective manner: “By spending a
Concept To Reality Summer/Fall 2011
To receive more information about
PBS Professional, visit www.pbsworks.com.
www.altair.com/c2r