Intel® Solutions for Lustre* Reseller - Elite
* Other names and brands may be claimed as the property of others.
Atipa Solutions for Lustre
Performance, Density, Availability
Introduction
Data-intensive HPC applications pose strong demands on storage systems to feed large volumes of data into
complex simulations and multi-dimensional modeling. Traditional file systems used to share data between
servers, such as the Network File System (NFS), were not designed with large numbers of clients running I/Ointensive applications in mind. These file systems are not scalable and therefore quickly become a bottleneck that can grind an otherwise optimally functioning cluster to a halt. Distributed file systems, such as
Lustre, on the other hand, spread the I/O workload over many servers and scale in both capacity and
performance to fit the needs of the most data-intensive computational environments.
Figure 1. Traditional network file systems vs. Lustre.
Lustre
Lustre is the most widely adopted parallel file system for supercomputing, powering over 70% of the Top100
supercomputers in the world. The file system has been in use in production environments under the most
demanding circumstances for more than a decade and has evolved into a mature, highly trusted, reliable file
system for distributed computing. Production systems are running with up to 55PB of storage and achieving
1
Atipa Technologies
throughput rates of up to 2.5TB/s. A commercial Lustre distribution is now also available as Intel® Enterprise
Edition for Lustre (IEEL).
A Lustre storage system is made up of multiple components, each of which has a well-defined role in the
operation of the file system. The three main components are the Management Server (MGS), the Metadata
Servers (MDS), and the Object Storage Servers (OSS). The Management Server and Metadata Servers are the
brains of the file system, keeping track of file system configuration and file locations, while the Object
Storage Servers are the workhorses moving bulk data between the backend storage and the clients.
The Management Server, Metadata Servers, and Object Storage
Servers each have their own backend block storage, called
targets, designed to match their specific duties. The
Management Target (MGT) stores file system information, the
Metadata Targets (MDT) store file attributes, and the Object
Storage Targets (OST) store actual file data. The metadata
contains information such as filenames, directories,
permissions, and location of files on the object storage servers.
When a client needs to access a file, it requests the location of
the file on the OSTs from the MDS. Once the file is opened, data
is transferred directly between the client and the OSSs. The
MDS itself does not participate in any I/O to and from the
file. The OSTs may be thought of as virtual disks. Lustre
aggregates the performance of these virtual disks by striping
files over multiple OSTs. Data transfer between clients and
storage servers typically takes place over high-performance
fabrics such as Infiniband or Intel® Omni-Path.
Figure 2. Lustre file system data flow.
OpenZFS
Atipa Solutions for Lustre are built on the OpenZFS file system.
OpenZFS distinguishes itself from other file systems by its built-in volume management and robust data
integrity. OpenZFS also offers fast and efficient software RAID schemes equivalent to RAID-1, RAID-5, and
RAID-6. The Copy-on-Write (COW) transactional model ensures data is always consistent on disk. This means
that when data is changed, it is not overwritten, but instead it is written to a new block and checksummed
before pointers to the most recent copy of the data are changed. User data is protected against silent data
corruption by 256-bit checksums which are stored separate from the data. When a data block is read,
OpenZFS calculates its checksum and compares it to the stored checksum. If the block is corrupted, it is
repaired transparently and on the fly.
OpenZFS key features






2
End-to-end data integrity
Always consistent on disk (COW)
Pooled data storage
256ZiB storage pools
Mirror, RAID-Z, RAID-Z2
Hybrid SSD/HDD storage pools






Snapshots
Data scrubbing
Compression
Deduplication
Online capacity expansion
Dynamic striping
Atipa Technologies
Atipa Solutions for Lustre
As an Intel® Solutions for Lustre – Elite partner, Atipa designs, builds, deploys, and supports affordable High
Performance Storage solutions based on Intel® Foundation Edition for Lustre and Intel® Enterprise Edition for
Lustre Software:

Intel® Enterprise Edition for Lustre* Software. The most trusted, reliable and easy to use version of
Lustre* anywhere. Intel® Enterprise Edition for Lustre* Software couples the performance and
scalability of the fastest file system in the world, with advanced storage management tools for
enterprises needing scale-out storage to fuel their technical applications or Hadoop* MapReduce*
workloads.

Intel® Foundation Edition for Lustre* Software. For those who must have the very latest cutting
edge version of Lustre*, you can now have it with the confidence of Intel® support and a
maintenance release stream to ensure the highest reliability possible. It is ideal for organizations
who prefer to design and deploy their own community configurations, and want to augment inhouse management and support capabilities with Atipa and Intel® expertise.
Compare Editions
Intel® Foundation
Edition for Lustre*
Software
Global 24/7 technical support
√
√
√
√
√
√
√
√
Developed and supported by Intel®, the largest contributor to
Lustre community releases
√
√
Intel® backed support for debugging and patches
√
√
Stable enterprise platform, built for reliability and performance –
with long term support
Intel® Manager for Lustre software for simplified installation and
configuration
√
√
Centralized, GUI-based administration for easy management
√
Integrated support for Hadoop* MapReduce* applications with
Lustre storage
Open, documented application program interfaces for deep
integration
√
Enormous storage capacity and I/O
Rigorously tested stable software
Resilient, highly available storage configurations
3
Intel® Enterprise
Edition for Lustre*
Software
√
√
Atipa Technologies
Atipa offers both entry-level and enterprise-level turn-key Lustre solutions at affordable prices for use in
Academia, Government, and Industry. Our High-Performance Storage solutions are designed from the ground
up to harness the power of the Lustre parallel file system on top of the resiliency and data integrity of
OpenZFS. The LZ-60v2 is our entry-level Lustre solution targeted towards users who need a fast, affordable
storage solution without all the bells and whistles of an enterprise appliance. The LZE-576v2, on the other
hand, is our flagship enterprise appliance equipped with high-availability features and 24/7 support. In
addition to our reference architectures, we also offer flexible custom storage solutions designed to fit your
needs.
Atipa partners with Intel® to offer world class 24/7 Lustre support around the world for enterprise appliances.
Atipa engineers are the central point of contact for all support tickets and take the lead in Level 0, Level 1,
and Level 2 support issues. This includes, but is not limited to, assisting in collecting diagnostics information,
reproducing the problem, and eliminating hardware faults. If needed, support tickets are escalated to Intel ®
for Level 3 support (code fixes) with a severity representing the operational impact of the reported issue.
Severity 1 issues (file system is down) trigger a broadcast to the entire Intel® High Performance Data Division
(HPDD) team and have a response time of less than 4 hours.
Atipa added value
Intel® Enterprise
Edition for Lustre*
Software
Intel® Foundation
Edition for Lustre*
Software
Single-point of contact for all support tickets by Atipa
√
√
√
√
√
√
√
√
√
√
√
√
√
√
Silver or gold 3-year warranty and optional 4th and 5th year
warranty by Atipa
√
√
Optional firmware and software upgrade services by Atipa
√
√
Canned solutions and custom configurations by Atipa
Robust hardware burned in and configured by Atipa
Lustre pre-installed and configured by Atipa
Performance validated by Atipa
On-site installation and training by Atipa
Level 0, 1, 2 support by Atipa
Entry-Level Atipa Lustre Solutions
The LZ-60v2 is Atipa’s entry level Lustre appliance for users in need of
fast scratch space with small to medium capacity or for users looking
for a proof-of-concept that Lustre will accelerate their workload. The
solution consists of one combined metadata/management server and
three dedicated object storage servers with a total storage
capacity of up to 60TB in only 4U of rack space. The system can
sustain a sequential write throughput of over 9GB/s and a
sequential read throughput of over 4GB/s. Storage capacity can be
expanded with the LZ60v2-ext expansion module, which increases the
available storage up to 140TB1 and adds an additional 4U of rack
space to the solution. Performance can be increased nearly linearly by
adding additional LZ-60v2 modules.
4
Atipa Technologies
Key features






Purpose built for fast, no-hassle,
scale-out storage
Turn-key solution
Sequential write > 9GB/s
Sequential read > 4GB/s
Single global namespace
POSIX compliant




Fully tested and configured before
shipping
On-site integration by experienced
engineers
Based on Intel® Foundation Edition
for Lustre
Quality support by Atipa and Intel®
LZ-60v2 Scalable Lustre System Modules
Lustre Server Module
Storage Expansion Module
SKU
LZ-60v2
Lz60v2-ext
Servers and Storage Enclosures
One combined MDS/MGS node and three
OSS nodes
24-bay hot-swappable JBOD enclosure
Server Processors
Dual Intel® Xeon E5-2600v4
N/A
Server Memory
Up to 512GB
N/A
Server Network
Single-port FDR Infiniband, EDR Infiniband,
Intel® Omni-Path, or 10GbE per node
N/A
Server/Storage Interconnect
12Gb/s SAS-3
12Gb/s SAS-3
Sequential Read/Write
Throughput
Read: 4GB/s (FDR and Omni-Path)
Write: 9GB/s (FDR and Omni-Path)
Up to 1.2TB usable for MDT and up to 60TB
usable for OSTs1
4U
CentOS Linux or Red Hat Enterprise Linux
Intel® Foundation Edition for Lustre
ZFS
Storage Capacity
Rackspace
Operating System
Lustre Edition
Lustre Backend File System
N/A
Up to 80TB additional usable for OSTs1
4U
N/A
N/A
ZFS
Enterprise-Level Atipa Lustre Solutions
The LZE-576v2 is Atipa’s 2nd generation flagship Lustre appliance for enterprises. The starting configuration
consists of one Metadata/Management Module and one Object Storage Module. The Metadata/Management
Module contains one metadata server, one management server, one metadata target, and one management
target. The Object Storage Module contains two object storage servers and eight object storage targets. The
metadata/management servers and the object storage servers in each module are configured as active/active
failover pairs. Hence, when the LZE-576v2 is fully functional, all servers participate in serving data at
optimal performance. When a server in a failover pair fails, it’s companion transparently takes over its duties
and I/O to and from the Lustre clients continues uninterrupted, albeit with reduced performance.
1
5
Based on 4TB hard disk sizes, larger capacities possible with 6TB or 8TB hard disk drives.
Atipa Technologies
Key features






Turn-key solution
High-availability configuration
Sequential write > 15GB/s
Sequential read > 17GB/s
576TB usable storage per 8U rack space
Enterprise-class Reliability, Availability,
and Serviceability




On-site integration by
experienced engineers
Based on Intel Enterprise Edition
for Lustre
Real-time monitoring with Intel
Manager for Lustre
24/7 support by Atipa and Intel®
The starting LZE-576v2 configuration provides 4.4TB of
metadata storage and 576TB of usable Lustre storage in 12U
of rack space. The system connects to a computational
cluster or data center infrastructure through two networks: a
gigabit Ethernet network for management and health
monitoring and a high-speed fabric such as Infiniband or
Intel® Omni-Path for data transfer. When connected to an
FDR Infiniband fabric, the LZE-576v2 sustains sequential
read throughputs of 10GB/s and sequential write
throughputs of 11GB/s. However, the LZE-576v2 truly shines
with a high-performance Intel® Omni-Path interconnect,
sustaining sequential read and write throughputs of 17GB/s
and 15GB/s, respectively. The modular architecture furthermore provides an easy way to scale both capacity
and performance by adding additional Metadata/Management Modules and/or Object Storage Modules.
6
Atipa Technologies
LZE-576v2 Scalable Enterprise-Level Lustre System Modules
Metadata/Management Module
Object Storage Module
SKU
LZE-576v2-MMM
LZE-576v2-OSM
Servers
Active/active MDS/MGS failover pair
Active/active OSS failover pair
Server Processors
Dual Intel Xeon E5-2600v4
Dual Intel® Xeon E5-2600v4
Server Memory
Up to 512GB
Up to 512GB
Server Network
Single-port FDR Infiniband, EDR Infiniband,
Intel® Omni-Path, or 10GbE per node
Single-port FDR Infiniband, EDR Infiniband,
Intel® Omni-Path, or 10GbE per node
Server OS Drives
Dual hot-swappable mirrored HDDs or SSDs
Dual hot-swappable mirrored HDDs or SSDs
Server/Storage Interconnect
Storage Enclosures
Sequential Read/Write
Throughput
Storage Capacity
Storage Hot Spares
Rackspace
Operating System
Configured for High-Availability
Lustre Edition
Lustre Backend File System
®
12Gb/s SAS-3 Host Bus Adapter or 12Gb/s
SAS-3 RAID HA controller
24-bay hot-swappable drive enclosure with
dual redundant expander boards, hotswappable power supplies, and hotswappable fan modules
Up to 4.4TB usable
2
4U
Red Hat Enterprise Linux or CentOS Linux
Yes
Intel® Enterprise Edition for Lustre
ZFS or ldiskfs
12Gb/s SAS-3 Host Bus Adapters
90-bay hot-swappable drive enclosure with
dual redundant, hot-swappable expander
modules, hot-swappable power supplies, and
hot-swappable fan modules
FDR: Read >10GB/s, Write >11GB/s
OPA: Read > 17GB/s, Write > 15GB/s
Up to 576TB usable
2
8U
Red Hat Enterprise Linux or CentOS Linux
Yes
Intel® Enterprise Edition for Lustre
ZFS
The LZE-576v2 is powered by Intel® Enterprise Edition
(EE) for Lustre, designed to enable fully parallel I/O
throughput across thousands of clients, servers, and
storage devices. Intel® EE also includes Intel® Manager for
Lustre (IML), a management solution purpose-built by the
Lustre experts at Intel® for the Lustre file system. Intel®
Manager for Lustre* software reduces management
complexity and costs, enabling storage superusers to
exploit the performance and scalability of Lustre storage,
and accelerate critical applications and work flows. Intel®
Manager for Lustre* software greatly simplifies the
management of the LZE-576v2 Lustre file systems, using
an easy-to-use browser-accessible graphical user interface
(GUI) or a command line interface (CLI). The GUI
dashboard lets you monitor one or more distributed Lustre
file systems. Real-time storage-monitoring lets you track Lustre file system usage, performance metrics,
events, and errors at the Lustre level. Fully-configurable color charts display a variety of real-time
performance metrics for single or multiple file systems, down to individual servers and targets, and reveal
metrics such as read/write heat maps, OST balance, file system capacity, metadata operations, read/write
7
Atipa Technologies
operations, job statistics, and various resource usage parameters, among others.
Summary
Atipa Solutions for Lustre consist of the entry-level LZ-60v2 and the enterprise-class LZE-576v2. The LZ60v2 is designed for users looking for fast scratch space or interested in exploring the benefits for Lustre for
their workload. This turn-key appliance starts off with 60TB of usable storage in only 4U of rack space, and
offers read and write throughputs of 4GB/s and 9GB/s, respectively. The LZE-576v2 is our flagship appliance
for users needing a high-performance, high-availability, and easy-to-maintain solution. The starting LZE576v2 configuration provides 576TB of usable storage in 12U of rack space, and can sustain sequential read
throughputs of more than 17GB/s and sequential write throughputs of more than 15GB/s. The appliance is
modular and can be easily expanded in increments of 576TB. In addition to the LZ-60v2 and LZE-576v2, we
also offer solutions customized to your needs upon request.
For more information or customized configurations, contact [email protected].
Atipa Technologies
4921 Legends Drive
Lawrence, KS 66046
(785) 841-9559
[email protected]
www.atipa.com
8
Atipa Technologies