Using OpenStack Sahara & Manila
to run Analytics
against NFS Datastores
January 2017
Presenter: Chris Madden
NetApp
1
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
MISTRAL
MURANO
TROVE
KOLLA
MAGNUM
OpenStack
Keeps Growing!
CONGRESS
IRONIC
CEILOMETER
ZAQAR
OpenStack
Today
HORIZON
BARBICAN
KARBOR
SAHARA
NOVA
SWIFT
SENLIN
CINDER
HEAT
MANILA
NEUTRON
GLANCE
KEYSTONE
TACKER
TRIPLE-O
DESIGNATE
MONASCA
HORIZON
KURYR
CHARMS
RALLY
Lots of data in shared file systems
NFS
CIFS .
Gluster _
HDFS .
…
3
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
65
%
storage sold
for Shared File Systems
(IDC)
OpenStack Foundation User Survey Results
Shared File System Drivers in Use (April 2016)
NetApp
23%
Generic
20%
GlusterFS
17%
IBM GPFS 9%
HDFS 8%
Windows SMB 12%
EMC 5%
Quobyte 3%
Huawei 3%
 NetApp is the founder of, leading contributor to, and most widely deployed option for Manila
4
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
OpenStack Manila
The OpenStack Shared File Service Program
 Manila delivers file systems as
a service:
 NFS, CIFS, HDFS, and more
Manila
 Production-ready in Liberty
 Supports a wide range of
network topologies
5
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
ONTAP
OpenStack Compute
Admin
R&D
Eng
VM
VM
VM
VM
Hadoop Clusters are not easy!
Same for Spark, Storm, etc…
 Difficult to configure & maintain
 Adaption to changing demands
 Hardware headaches
 Commodity hardware is cheap but requires more work
 Reliable hardware is expensive
 Hadoop dev is very difficult without a real
cluster
6
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
OpenStack Sahara
The OpenStack Data Processing Program
 Put Hadoop and alike in a OpenStack
cloud!
 Then have easy-to-use, standardized
interfaces:






To create clusters (reliably and repeatedly)
To scale clusters
To run data processing jobs
On any popular data processing framework
With sensible defaults that just work
And sophisticated configuration management for
expert users
 That's OpenStack Sahara!
7
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
Technical Overview
Virtual Cluster
Virtual Cluster
Virtual Cluster
Computing
Task
Computing
Task
Computing
Task
Manila Service
HDFS Driver
VM
HDFS
Host
NetApp
Hadoop NFS
Connector
VM
Manila Service
NFS Driver
NFS
Host
Local
Volume
VM
Manila Service
Gluster
Drivers or
others
GlusterFS
Host
External HDFS via Manila
NFS directly via Manila
Local Storage in Manila
 Sahara can support external
HDFS by using the HDFS
driver in Manila
 NetApp Hadoop NFS
Connector brings the NFS
capability into Hadoop
 Use local storage in Hadoop
to remote mount any type of
storage in Manila
8
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —
Summary
 Manila adds Fileshares-aaS to OpenStack
 Sahara adds Analytics-aaS to OpenStack
 NetApp NFS Connector for HDFS allows access of
NFS data through HDFS protocol
 All together:
Easy analytics against existing datasets in file shares
 Resources
 https://www.youtube.com/watch?v=Tf_8v0OvR80
 https://github.com/NetApp/NetApp-Hadoop-NFS-Connector
 https://www.netapp.com/us/media/tr-4464.pdf
9
© 2017 NetApp, Inc. All rights reserved. — NETAPP CONFIDENTIAL —