Collective Intelligence for Data Center
Operations Management
Xiaojun (XJ) Liu, Chief Scientist
Spark Summit 2013
Outline
●
Operations Management SaaS
●
Experiences with Spark
●
Ongoing Work
Operations Management SaaS
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 3 of 11
Data Pipeline
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 4 of 11
The Data We Collect
●
Configuration, performance, tasks and more
●
From virtual machines, servers, networks and storage
●
100 billion+ metric samples per day
●
On average 1.3 million properties per datacenter
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 5 of 11
Cross-User Analysis Setup
●
●
●
Spark Cluster (EC2)
Resource Sizing
Consolidation Ratio
Utilization …
(Standalone App in Scala)
Master
Tesla
CloudPhysics library for Spark
Data query
●
Data Repository (S3)
References
Analysis App:
Worker
Data History (HBase)
range, firstAfter, …
Extraction
●
config/perf/task
Segment
●
Status Server (Play! App)
by running vm or host count
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 6 of 11
Analyses We Have Done
●
VM configured resources and utilization
●
Server CPU and memory utilization distribution
●
Storage array and interconnect adoption
●
Virtualization product feature adoption
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 7 of 11
VM vCPU and Memory Size Distribution
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 8 of 11
Experiences with Spark on EC2
+ Quick getting up to speed
+ Great EC2 support
+ Tolerating variations in task execution time
- Variations in EC2 instance performance
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 9 of 11
Next Steps
●
●
●
Create RESTful API for frequently used analyses to
make update and consumption easier
Utilize Shark and MLbase for cross-user data analysis
Utilize Spark Streaming for near-realtime performance
data analysis
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 10 of 11
Thank You!
www.cloudphysics.com
[email protected]
@cloudphysics
Xiaojun (XJ) Liu
[email protected]
Spark Summit Dec 2, 2013
Copyright © CloudPhysics 2013
Slide 11 of 11