R-Storm:
Resource Aware Scheduling in Storm
Boyang Peng
Mohammad Hosseini
Zhihao Hong
Reza Farivar
Roy Campbell
Introduction
• STORM is an open source distributed real-time data
stream processing system
– Real-time analytics
– Online machine learning
– Continuous computation
2
R-Storm: Resource Aware Scheduling in
Storm
• A scheduler that takes into account
resource availability and resource
requirement
– Satisfy resource requirements
– Improve throughput by maximizing resource
utilization and minimizing network latency
3
Your date comes here
Storm Overview
Reference: storm.apache.org
4
Your footer comes here
Your date comes here
Storm Topology
Reference: storm.apache.org
5
Your footer comes here
Your date comes here
Definitions of Storm Terms
• Tuples - The basic unit of data that is processed.
• Stream - an unbounded sequence of tuples.
• Component - A processing operator in a Storm
topology that is either a Bolt or Spout
• Tasks - A Storm job that is an instantiation of a
Spout or Bolt.
• Executors - A thread that is spawned in a worker
process that may execute one or more tasks.
• Worker Process - A process spawned by Storm
that may run one or more executors.
6
Your footer comes here
Your date comes here
Logical connections in a Storm Topology
7
Your footer comes here
Your date comes here
Storm topology physical connections
8
Your footer comes here
Your date comes here
Motivation
• Naïve round robin scheduler
• Naïve load limiter (Worker Slots)
• Resources that don’t have graceful
performance degradation
– Memory
9
Your footer comes here
Your date comes here
R-Storm: Resource Aware Scheduling in
Storm
• Micro-benchmark
– 30-47% higher throughput
• For Yahoo! Storm applications
– R-Storm outperforms default Storm by
around 50% based on overall throughput.
10
Your footer comes here
Your date comes here
Problem Formulation
• Targeting 3 types of resources
– CPU, Memory, and Network
• Limited resource budget for each node
• Specific resource Goal:
needs for each task
Improve throughput by maximizing
utilization and minimizing network
latency
11
Problem Formulation
• Set of all tasks Ƭ = {τ1 , τ2, τ3, …}, each task τi
has resource demands
– CPU requirement of cτi
– Network bandwidth requirement of bτi
– Memory requirement of mτi
• Set of all nodes N = {θ1 , θ2, θ3, …}
– Total available CPU budget of W1
– Total available Bandwidth budget of W2
– Total available Memory budget of W3
12
Problem Formulation
• Qi : Throughput contribution of each node
• Assign tasks to a subset of nodes N’ ∈ N
that minimizes the total resource waste:
13
Problem Formulation
 Quadratic Multiple 3D Knapsack Problem
– We call it QM3DKP!
– NP-Hard!
• Compute optimal solutions or
approximate solutions may be hard and
time consuming
• Real time systems need fast scheduling
– Re-compute scheduling when failures occur
14
Soft Constraints Vs Hard Constraints
• Soft Constraints
– CPU and Network Resources
– Graceful performance degradation with over
subscription
• Hard Constraints
– Memory
– Oversubscribe -> Game over
15
Your footer comes here
Your date comes here
Observations on Network Latency
1. Inter-rack communication is the slowest
2. Inter-node communication is slow
3. Inter-process communication is faster
4. Intra-process communication is the
fastest
16
Your footer comes here
Your date comes here
Heuristic Algorithm
• Greedy approach
• Designing a 3D resource space
– Each resource maps to an axis
– Can be generalized to nD resource space
– Trivial overhead!
• Based on:
– min (Euclidean distance)
– Satisfy hard constraints
17
Heuristic Algorithm
18
Your footer comes here
Your date comes here
Heuristic Algorithm
1
3
Switch
2
4
5
6
19
Your footer comes here
Your date comes here
Heuristic Algorithm
• Our proposed heuristic algorithm has the
following properties:
1) Tasks of components that communicate
will each other will have the highest
priority to be scheduled in close network
proximity to each other.
2) No hard resource constraint is violated.
3) Resource waste on nodes are minimized.
20
Evaluation
• Micro-benchmarks
– Linear Topology
– Diamond Topology
– Star Topology
• Industry inspired benchmarks
– PageLoad Topology
– Processing Topology
• Network Bound versus Computation Bound
21
Evaluation Setup
• Used Emulab.net as testbed and to
emulate inter-rack latency across two
sides
• 1 host for Nimbus + Zookeeper
• 12 hosts as worker nodes
0
• All hosts:
 Ubuntu 12.04 LTS 1
7
6
2
8
5
3 4
9
 1-core Intel CPU
 2GB RAM+ 100Mb NIC
V
0
22
V
1
1
0
1
1
1
2
Micro-benchmarks
Linear Topology
Diamond Topology
23
Star Topology
Your footer comes here
Your date comes here
Network Bound vs CPU Bound
• Network Bound
– Network Bottleneck
– Speed of light test
• CPU Bound
– Computation Bound
– Arbitrary computation (Find prime numbers)
24
Your footer comes here
Your date comes here
Network-bound Micro-benchmark
50%
improvement
25
Your footer comes here
Your date comes here
Network-bound Micro-benchmark
Topologies
30%
improvement
26
Your footer comes here
Your date comes here
Network-bound Micro-benchmark
Topologies
47%
improvement
27
Your footer comes here
Your date comes here
Computation Bound Micro-benchmark
28
Your footer comes here
Your date comes here
CPU Utilization
29
Your footer comes here
Your date comes here
Lesson Learn for Computation bound
• More nodes might not be better while
parallelism is constant.
• Need the right amount
30
Your footer comes here
Your date comes here
Yahoo Sample Topologies
• Layout of topologies provided by Yahoo
• Implemented in abstractly for our
evaluation
31
Your footer comes here
Your date comes here
PageLoad Topology
50%
improvement
32
Your footer comes here
Your date comes here
Processing Topology
47%
improvement
33
Your footer comes here
Your date comes here
Throughput comparison of running
multiple topologies
34
Your footer comes here
Your date comes here
Multiple Topologies Results
• PageLoad topology
– R-Storm (25496 tuples/10sec)
– Default Storm (16695 tuples/10sec)
– R-Storm is around 53% higher
• Processing topology
– R-Storm (67115 tuples/10sec)
– Default Storm (10 tuples/sec).
– Orders of magnitude higher
35
Your footer comes here
Your date comes here
Current Status
• Merged with Apache Storm 0.11 Release
– https://github.com/apache/storm/
• Starting to be used a Yahoo! Inc.
36
Your footer comes here
Your date comes here
Future Work
• Experiments to quantify latency
• Multi-tenancy
– Topology Priorities and Per User Resource
Guarantees
• Elasticity
– Dynamic adjusting parallelism to meet SLA
37
Your footer comes here
Your date comes here
Questions?
38
Your footer comes here
Your date comes here