Large-scale Machine Learning
using DryadLINQ
Mihai Budiu
Microsoft Research, Silicon Valley
HPA Workshop, Columbus, OH, May 1 2010
“What’s the point if I can’t have it?”
• Dryad+DryadLINQ available for download
– Academic license
– Commercial evaluation license
• Runs on Windows HPC platform
• Dryad is in binary form, DryadLINQ in source
• 3-page licensing agreement
• http://connect.microsoft.com/site/sitehome.aspx?SiteID=891
2
Goal of DryadLINQ
3
Software Stack
Machine learning
.Net
DryadLINQ
Dryad
Cluster storage
Cluster services
Windows
Server
Windows
Server
Windows
Server
Windows
Server
4
•
•
•
•
•
Introduction
Dryad
LINQ & DryadLINQ
Machine learning on DryadLINQ
Conclusions
5
Dryad
•
•
•
•
•
•
Deployed since 2006
Running 24/7 on >> 104 machines
Sifting through > 10Pb data daily
Clusters > 3000 machines
Jobs with > 105 processes each
Platform for rich software ecosystem
• Written at Microsoft Research, Silicon Valley
6
2-D Piping
• Unix Pipes: 1-D
grep | sed | sort | awk | perl
• Dryad: 2-D
grep1000 | sed500 | sort1000 | awk500 | perl50
7
Virtualized 2-D Pipelines
8
Virtualized 2-D Pipelines
9
Virtualized 2-D Pipelines
10
Virtualized 2-D Pipelines
11
Virtualized 2-D Pipelines
• 2D DAG
• multi-machine
• virtualized
12
Fault Tolerance
•
•
•
•
•
Introduction
Dryad
LINQ & DryadLINQ
Machine learning on DryadLINQ
Conclusions
14
LINQ Data Model
.NET objects of type T
Collection
IQueryable<T>
LINQ Language Summary
Input
Where (filter)
Select
(map)
GroupBy
OrderBy (sort)
Aggregate (fold)
Join
16
LINQ => DryadLINQ
Dryad
17
•
•
•
•
•
Introduction
Dryad
LINQ & DryadLINQ
Machine learning on DryadLINQ
Conclusions
18
K-Means Clustering in LINQ
Vector NearestCenter(Vector point, IQueryable<Vector> centers)
{
var nearest = centers.First();
foreach (var center in centers)
if ((point - center).Norm() < (point - nearest).Norm()) nearest = center;
return nearest;
}
IQueryable<Vector> KMeansStep(IQueryable<Vector> vectors, IQueryable<Vector> centers)
{
return vectors.GroupBy(vector => NearestCenter(vector, centers))
.Select(g => g.Aggregate((x,y) => x+y) / g.Count());
}
IQueryable<Vector> KMeans(IQueryable<Vector> vectors, IQueryable<Vector> centers, int iter)
{
for (int i = 0; i < iter; i++)
centers = KMeansStep(vectors, centers);
return centers;
}
LINQ = .Net+ Queries
IQueryable<Vector>
KMeansStep(IQueryable<Vector> vectors,
IQueryable<Vector> centers)
{
return vectors
.GroupBy(vector => NearestCenter(vector, centers))
.Select(g => g.Aggregate((x,y) => x+y) / g.Count());
}
20
DryadLINQ Data Model
.Net objects
Partition
Collection
21
DryadLINQ = LINQ + Dryad
IQueryable<Vector>
KMeansStep( IQueryable<Vector> vectors,
IQueryable<Vector> centers)
{
return vectors
.GroupBy(vector => NearestCenter(vector, centers))
.Select(g => g.Aggregate((x,y) => x+y) / g.Count());
}
collection
C#
C#
C#
C#
Dryad job
results
22
Vectors
Initial Centers
NearestCenter
Iter 1
GroupBy(centers)
Average(group)
Updated Centers
Iter 2
K-Means
DryadLINQ Machine-Learning Apps
Decision trees
Markov chains
Singular value decomposition
Expectation maximization
K-means
Linear regression
Probabilistic Index Maps
Principal component analysis
Probabilistic Latent Semantic Indexing
Road network shortest-path preprocessing
Epitome computation
Neural network training
Graphical models
24
M
map
Q
public static IQueryable<S> MapReduce<T,M,K,S>(
this IQueryable<T> input,
G1
Func<T, IQueryable<M>> mapper,
R
Func<M,K> keySelector,
Func<IGrouping<K,M>,S> reducer)
D
{
var map = input.SelectMany(mapper);
var group = map.GroupBy(keySelector);
var result = group.Select(reducer);
return result;
}
Q
Q
sort
G1
G1
groupby
R
R
reduce
D
D
distribute
MS
MS
mergesort
G2
G2
groupby
R
R
reduce
MS
MS
mergesort
G2
G2
groupby
R
R
reduce
X
X
consumer
partial aggregation
M
reduce
M
map
Aside: Map-Reduce in LINQ
25
Real Example: Natal Training
26
Natal Problem
• Recognize players from depth map
• At frame rate
• Minimize resource usage
27
Learn from Data
Rasterize
Motion Capture
(ground truth)
Training examples
Machine
learning
Classifier
28
Running on Xbox
29
Cluster-based training
Classifier
Training examples
Machine learning
DryadLINQ
Dryad
30
machine
Highly efficient parallellization
time
31
Conclusions
=
32
32
Backup Slides
Select
Where
SelectMany
GroupBy
Aggregate
c
m
Nested query
(collections c, m)
c.Select(e =>
new HashSet(m).Contains(e))
left
Join
right
V
Cholesky
A
AT
records
Tree layer
Vectors
Initial Centers
100G
350B
Compute local nearest center
Group on center
24K
Compute nearest center
Group on center
Compute new centers
350B
Iter 1
Merge new centers
100G
24K
Iter 2
350B
V
35M
Cholesky
96B
Repartition
Merge
Join
71M
V x Cholesky Sum, Repartition
Merge
36M
A
Join
20G
2G
A x V Sum, Repartition
74M
AT
Merge
20G
Join
AT x A x V
Sum
Plan in box is repeated 5 times
1G
Decision Tree Training
records
12G
a
500K
b
12K
c
3K
d
16B
Tree layer
Expectation Maximization
• 160 lines
• 3 iterations shown
41
Probabilistic Index Maps
Images
features
42
Design Space
Internet
Dataparallel
Private
data
center
Shared
memory
Latency
Throughput
43
Data-Parallel Computation
Application
SQL
Language
Execution
Storage
Parallel
Databases
Sawzall
≈SQL
LINQ, SQL
Sawzall
Pig, Hive
DryadLINQ
Scope
MapReduce
Hadoop
Dryad
GFS
BigTable
HDFS
S3
Cosmos,
HPC, Azure
Cosmos
Azure
SQL Server
44
Dryad System Architecture
data plane
job schedule
Files, TCP, FIFO, Network
NS,
Sched
Job manager
control plane
V
V
V
PD
PD
PD
cluster
45
Dryad Job Structure
Channels
Input
files
Stage
sort
grep
Output
files
awk
sed
perl
sort
grep
awk
sed
grep
Vertices
(processes)
sort
46
Dryad = Execution Layer
Job (application)
Dryad
Cluster
Pipeline
≈
Shell
Machine
47