A. Craik, C Black, P Doyle 4-Nov-14
Mincer: a distributed automated
problem determination tool
© 2013 IBM Corporation
Debugging Optimizing Compilers
Nearly every optimizing transformation is optional
Adopt the scientific method – run experiments varying
compilation control parameters to find failure
Problems
Time consuming & tedious to configure & run
Resource constraints limit number of experiments
Experience needed to choose the best experiments
Environment may cause problem to occur intermittently
 Resources wasted repeatedly running the same
experiments
© 2013 IBM Corporation
Objectives
Tool to run tests in a distributed machine farm
Distributed experiment selection and execution logic for
scalability
Configurability & extensibility to support new transformations
and new debugging techniques
Reduce resource waste – gain knowledge from every
experiment run
Tolerate false-positive & false-negative results and handle
variable failure rates
© 2013 IBM Corporation
Problem Conceptualization
A problem determination tactic aims to isolate a problem by
varying an experimental parameter and observing the effect
Problem has two discrete components:
– Deciding which problem determination tactics to use
– Applying chosen problem determination tactics to failure
Each experimental parameter can be mapped back to an
integer search space
Mathematically model the selection of experiments and the
results they produce
© 2013 IBM Corporation
Mincer Architecture
mincer.pl
Sequential
Subset
OptLevel
LastOptIndex
LimitFile
LastOptSub
Index
Results DB
Solvers
GatherLogs
Data
Layer
Tactics
Tactic
Execution
Engine
© 2013 IBM Corporation
Experiment Selection Logic
© 2013 IBM Corporation
The Numerical Solvers
Solvers use Bayesian Inference to update the underlying
hyperparameters of the experiments
Solvers use Information Theory to make optimal test
parameter choices
Solve to a user-defined confidence level (-c option)
Currently using 2 solvers:
- sequential (contiguous subranges from 1 to some n)
- subset (unordered subsets)
Solvers also track failure rate to guide experiment selection
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Sequential Solver
© 2013 IBM Corporation
Current Status
Mincer being used to debug problems in production
We have implemented tactics for several different
experimental parameters on our Java JIT
Working on Infinite Sequential Solver to allow us to isolate
intermittent failures to specific changesets
© 2013 IBM Corporation
Q&A
© 2013 IBM Corporation
Sequential Solver
Theory
Have a finite, ordered set, T
Random Variable P, support on T
Random Variable f, support on [0, 1]
Random Variables {Xi}, binary
© 2013 IBM Corporation
Sequential Solver
Theory
Each Xi is associated with an index ni in T
Pr(Xi passing | f, P) = (1-f) if ni ≥ P
Pr(Xi passing | f, P) = 1 if ni < P
Choose index to associate with next Xi optimally
© 2013 IBM Corporation
Subset Solver
Theory
Each Xi is associated with a subset Ni of T
Pr(Xi passing | f, P) = (1-f) if P ϵ Ni
Pr(Xi passing | f, P) = 1 otherwise
Choose next subset optimally
Faster convergence than possible with Sequential Solver
© 2013 IBM Corporation