An Empirical Study of the Prediction
Performance of Space-filling Designs
Rachel T. Johnson
Douglas C. Montgomery
Bradley Jones
Computer Models
•
•
•
•
•
•
Widely used in engineering design
Use continues to grow
May have lots of variables, many responses
Can have long run times
Complex output results
Need efficient methods for designing the
experiment and analyzing the results
Johnson QPRC 2009
2
Computer Simulation Types
Computer
Simulation Models
Stochastic
Simulation Models
Deterministic
Simulation Models
Discrete
Event
Simulation
(DES)
Computational
Fluid
Dynamics
(CFD)
Johnson QPRC 2009
3
Comparing Designs
• Space-filling designs compared
–
–
–
–
Sphere packing (SP)
Latin Hypercube (LH)
Uniform (U)
Gaussian Process Integrated Mean Square Error
(GP IMSE)
• Assumed surrogate model
– Gaussian Process model
• Comparisons based on prediction
Johnson QPRC 2009
4
Designs
Latin Hypercube
Latin
Hypercube:
1
• McKay
et al. (1979)
0.5
• Maximizes the
minimum
distance
0
between
pairs of design
-0.5
points
0.5
Uniform:
X2
X2
• A random n x s matrix,
0
in which
columns are a
random
permutation of
-0.5
{1,-1 . . ., n}
-1
Uniform
1
• 0.5Feng (1980)
• A0 set of n points
uniformly scattered
-0.5
within
the design space
X2
Packing
Sphere Sphere
Packing:
1
• Johnson et al. (1990)
-1
-0.5
0
0.5
1
-1
-0.5
0
X1
0.5
GASP IMSE
0.75
• Sacks et al. (1989)
•0.25Minimizes the
0
integrated
mean square
-0.25
error of the Gaussian
Process
model
-0.5
X1
1
0.5
X2
X2
• Shewry and Wynn
0.5
(1987)
0
• Maximizes
the
information
contained in
-0.5
the distribution of a data
-1
set
0.5
-0.75
-0.75
-0.5
-0.25
0
0.5
1
I - Optimal
GP IMSE:
0
0
X1
Maximum
Entropy:
1
-0.5
-0.5
X1
Maximum Entropy
-1
-1
1
0.25
0.5
X1
Johnson QPRC 2009
I -1 Optimal:
• 0.5Box and Draper (1963)
• Minimizes the average
0
prediction
variance (of a
linear
regression model)
-0.5
over
a design region
-1
X2
-1
-1
-0.5
0
0.5
1
X1
5
Model Fitting Techniques
• Gaussian Process Model
– Assumes a normal distribution
– Interpolator based on correlation between points
– Prediction variance calculated as
Johnson QPRC 2009
6
Evaluation of Designs for the GP Model
• Recall the prediction variance:
• Prediction variance dependent on:
–
–
–
–
Design
Value of unknown θ
Sample size
Dimension of x
• Design of Experiments can be used to evaluate
the designs
Johnson QPRC 2009
7
Example FDS Plots
Maximum
Entropy
Sphere Packing
Uniform
Latin
Hypercube
GP IMSE
Johnson QPRC 2009
8
Research Questions
• Does it matter in practice what design you choose?
– Is there a dominating experimental design that performs better in terms
of model fitting and prediction?
• What is the role of sample size in experimental designs used to
fit the GASP model?
– At what point does prediction error variance and other measures of
prediction performance become reasonably small with respect to N, the
sample size chosen?
Johnson QPRC 2009
9
Empirical Comparison
• Used test functions to act as surrogates to
simulation code
• Evaluated designs based on RMSE and AAPE
• Interested in effect of:
–
–
–
–
Design type
Sample size
Dimension of design
*Gaussian Correlation Function
Johnson QPRC 2009
10
Comparison Procedure
•
•
•
•
•
•
Step 1: Choose a test function
Step 2: Choose a sample size and space-filling design
Step 3: Create the design with the number of factors equal to the number in the test
function chosen in step 1) and the specifications set in step 2)
Step 4: Using the test function in 1) find the values that correspond to each row in
the design
Step 5: Fit the GASP model
Step 6: Generate a set of 40,000 uniformly random selected points in the design
space and compare the predicted value (generated by the fitted GASP model) to the
actual value (generated by the test function) at these points. Compute the Root
mean square error (RMSE).
Johnson QPRC 2009
11
Test Functions
Test Function
#1
Test Function
#3
Test Function
#2
Test Function
#4
Johnson QPRC 2009
12
Test Function #1 Results
Johnson QPRC 2009
13
Test Function #2 Results
Johnson QPRC 2009
14
Test Function #3 Results
Johnson QPRC 2009
15
Test Function #2 Results
Johnson QPRC 2009
16
ANOVA Analysis
Johnson QPRC 2009
17
Jackknife Plots
Test Function #2 – LHD with
a sample size of 20
Test Function #2 – LHD with
a sample size of 50
Johnson QPRC 2009
18
Conclusions
• No one design type is better than the others
• Increasing sample size decreases RMSE
• There is a strong interaction between sample size and
test function type – the more complex the function the
more runs required
• Jackknife plot is an excellent indicator of a “good fit”
Johnson QPRC 2009
19
References
• Fang, K.T. (1980). “The Uniform Design: Application of NumberTheoretic Methods in Experimental Design,” Acta Math. Appl.
Sinica. 3, pp. 363 – 372.
• Johnson, M.E., Moore, L.M. and Ylvisaker, D. (1990). “Minimax
and maxmin distance design,” Journal for Statistical Planning and
Inference 26, pp. 131 – 148.
• McKay, N. D., Conover, W. J., Beckman, R. J. (1979). “A
Comparison of Three Methods for Selecting Values of Input
Variables in the Analysis of Output from a Computer Code,”
Technometrics 21, pp. 239 – 245.
• Sacks, J., Welch, W. J., Mitchell, T. J., and Wynn, H. P. (1989).
“Design and Analysis of Computer Experiments,” Statistical
Science 4(4), pp. 409 – 423.
• Shewry, M.C. and Wynn, H.P. (1987). “Maximum entropy
sampling,” Journal of Applied Statistics 14, pp. 898 – 914.
Johnson QPRC 2009
20
Correlation Function
Gaussian Correlation Function
Cubic Correlation Function
Boxplot of RMSE
4
RMSE
3
2
1
0
Design
SE
RM
Sample Size
F
CC
SE
RM
SS
10
F
GC
SE
RM
F
CC
SE
RM
SS
20
F
GC
SE
RM
F
CC
SE
RM
SS
F
GC
SE
RM
40
Johnson QPRC 2009
F
CC
SE
RM
SS
F
GC
50
21