Construction and Evaluation of Response
Surface Designs Incorporating Bias from
Model Misspecification
Connie M. Borror, Arizona State University West
Christine M. Anderson-Cook, Los Alamos National Laboratory
Bradley Jones, JMP SAS Institute
Motivation

Response surface design evaluation (and
creation) assuming a particular model




Single number efficiencies
Prediction variance performance
Mean-squared error
Model misspecification?

What effect does this have on prediction and
optimization?
Motivation

Examine effect of model misspecification





Expected squared bias
Prediction variance
Expected mean squared error
Using fraction of design space (FDS) plots and
box plots
Evaluate designs based on the contribution of
ESB relative to PV.
Scenario



Cuboidal regions
True form of the model is of higher order than
the model being fit.
Examine


Response surface models when the true form is
cubic
Screening experiment when the true form is full
second order.
Model Specifications

The model to be fit is
Y = X11 + ε


X1 = n × p design matrix for the assumed form of
the model
The true form of the model is
Y = X11+ X22 + ε

X2 = n × q design matrix pertaining to those
parameters (2) not present in the model to be fit
(assumed model).
Model Specifications


2 in general, are not fully estimable
Assume 2 ~ N(0, β )
2
 21


Σβ2  

 0

 2
2
0 




 2q 
Criteria

Mean-squared error
MSE[ yˆ (x)]   2 w  X1'X1  w  β 2  wA  z  z  wA  β2
1

Expected squared bias (ESB):
ESB  

2
β2
z  Awz  wA
Expected MSE sum of PV and ESB
Fraction of Design Space (FDS) Plots


9
CCD-1CR
CCD-3CR
Hexagon-1CR
8
7
VPS

Zahran, Anderson-Cook,
and Myers (2003) scaled
prediction variance values
are plotted versus the
fraction of the design
space that has SPV at or
below the given value
Adapt this to plot ESB
and EMSE as well as PV.
We use FDS plots and
box plots to assess the
designs
6 100% G-eff
5
4
3
0
0.25
0.5
0.75
Fraction of Design Space
1
Cases

I. Two-factor response surface design

Assume a second-order model:
y  0   i xi   ij xi x j   ii xi2  

True form of the model is cubic:
y  0   i xi   ij xi x j   ii xi2   ij xi x2j   iii xi3  
Case I Designs







Central Composite Design (CCD)
Quadratic I-optimal (Q I-opt)
Quadratic D-optimal (Q D-opt)
Cubic I-optimal (C I-opt)
Cubic D-optimal (C D-opt)
Cubic Bayes I-optimal (C Bayes I-opt)
Cubic Bayes D-optimal (C Bayes D-opt)
Case I

CCD
If we assume  B2  1 2
Case I Designs

CCD (ESB and EMSE performance as bias increases)
Case I Designs

PV for all designs
Case I Designs

ESB for all designs
Case I Designs

EMSE for all designs
Case I Designs

FDS for EMSE for all designs
Case II

Four factor response surface design

Assume a second-order model:
y  0   i xi   ij xi x j   ii xi2  

True form of the model is cubic:
y  0   i xi   ij xi x j   ii xi2   ij xi x2j   iii xi3  

20 additional terms as we move from the secondorder model to cubic.
Case II Designs

Six possible designs, with n = 27 runs






Central Composite Design (CCD)
Box Behnken Design (BBD)
Quadratic I-optimal (Q I-Opt)
Quadratic D-optimal (Q D-Opt)
Cubic Bayes I-optimal (C Bayes I-Opt)
Cubic Bayes D-optimal (C Bayes D-Opt)
Note: Cubic I- and D-Optimal not possible with
available size of design
Case II

PV for all designs
Case II

EMSE for all designs
Case II
FDS plot of EMSE for Four Factors
Case III

Eight-factor Screening Design

Assume a first-order model:
y  0   i xi  

True form of the model is full second-order:
y  0   i xi   ij xi x j   ii xi2  
i j
Case III Designs




28-4 fractional factorial design with 4 center
runs
D-optimal (for first order)
Bayes I-optimal (for second order)
Bayes D-optimal (for second order)
Case III Designs

The difference in the number of terms from
the assumed to the true form of the models
increases from 8 to 44.

We would expect bias to quickly dominate
EMSE.
Case III

PV for all designs
Case III

ESB for all designs
Case III

EMSE for all designs
Design Notes

For the two-factor case:



The I-optimal and CCD were equivalent.
They performed the best based on minimizing the
maximum EMSE
They performed the best based on prediction
variance
Design Notes

For the four factor case,

the BBD was best based on EMSE criteria (in
particular, the 95th percentile, median, mean)



when size of the coefficients of missing terms are
moderate to large
The I-optimal design was competitive for this case
only if small amounts of bias were present.
As the number of missing cubic terms increases,
the BBD was best for EMSE.
Design Notes


I-optimal designs were highly competitive
over 95% of the design region; not with
respect to the maximum PV, ESB, and EMSE.
Cubic Bayesian designs did not perform well.
Design Notes

In the screening design example:

The D-optimal designs best if the assumed model is
correct, but break down quickly if quadratic terms are in
the model




Much more pronounced than in the response surface design
cases.
Quadratic Bayesian I-optimal design was best based on
mean, median, and 95th percentile of EMSE
The 28-4 fractional factorial design was best with respect
to the maximum EMSE.
The 28-4 design was best for both PV and ESB when the
PV and ESB contribution to the model were balanced.
Conclusions



Appropriate design can strongly depend on the assumption
that we know the true form of the underlying model
If we select designs carefully it is often possible to select a
model that predicts well in the design space, and provide
some protection against missing model terms.
The ESB approach to assessing the effect of missing terms
provides is advantageous:


do not have to specify coefficient values for the true underlying
model,
Instead, the relative size of the missing terms can be calibrated
relative to the variance of the observations.
Conclusions


Size of the bias variance relative to observational
error needed to balance contributions from PV and
ESB is highly dependent on the number of missing
terms from the assumed model.
As the number of missing terms increases, the ability
of designs to cope with the bias decreases
substantially

different designs are able to handle this increasing bias
differently.