Bayesian Average Error Based Approach to
Sample Size Calculations for
Hypothesis Testing
Eric M Reyes and Sujit K Ghosh
Department of Statistics
North Carolina State University
NCSU Bayesian Seminar Series
01 September 2011
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
A Motivating Example
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Example: Binary Response, Two Independent Samples
Sampling Distribution
Two independent groups that have event rates θ1 and θ2 .
Hypothesis of Interest
H0 : θ1 = θ2 vs.
H1 : θ1 6= θ2
Decision Rule
For test statistic T (X ), we reject H0 if T (X ) > t for some t.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Classical Approach
The sample size is given by the following rule:
nZ ≥
Z α∗
2
p
p
2θ̄(1 − θ̄) + Zβ ∗ θ1 (1 − θ1 ) + θ2 (1 − θ2 )
(θ2 − θ1 )2
where α∗ and β ∗ represent the Type-I and Type-II error, respectively, and
θ̄ = (θ1 + θ2 ) /2 [Friedman 1998, Fund. of Clin. Trials].
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Limitations of Classical Approach
nZ ≥
Z α∗
2
p
p
2θ̄(1 − θ̄) + Zβ ∗ θ1 (1 − θ1 ) + θ2 (1 − θ2 )
(θ2 − θ1 )2
Calculations rely on posited values for the parameters of interest.
Pivot quantitites not guaranteed to exist [Adcock 1997, Stat.].
Asymptotic normal approximations may be questionable
[M’Lan 2008, Bayes. Anal.].
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Methodology
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Problem Statement
Sampling Distribution
X ∼ f (x|θ) where θ ∼ π(θ) and θ ∈ Θ
Hypothesis of Interest
H0 : θ ∈ Θ0 vs. H1 : θ ∈ Θ1 ,
where Θ0 ∩ Θ1 = ∅ and Θ0 ∪ Θ1 ⊆ Θ
Decision Rule
For test statistic T (X ), we reject H0 if T (X ) > t for some t.
Key Assumption
Pr (θ ∈ Θj ) =
R
Θj
π(θ)dθ > 0 for j = 0, 1
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Bayesian Average Errors
1.0
Average Bayesian Type-I Error :
0.5
AE1 (t) = Pr (T (X ) > t|θ ∈ Θ0 )
Average Bayesian Type-II Error :
t0
AE2 (t) = Pr (T (X ) ≤ t|θ ∈ Θ1 )
0.0
Density of Test Statistic T(X)
1.5
Density Under H0
Density Under H1
AE1(t0)
AE2(t0)
0.0
0.5
1.0
1.5
2.0
Cutoff t
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Optimal Choice of Cut-off t
For a given w ∈ (0, 1), define the Total Weighted Error (TWE) as
TWE(t, w ) = w AE1 (t) + (1 − w )AE2 (t).
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Optimal Choice of Cut-off t
For a given w ∈ (0, 1), define the Total Weighted Error (TWE) as
TWE(t, w ) = w AE1 (t) + (1 − w )AE2 (t).
Definition of Optimal Cut-off
t0 (w ) = arg mint {TWE (t, w )}
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Bayes Factor
The Bayes Factor BF(X ) is often used to quantify the evidence in favor of
H1 [Weiss 1997, Stat.]:
Pr (θ ∈ Θ1 |X )
Pr (θ ∈ Θ0 )
BF(X ) =
Pr (θ ∈ Θ0 |X )
Pr (θ ∈ Θ1 )
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Primary Result
Theorem
Consider testing the hypothesis as described previously. Let BF(X ) denote
the Bayes Factor and let
ϕ(X ) : x → [0, 1]
represent a randomized test for the hypothesis. Then, for a given value of
w ∈ (0, 1), ϕ̂(X ) minimizes TWE where
ϕ̂ = I (log(BF(X )) > log(w /(1 − w ))) .
Implications
T (X ) = log(BF(X ))
and
t0 = log
w
1−w
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Rule for Sample Size Determination
Define Total Error (TE) as
TE(t) = AE1 (t) + AE2 (t).
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Rule for Sample Size Determination
Define Total Error (TE) as
TE(t) = AE1 (t) + AE2 (t).
Specify a bound α ∈ (0, 1) on TE.
Specify a weight w ∈ (0, 1).
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Rule for Sample Size Determination
0.29
w 
t0 = log
1 − w
Define Total Error (TE) as
0.28
n = n1
TE(t) = AE1 (t) + AE2 (t).
n = n2
0.27
0.26
Total Error
n = n3
Specify a bound α ∈ (0, 1) on TE.
Specify a weight w ∈ (0, 1).
Total Error Bound α
0.24
TE(t0 (w )) ≤ α.
0.25
Choose minimum n > 1 such that
0.0
0.5
1.0
1.5
2.0
Cutoff t
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Example
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Example: Binary Response, Two Independent Samples
Sampling Distribution
ind.
X = (x1 , x2 ) where xk |θk ∼ Bin (n, θk ), for k = 1, 2
θ = (θ1 , θ2 ) ∈ [0, 1]2 .
Hypothesis of Interest
H0 : θ1 = θ2 Θ0 = {θ ∈ [0, 1]2 : θ1 = θ2 }
H1 : θ1 6= θ2 Θ1 = {θ ∈ [0, 1]2 : θ1 6= θ2 }
Prior Distribution
π(θ) = uI (θ1 = θ2 = η) p(a0 ,b0 ) (η)+
(1 − u)I (θ1 6= θ2 ) p(a1 ,b1 ) (θ1 )p(a2 ,b2 ) (θ2 )
where u = Pr (θ1 = θ2 ) and p(a,b) (θ) denotes a Beta(a, b) density.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Choices for Classical Approach
nZ ≥
Z α∗
2
p
p
2θ̄(1 − θ̄) + Zβ ∗ θ1 (1 − θ1 ) + θ2 (1 − θ2 )
(θ2 − θ1 )2
Set θ1 and θ2 such that θ̄ = 0.5.
Choose α∗ = 0.05 and β ∗ = 0.20.
Choose α = 0.25.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
2.5
Prior Parameters
2.0
θ1 = θ2
θ1
θ2
0.0
0.5
1.0
Density
1.5
d = θ2 − θ1
0.0
0.2
0.4
0.6
0.8
1.0
θ
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Results: Binary Response, Two Independent Samples
0
nZ
nw =0.9
nw =0.5
nw =0.1
172
111
827
d = θ2 − θ1
0.1 0.2 0.3 0.4
0.5
392
159
103
762
15
32
20
136
97
127
82
603
43
87
56
404
24
54
35
240
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Discussion
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Critiques
Critique 1: Computation
Increasing complexity and/or sample sizes increases computation.
An observed relationship between log n and log α may decrease the
computational burden.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
log n - log α Relationship
w = 0.9
w = 0.5
w = 0.1
6
●
●
●
●
●
●
●
5
●
●
●
●
●
●
●
●
●
●
4
^ (α)
logn
7
●
−1.8
−1.6
−1.4
●
●
●
●
−1.2
Log Total Error Bound α
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Critiques
Critique 2: Sensitivity of Priors
Bayes factor is sensitive to choice of priors.
Our key assumption does not allow for use of improper reference
priors.
Using “alternative Bayes factor” for T (X ) may yield a promising option.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Future Work
Use of alternative Bayes factor for T (X ).
Better understanding the relationship between n and α.
Using different conditional errors: Pr (θ ∈ Θj |X )
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Conclusions
This methodology is a general approach to hypothesis testing and sample
size determination that is broadly applicable to simple and complex
hypothesis tests.
R package available at http://www4.ncsu.edu/∼emreyes/
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Conclusions
This methodology is a general approach to hypothesis testing and sample
size determination that is broadly applicable to simple and complex
hypothesis tests.
R package available at http://www4.ncsu.edu/∼emreyes/
This research was supported by NIH grant T32HL079896.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
References
I
Adcock CJ.
Sample size determination: a review.
The Statistician, 46:261-283, 1997.
I
Friedman LM, Furberg CD, DeMets DL.
Fundamentals of clinical trials. 3rd edition.
Springer-Verlag, 1998.
I
M’Lan CE, Joseph L, Wolfson DB.
Bayesian sample size determination for binomial proportions.
Bayesian Analysis, 3:269-296, 2008.
I
Weiss R.
Bayesian sample size calculations for hypothesis testing.
The Statistician, 46:185-191, 1997.
NCSU Seminar 2011
Motivation
Methodology
Example
Discussion
Appendices
Total Weighted Error
TWE(t, w ) = w AE1 (t) + (1 − w )AE2 (t)
Z
= (1 − w ) − I (BF(x) > t) BF(x) −
R
m0 (X ) =
f (x|θ)π(θ)dθ
R
Θ0 π(θ)dθ
Θ0
w
1−w
(1 − w )m0 (x)dx
BF(X ) =
m1 (X )
m0 (X )
NCSU Seminar 2011