1. Scientific Setting
1. Scientific Setting
1.1 Introduction and overview
(a) Motivation
(b) Review of statistical foundations
(c) The scientific method (scientist game)
1.2 The study question
(a)
(b)
(c)
(d)
What is the treatment?
Phase I-IV clinical trials
Nature of the clinical question
1- versus 2-sided questions
1.3 Case Study
1
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
1 / 21
1.1 Introduction and Motivation
1.1(a) Review of statistical foundations
(i)
(ii)
(iii)
(iv)
What is statistical inference?
Four required elements
Properties of estimators
Interpretation of interval estimates
2
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
2 / 21
1.1(a) Review of statistical foundations
Why am I reviewing statistical foundations?
We are discussing the scientific setting.
As a scientific experiment, the results of a clinical trial are used to
rule out (or rule in) hypotheses about treatment effects. The
standards for rejecting (or accepting hypotheses) are based on
statistical criteria.
We need a basic understanding of statistical foundations in order
to discuss the scientific setting and the role of uncertainty.
I will rigorously develop the statistical foundations in chapters 5
and 7 (section 2 of the course).
3
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
3 / 21
(i) What is statistical inference?
Underlying Population
θ denotes unknown center
Inference about θ
Sample
Statistics
Sample summary measure: θ^
4
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
4 / 21
(ii) Four required elements of statistical inference
We use θ̂ (observed trial result) to estimate the true underlying value θ
1. Point estimate: θ̂ is the “best” estimate of θ.
2. Interval estimate: Values of θ that are consistent with the trial
results.
3. Expression of uncertainty (p-value): To what degree is a particular
hypothesis (the “null” hypothesis) consistent with the observed
trial results?
4. Decision: Based on the above measures, what decision should be
reached about the use of a new therapy?
5
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
5 / 21
Example
Daptomycin versus Standard Therapy for Bacteremia and Endocarditis
Caused by S. aureus (Fowler, VG. NEJM 355: 653-65).
“...a successful outcome was documented for 53 of 120 patients
who received daptomycin as compared with 48 of 115 patients who
received standard therapy (44.2 percent vs. 41.7 percent; absolute
difference, 2.4 percent; 95 percent confidence interval, −10.2 to
15.1 percent).”
Note: p = 0.71
6
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
6 / 21
Example (con’t)
Setting:
I
I
I
Primary endpoint: successful outcome at 42 days
Summary of outcome: mean success rate denoted by
θ1 (daptomycin) and θ0 (standard care)
Measure of treatment effect: difference in success rates: θ = θ1 − θ0 .
Observed effect:
I
I
Observed summary outcomes: θ̂1 = 0.442; θ̂0 = 0.417
Observed treatment effect: θ̂ = θ̂1 − θ̂0 = 0.0243.
Inference:
I
I
I
I
Point estimate: 0.0243
Interval estimate: -10.2% to 15.1%
Uncertainty: p = 0.71
Decision: to use or not to use?
7
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
7 / 21
(iii) Properties of estimators
What are the desirable properties of:
Point estimate?
Interval estimate?
P-value?
Decision?
8
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
8 / 21
(iii) Properties of estimators
What are the desirable properties of:
Point estimate?
I
I
Unbiased and consistent: the long-run average of θ̂ is very close to θ
Small variance (Uniform Minimum Variance Unbiased Estimator)
Interval estimate?
I
I
Correct coverage probability (e.g., 95% of all 95% confidence
interval include θ).
As narrow as possible while maintaining the correct coverage
probability.
P-value?
I
Correct size
Decision?
I
Decision criteria maintain the appropriate type I statistical error
rate.
9
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
9 / 21
(iv) Interpretation of an interval estimator
What is the interpretation of a 95% confidence interval?
10
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
10 / 21
(iv) Interpretation of an interval estimator
What’s wrong with the following picture?
θL
θ^obs
θu
11
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
11 / 21
(iv) Interpretation of an interval estimator
Proper depiction of an interval estimator:
θL
θ^obs
θu
If θ < θL or if θ > θU then the observed result θ̂ = θ̂obs would be
unusual.
12
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
12 / 21
(iv) Interpretation of an interval estimator
The scientific objective is to identify the hypotheses that have (or
have not) been ruled out by the trial’s results.
Let U (θref |θ̂obs ) represent a statistical measure of the consistency
between the trial’s result θ̂ = θ̂obs and the hypothesis θ = θref .
By usual frequentist criteria this measure is equal to the smaller of:
P (θ̂ ≥ θ̂obs |θ = θref )
P (θ̂ ≤ θ̂obs |θ = θref )
Reject the hypothesis θ = θref when U (θref |θ̂obs ) is small;
specifically when:
α
U (θref |θ̂obs ) <
2
13
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
13 / 21
(iv) Interpretation of an interval estimator
We seek the values of θ that cannot be rejected; specifically:
Find the set of θref such that U (θref |θ̂obs ) ≥ α/2 using α = 0.05.
If θ̂ ∼ N (θ, V ) then the non-rejection region is given by [θL , θU ]
where
√
θL = θ̂obs − 1.96 V
√
θU = θ̂obs + 1.96 V
(See graph above)
14
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
14 / 21
(iv) Interpretation of an interval estimator
What assumptions are necessary to assure that the above interval has
the correct properties?
15
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
15 / 21
(iv) Interpretation of an interval estimator
What assumptions are necessary to assure that the above interval has
the correct properties?
No assumption about the distribution of the individual data
elements is necessary.
The estimated effect θ̂ must follow a Normal distribution:
I
I
Central limit theorem assures θ is Normally distributed as long as
the sample size is not too small.
For interpretation as a non-rejection region, we must know the
mean-variance relationship.
16
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
16 / 21
A note on mean-variance relationships:
θL
I
I
I
θ^obs
θu
With a mean-variance relationship the confidence interval can have
the correct coverage probability, but may not be a non-rejection
region.
Interventions often change both the mean and the variance.
We will return to this issue in chapter 7.
17
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
17 / 21
(iv) Interpretation of an interval estimator
In summary...
Scientific decisions must consider the magnitude of the effect
(point estimate) and the hypotheses that remain viable based on
the trial’s results (the interval estimate).
I will appeal to these considerations as I describe the scientific
setting.
18
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
18 / 21
(iv) Interpretation of an interval estimator
Example (Daptomycin trial CI: -10.2% to 15.1%):
The treatment success rate with daptomycin was slightly higher
(2.4%) with daptomycin treatment when compared with standard
treatment. The variability in the results allows us to rule out
(with 95% confidence) increases in the success rate with
daptomycin that are larger than 15.1% and decreases in the
success rate greater than 10.2%.
Note: the following is incorrect:
I
There is a 95% chance (or probability) that the true underlying
difference is between -10.2% and 15.1%.
Note: the following is sometimes accepted, but misleading:
I
We are 95% confident that the true underlying difference is between
-10.2% and 15.1%.
19
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
19 / 21
1.1 (c) The scientific method
The scientific method is an iterative process of posing and
evaluating hypotheses using carefully designed experiments.
A clinical trial is an experiment and should be built on
carefully-framed hypotheses:
I
I
I
I
What
What
What
What
is the treatment?
is θ (the measure of treatment effect)?
are important differences?
differences support recommending use of a new treatment?
The trial must be designed to be informative relative to the
hypotheses (the scientist game)
Upon completion the range of viable hypotheses that remain is
determined by the experimental results
20
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
20 / 21
1.1 (c) The scientific method:
The scientist game
Assignment: Try the scientist game:
htpp://www.emersonstatistics.com/ScientistGame
Careful consideration of what you want to know upon trial
completion is essential. The ‘obvious’ choice is often not the best
choice.
The scientist game is illustrative of the scientific importance of all
aspects of the design including:
I
I
I
I
I
I
Specification of the treatment
Selection and definition of the outcome(s)
Choice of control group
Definition of design hypotheses
Statistical standard for evidence
Choice of sample size
21
1. Scientific Setting ()
1.1 Introduction and Motivation
26 Jan 2011
21 / 21