Quasi Experimental
Designs
Chapters 4 & 5
Som Nwegbu and Binh Le
H615: Advanced Evaluation and Research Design
October 18, 2013
A Few Helpful Definitions
Quasi – “seemingly; apparently but not really.”
Synonyms: supposedly, seemingly, apparently, allegedly,
ostensibly, on the face of it, on the surface, to all intents and
purposes, outwardly, superficially, purportedly, nominally”
Experiment – “A test under controlled conditions that is made to
demonstrate a known truth, examine the validity of a
hypothesis, or determine the efficacy of something previously
untried.”
Quasi Experiment – Why is it ‘quasi’ ?
It’s “…an experiment in which units are not assigned to conditions
randomly.” (SCC)
Quasi-Experimental Designs
Lack a Control Group or Lack Pretest Observations on
Outcome
Why use designs?
Devote more resources to construct validity and external
validity
Necessities imposed by funding, ethics, administrators, or
logistical constraints
Sometimes the best design for the study, even if causal
inference might be weaker
Logic of Quasi Experiments
Causal inference must meet requirements: That cause precede
effect, that cause covary with effect, and alternative
explanations unlikely.
•
•
•
Randomized and quasi-experiments manipulate treatment to
force it to occur before the effect
Covariation between cause and effect accomplished during
statistical analysis
Alternative explanations implausible by ensuring random
distribution
Identification and study of plausible threats to internal validity
Primary of control by design
Coherent pattern matching
Quasi Experimental Designs
w/o Control Groups
One-Group Posttest-Only
X
O1
Examples…
Weaknesses…
One-Group Posttest-Only with Multiple Posttests
X1
Examples…
Weaknesses…
{O1A O1B…O1N}
Quasi Experimental Designs
w/o Control Groups
One-Group Pretest-Posttest
O1
X
O2
Examples…
Weaknesses…
One-Group Pretest-Posttest Using Double Pretest
O1
O2
X
O3
Examples…
Weaknesses…
One-Group Pretest-Posttest Using Nonequivalent Dependent Variable
{O1A
,
O1B}
X
{O2A
,
O2B}
Quasi Experimental Designs
w/o Control Groups
Removed-Treatment
O1
X
O2
O3
χ
O4
Examples…
Weaknesses…
Repeated-Treatment
O1
Examples…
Weaknesses…
X
O2
χ
O3
X
O4
Quasi-Experimental Designs w/Control
Groups but no Pretest
Posttest-Only Design with Nonequivalent Groups
NR
NR
X
O1
O2
Posttest-Only Design Independent Pretest Sample
NR
NR
O1 | X
O1 |
O2
O2
Posttest-Only Design Proxy Pretests
NR
NR
OA1
OA1
X
OB2
OB2
Improving the Posttest-Only
Design
Using Matching or Stratifying
Internal Controls
Multiple Control Groups
Predicted Interaction
Constructing Contrasts other than
with Independent Groups
Regression Extrapolation Contrasts
Compares obtained posttest score of the
treatment with the score predicted from other
information
Normed Comparison Contrasts
Treatment group at pretest and posttest
compared with published norms
Secondary Source Contrasts
Construct opportunistic contrasts from secondary
sources
Case Control Design
Also called case-referent, case-comparative, case-history,
or retrospective design
One group of cases with outcome of interest and another
group of controls without outcome
Typically dichotomous outcome
Generating hypotheses about causal connections
More feasible than experiments in cases, logistically
easier to conduct, decrease risk of participants, and easy
examination of multiple causes
Case Control Design
Methodological problems
Decision on what counts as the presence or absence of an
outcome
Disagreement about the decision, if they do, assessing the
outcome may be unreliable or low validity
Selection of control cases is difficult
Randomly sampled controls are ideal but when not feasible,
matching is the next option
Matching controls can still differ from cases in unobserved
ways
Threats to Validity
Reading on the field(5):
One-sided reference bias
Positive results bias
Hot stuff bias
Specifying and selecting the study sample(22):
Diagnostic access bias
Unacceptable disease bias
Membership bias
Executing the experimental maneuvers (5):
Contamination bias
Withdrawal bias
Threats to Validity
Measuring Exposures and Outcomes (13):
Underlying cause bias
Expectation bias
Attention bias
Analyzing the Data (5):
Scale degradation bias
Tidying-up bias
Interpreting the Analysis (6):
Magnitude bias
Significance bias
Correlation bias
Quasi-experimental
Designs that Use Both
Control Groups and
Pretests
Benefits of a Pretest
Addresses the issue of bias resulting from nonrandom selection
NB: However, ‘no difference’ between intervention and
control groups at pretest does not guarantee zero
selection bias.
Gives us a baseline to compare against (statistical
analysis)
Limitations of a Pretest
We cannot assume that any covariates unaccounted
for, but present at pretest, are unrelated to outcome
In a randomized experiment, this would have been
controlled for by random selection.
Key:
NR = Non-random assignment
01 = pre-test
02 = second pre-test (if any)
03 = post-test
X = Test/intervention
X+
} = Reversed treatment
X-
Untreated Control Group Design with
Dependent Pre-test & Post-test samples
NR
O1 X O2
NR
O1
O2
Most common and most basic used
Others (to come) are an attempt to improve
internal validity and vary depending on context
and resources available to the researcher.
Why do you think this is the most commonly used?
Limitations and/or weaknesses
aka ‘Threats to internal validity’
• Selection-maturation (various subtypes)
Pretest difference b/w intervention and con groups increases when
intervention leads to improvement in group that was better to begin with
• Selection-instrumentation
Detectable pretest difference between intervention and control groups pretest started at different points e.g., one group starts at Q50 and another at
Q1
• Selection-regression??? Pg 139***
• Selection History
Events occurring midway b/w pre and posttest affect one group more than the other,
thus widening/narrowing the observed pretest difference.
Possible versus plausible
threat
Possible – might have occurred, but highly unlikely.
Plausible – most probably did occur
We want to be able to, as much as we can, rule out all the
possibles and pursue the plausibles
Question:
How do we know which cases to worry about (plausibles) and
which we can safely ignore (possibles)? (discuss)
Answer:
Analyze results in context
Outside of your study, what do you already know about the
threats?
What is the observed pattern of outcomes:
Groups grow apart in the same direction
No change in control group
Initial pretest difference (in favor of the treatment group) but then
diminishes over time
Initial pretest difference (in favor of control group) but then diminishes
over time
Outcomes that crossover (you wish!)
Point to note
In view of the sub-types of sub-maturation threat,
one must be prepared to present and justify a study’s
assumptions about maturational differences (if using
the basic quasi-experimental design).
Ways to improve on the internal validity of
inferences made using the basic design:
(1) DOUBLE PRETEST
NR
NR
O1
O1
O2
O2
X
O3
O3
Exposes selection maturation where present
Helps reveal regression effects if present
Helps statistical analysis by establishing more precise correlation
between observations. Put simply, it gives us a baseline to
compare against.
(2)SWITCHING REPLICATIONS
NR
O1
NR
O1
X
O2
O2
O3
X
O3
• Treatment is administered at a later time point for the group
that initially served as a control.
• May be employed where it would be unethical to withhold
treatment/intervention (particularly if the treatment has
been proven to be beneficial).
• Helps test both internal and external validity. (Discuss)
(3) REVERSED TREATMENTCONTROL GROUP
NR
O1
X+
O2
NR
O1
X─
O2
•
Advantageous, particularly in terms of potential for improved construct
validity. (Discuss)
•
Allows for ruling out potential of Hawthorne effect.
•
Assumptions/weakness – This design depends on the assumption that there are
no historical or other extrinsic behavior modifying events occurring while the
study is ongoing.
(4) DIRECT MEASUREMENT
OF THREAT
Researcher tries to conceive of every possible threat
to validity and then put in checks to reduce such
threats.
Demerits?
Merits?
MATCHING USING COHORTS AS
CONTROLS
Cohort (in this context) –
“…a group of subjects who have shared a particular event together during a
particular time span e.g., people born in Europe between 1918 and 1939.”
Benefits?
Cost
Convenience
Allows one to make good use of well kept records
where available
Others as outlined in SCC page 149 [paragraph 1]
It gets even better. . .
(Don’t you mean more complicated?)
DESIGNS THAT COMBINE MANY DESIGN ELEMENTS
• Depending on context and need, one may use
combinations of the above examples, or further
variations, to improve internal validity and enable
causal inference (or something close).
• Be prepared to explain and defend it though.
Moving on from threats to
Internal validity to:
Improving Statistical conclusion validity
HDFS 532 (or equivalent) highly recommended
Topics such as, SEM and Selection Bias Modeling
covered in detail.
Further comments or contributions?
To wrap up
We can never be 100% certain of our claims when
using quasi-experimental designs and we must be
prepared to own it and state our limitations upfront.
Still, there are numerous ways to strengthen the
validity of our claims and these can be applied at
different stages of the research study:
Assignment
Measurement
Use of comparison groups
Treatment
Statistical Analysis