Quasi-Experimental Designs 101:
What Works?
The Need To Know Team
January 31 – February 1,
2005
Patricia J. Martens PhD
Outline




Reviewing X’s and O’s
Quasi-experimental
time series designs with
comparison groups
The Population Health
Research Data
Repository: what data
do we have?
Brainstorming ideas
Tic Tac Toe anyone?
Key features of study designs
Artificial manipulation?
(experimental or observational)
Experimental:



Are the groups randomly assigned to receive or not
receive the intervention? (randomized controlled trial)
Are the groups selected to be as similar as possible, not
randomly? (quasi-experimental comparison groups)
Research Design Schema
Research Designs
Analytical
Descriptive
Experimental
Observational
Randomly selected
Cross-Sectional
Non-random
(quasi-experimental)
Longitudinal
Case-Control
Prospective
Cohort
Historical Prospective
(Retrospective)
Key Features of
Study Designs

Observational:
–
–
–
Information collected concurrently or over a time
period? (cross-sectional or longitudinal)
If over a time period, i.e. longitudinal, do you go
from exposure to disease (cohort) or from disease
back in time to examine exposures (casecontrol)?
Do you start now and go forward (prospective), or
do you have a “cohort” somewhere in the past
and you follow them forward (historical
prospective)?
Research Design Schema
Research Designs
Analytical
Descriptive
Experimental
Observational
Randomly selected
Cross-Sectional
Non-random
(quasi-experimental)
Longitudinal
Case-Control
Prospective
Cohort
Historical Prospective
(Retrospective)
Study design: observational

Cross-sectional studies
–

Prospective studies
–

studying all factors at once - both the hypothesized
explanatory and outcome variables
going forward in time, following a cohort and observing the
effect of exposure to a future outcome
Case-control studies
–
going backwards in time from the cases/controls to look at
differential exposures
Research Design Schema
Research Designs
Analytical
Descriptive
Experimental
Observational
Randomly selected
Cross-Sectional
Non-random
(quasi-experimental)
Longitudinal
Case-Control
Prospective
Cohort
Historical Prospective
(Retrospective)
Study design: “What Works” proposal

Randomized Controlled (Clinical) Trial
–

Quasi-experimental
–
–
–

designing a specific intervention and randomly assigning
people to receive it or not to receive it
using a comparison group which is not randomly assigned
Each RHA is a comparison group
A quasi-experimental time series with many comparison
groups (all other RHAs in the province)
Diagrammed and described by Campbell & Stanley
(1963)
Let’s play X’s and O’s
X is an intervention
O is an outcome measure
X
O
Let’s play X’s and O’s
O
X
O
Let’s play X’s and O’s
O
O
X
O
O
Let’s play X’s and O’s
R means randomly assigned
R
R
O
O
X
O
O
(pretest-posttest control group design)
Let’s play X’s and O’s
_ _ _ _ means not randomly
assigned (quasi-experimental
comparison)
O X O
-------O
O
Let’s play X’s and O’s
O X O
-------O
O
quasi-experimental pretest- posttest design
(non-randomized control group)
(non-equivalent pretest-posttest comparison
group design)
Examples of a quasi-experimental pretestposttest comparison group study to determine
effectiveness of hospital policy/education program
40
Hospital BFHI Compliance Scores
BFHI Compliance
site
30
intervention
20
control
10
0
1
2
Time (8 month interval)
Arborg
Pine Falls
Split-unit anova:
p=0.0009
Ten Steps and WHO
Code each assigned
4 points, for total
compliance of 44
Martens 2001
Let’s play X’s and O’s
O
O
X
O
O
Time series (quasi experiments)
proportion initiating breastfeeding
Example of a quasi-experimental time series
to determine effectiveness of a
community-based breastfeeding strategy
Breastfeeding Initiation 1992-97
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
1994 Breastfeeding study:
pregnant women interviewed
?
?
CHN at conference,
uses new techniques to
address prenatal feeding
intent
*
?
PC Training
begun ?
??
CHN hired
Video and breastfeedng
booklet completed, used in
individual prenatal
* p<0.05,
instruction by CHN
one-tailed, adjusted
1992 1993 1994 1995 1996 1997
year
Martens
2002
for birth weight
and parity
Let’s play X’s and O’s
Time series (quasi experiment with comparison
group)
O O X O O
--------------O O
O O
Example of a quasi-experimental time series
with comparison groups to determine
effectiveness of a regional teen pregnancy
reduction program
rate of teen pregnancies per 1000
females aged 15-19 years
Figure 4: Sample Analysis of Regions A through D
Teen Pregnancy Rates 1993 through 2002
250
200
150
Region D
Statistically significant
decline in Region A?
Region A
100
Region C
50
0
Region B
1993
1994
1995
1996
1997
1998
Year
1999
2000
2001
2002
From CIHR proposal submission September 2004
Additions of small amounts of
phosphorus to one section of ELA
Lake 226 caused surface blooms
of blue-green algae, and vividly
demonstrated the importance of
phosphate as a cause of
excessive algal growth or
eutrophication. This experiment
spurred legislation controlling the
input of phosphorus to many
water bodies.
A demonstration of the work of
Dr. David Schindler and the Experimental
Lakes project in NW Ontario
http://www.umanitoba.ca/institutes/fisheries/eutro.html
Study design:
Low internal validity

Anecdote/case study

Pre-experimental


just doing a pretest and posttest on one group and
seeing its effect
Cross-sectional

a snapshot in time: can’t tell which comes first, but only
that they are “associated”
Study design:
medium internal validity

Time series; Time series with qualitative layer
–

Case-control
–

looking over time to see change, with information about
when interventions occurred in the time frame
going backwards in time from the cases/controls to look at
different exposures to possible risk factors
Observational (prospective)
–
going forward in time, observing the effect of exposure on
a cohort to a future outcome
Study design:
high internal validity

Randomized Controlled (clinical) Trials, RCT



designing a specific intervention and randomly assigning
people to receive it or not to receive it
following people to observe the outcome of interest
Quasi-experimental comparison group
studies

using a comparison group which is not randomly
assigned, but very similar at onset
High
Internal validity
Randomized Controlled Trials RCT
Quasi-experimental comparison group studies
Time series with comparison
Observational (prospective)
Case-control
Time series with qualitative layer
Low
Cross-sectional
Pre-experimental
Anecdote/case study
“There is nothing
so useless
as doing
efficiently that
which should not
be done
in the first place.”
Peter Drucker
MCHP’s … “paperclips”
“Population Health Research Data
Repository”
Family
Services
Education
Hospital
Home Care
Pharmaceuticals
Census Data
EA/DA level
Cost
PopulationBased Health
Registry
Immunization
Medical
Nursing
Home
Provider
Vital
Statistics
National
surveys
Brainstorming: “What Works” proposal

Pick (a) a policy; and (b) a program
–

What OUTCOME measures would you think this
would impact?
–
–
–

Think of something that your region has done in the past,
somewhere between 1997 and the present (hopefully, with
a few years of data AFTER the onset of this)
Think of what you would expect to see if this intervention
was “working”
Are there specific target groups to which this intervention
applies? (e.g. teens, people living in a certain district of your
region?)
What measures of this intervention would be available
through the Repository data?
Brainstorm and report! (see sheet for recording)
Policy or
Program
Outcome
Target
Measure(s) Group
Outcome
Other
available in comments
Repository
?
Teen
pregnancy
reduction
Teen
pregnancy
rate
pregnancies Maybe birth
control pill
or
live births? use in Rx
data?
12-19 year
olds?
Certain
district?