Exploring Quasi-Experiments
Lab 5: May 9, 2008
Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated
instruction on motivation and strategy use in reading. Journal of
Educational Psychology, 92, 331-341.
1
Purpose of Research
• Causal relationship
– To identify whether the
Concept-Oriented Reading
instruction (CORI)
intervention produces
greater intrinsic motivation
and strategy-use than
traditional instruction
2
Variables of interest
• The Independent Variables (causes)
– Instructional intervention:
• CORI intervention
• traditional comparison group
• The Dependent Variables (outcomes)
– Intrinsic motivation (operationalized:
curiosity, involvement, and preference for
challenge)
– Extrinsic motivation (operationalized:
recognition and competition)
– Strategy use (operationalized: self-report of
cognitive strategies)
• The Covariates
– Past achievement (operationalized:
standardized reading achievement scores—
CBST/MAT)
3
Participant assignment
• Non-random assignment
into intervention and
comparison groups (i.e.,
into classrooms)
• Classrooms assigned
based on comparable
“subjective matching” of
teachers, students, and
school settings
4
Cook and Campbell’s UTOS
• Units: 3rd and 5th grade lowachieving students.
• Treatment: Conceptoriented reading instruction
(CORI)
• Observations: intrinsic and
extrinsic motivation, and
strategy use (also past
achievement).
• Setting: three different midAtlantic grade schools.
5
Threats to Internal Validity
Specific to Quasi-Experiments
• History
– Some event related to the outcome could
occur to one group but not the other
• Maturation
– Groups may differ in rate of change on the
outcomes prior to treatment
• Instrumentation
– Measurement might change from pre to post
test in only one group
• Statistical Regression
– Variable used to determine group selection
may be unreliable or unstable
6
Threats to Internal Validity
Specific to Quasi-experiments
• History: A celebrity may visit the schools to
discuss the importance of reading; CORI
students may be more susceptible to the
message and therefore show greater gains in
motivation (due to the celebrity, not the
intervention).
• Maturation: Children in the traditional
classrooms could be losing motivation at a
faster rate than those in the CORI classrooms
prior to treatment.
• Instrumentation: Likely not an issue given the
lack of pretest design.
• Statistical Regression: The choosing of
similar classrooms based on teacher, student,
and school make-up could have been based
on inaccurate, unreliable, or temporarily
skewed subjective judgments.
7
Additional
Threats to Internal Validity
• Lack of pretest makes it difficult to say
whether differences are due to CORI or
whether the differences existed at the onset of
the research
• Attrition: 11% for grade 3 and 17% for grade 5
due to moving.
• Resentful Demoralization: teachers or
students could have shown less motivation
knowing that they were not getting the
treatment (p.334/ p. 47)
• Compensatory Rivalry: comparison teachers
may have tried harder because of the study
(mentioned on p. 334 as the John Henry
effect).
• Third variables: Teacher expectations could
have produced results
• Authors did enhance their design by including
extrinsic motivation (p. 332) as a
nonequivalent dependent variable (p. 74).
8
Threats to Construct Validity
• Diffusion of treatment: Traditional teachers
frequently visited the CORI classrooms and
adopted some texts used in the CORI condition.
– The authors did collect video, interview, and
questionnaire data regarding the use of the
treatment integrity in the CORI classrooms.
• Low internal consistency: Several internal
consistencies were low; because reliability puts
a lid on validity, this is cause for concern.
• Mono-operation bias: Motivation is captured
using only aspects of the Motivation for Reading
Questionnaire (MRQ) when other methods may
be possible. (e.g., Teacher-reports on individual
students).
– The authors did address more than one aspect
of motivation (intrinsic and extrinsic)
9
Threats to External Validity
• Schools are chosen based on
need, therefore results may not
generalize to less needy
students/schools.
• Results may not generalize to
other grades
• Results may not generalize to
interventions that are less
intensive.
• Results may not generalize to
areas outside of the mid-Atlantic.
• Results may not generalize to
participants from dissimilar
schools.
10
Threats to
Statistical Conclusion Validity
• Violated statistical
assumptions: Group
administration of
treatment violates
assumption of
independent
observations (p. 49).
– Addressed by analyzing
the data using HLM
11
Improving the research
• Proximal Similarity: The traditional instruction
comparison group could have maintained
integrity and differentiation from CORI
instruction group. The measures could have
more closely approximated intrinsic motivation.
– Researchers did a good job employing similar
settings and units to whom they wished to
generalize.
– Researchers did a good job ensuring that CORI
instruction was being implemented in the CORI
classrooms
• Heterogeneous Irrelevancies: Triangulating
measures of motivation could have improved
construct validity.
– The authors did use two different Units and
three different Settings.
• Causal explanation: Rule out third variables and
include pretest.
12