Conducting Randomized Evaluations in
Schools
Lisa Markman
Associate Director
Education Research Section
Princeton University
Education Research Section
•
•
Promotes the use of research in education
decision-making.
Conducts and promotes high quality
education research through the use of
experimental and quasi-experimental
research designs
No Child Left Behind, Schools and
Research
• No Child Left Behind Act (2001) requires
programs and practices to be based on
“scientifically based research”.
• Term “scientifically based research” appears
throughout the law over 100 times.
• Applies to curricular interventions, professional
development programs, psychosocial
interventions and supplemental education
services.
What is Scientifically Based
Research ?
The term scientifically based research
means research that involves the
application of rigorous, systematic, and
objective procedures to obtain reliable
and valid knowledge relevant to
education activities and programs
(According to NJ Dept. of Ed)
Necessary Components of SBR
• Employs systematic, empirical methods that draw on
observation or experiment
• Involves rigorous data analysis that are adequate to
test the stated hypotheses and justify the general
conclusions drawn
• Relies on measurement or observable methods that
provide reliable and valid data
• Is evaluated using experimental or quasi-experimental
designs with appropriate control or comparison groups
(with a preference for random assignment)
• Reports results with sufficient detail and clarity
• Has been subject to external (peer) review
(According to NJ Dept. of Ed)
Evaluations in Education
Undesirable Features of Many
Evaluations
Desirable features of Evaluations
• No comparison group or
comparison group is not
comparable to treatment group
• Participants are randomly
assigned to “treatment” status
• Only “successfully” treated
schools/students are included
in the analysis
• All initial participants are
included in the analysis
• The evaluators are not
independent of the developers
of the program
• The evaluators are
independent of the developers
of the program
Recipe for an Ideal Measurement
Step 1:
Step 2:
Step 3:
Step 4:
Step 5:
Randomly assign students.
Obtain PERFECT test.
Test students in fall.
Test students in spring.
Set non-school parameters back to original
conditions.
Step 6: Repeat Steps 3 & 4.
Step 7: Impact =
(Step 4 – Step 3) Year 2 - (Step 4 – Step 3) Year 1
The “2nd Best” Option
Randomization
Randomly assign some
students, classes,
school districts, or states
to the reform in
question; the others are
randomly assigned to
the status quo (or
another reform).
Why Randomize ?
To obtain an appropriate
control group
Allows one to answer what
would happen in the absence
of the intervention
(creates a credible
counterfactual)
Common Concerns About
Randomized Evaluations
• It is too disruptive
• Unethical to deny service to some children
• Legal obligations require providing
services to all children
• It does not answer all questions
• Fear of “random”
Is It Too Disruptive ?
• It can be.
• Good researchers work with teachers,
schools and school districts to
understand school culture, and school
needs.
Example of a Randomized
Research Evaluation
• To determine the effectiveness of a
computerized math curriculum
• 3 Urban School districts
• Randomize on the class level by
school
Randomized Evaluation of A Computerized Math Curriculum
Current School Schedule
Teacher
Period 1
Period 2
Period 3
A
Alg 1
Alg 1A
Alg 1
B
Alg 1
Alg 1A
C
D
Alg 1A
Alg I
Period 4
Alg 1A
Alg I
Alg 1
Alg 1A
Randomized Evaluation of A Computerized Math Curriculum
Current School Schedule
Teacher
Period 1
Period 2
Period 3
A
Alg 1
Alg 1A
Alg 1
B
Alg 1
Alg 1A
C
D
Alg 1A
Alg I
Period 4
Alg 1A
Alg I
Alg 1
Alg 1A
Below is a sample schedule that would be returned to each school after random
assignment
Teacher
Period 1
Period 2
Period 3
A
Regular
Regular
LAB
B
LAB
LAB
C
D
Regular
Regular
Period 4
Regular
Regular
Regular
LAB
It’s Unethical/Illegal to Deny
Service
• We don’t know if program/reform actually
works
• Ways around completely excluding all
students
• All children receive “something” or can
contrast two programs
It Doesn’t Answer All Questions
• True
• Benefit of Mix-Methods Approach
The Fear of “Random”
• Usually can not treat all students
• Way to organize groups to get meaningful
information
To Sum….
• Need more high-quality impact evaluations
to help guide everyday decisions
• Conducting evaluations does not have to
be super disruptive
• Schools should make sure that
researchers work with them
• It’ll take cooperation from all to improve
the quality of education research on
impacts
Information for Librarians
• What Works Clearinghouse
• Education Databases
What Works Clearinghouse
• Created as a resource for decision makers in education. Provides
databases and reports regarding the effectiveness scientifically
based programs and practices.
• Current topics –
–
–
–
–
–
–
–
–
–
Adult Literacy,
Character Education,
Delinquency, Disorderly and Violent behavior,
Dropout Prevention,
English Language Learners,
Math (Elementary and Middle School)
Peer Assisted Learning
Reading,
Early Childhood
• http://www.whatworks.ed.gov
Examples of Education Databases
• National Center for Education Statistics (NCES)
– National Assessment of Educational Progress (NAEP)
– High School and Beyond (HS&B)
– The Common Core of Data (CCD)
– http://nces.ed.gov
• National Longitudinal Study of Youth (NLSY)
– NLSY97
– NLSY79
– http://www.bls.gov/nls
– Check websites for a list of all databases