LEARNING FROM EXPERIMENTATION IN
DEVELOPMENTAL MATH IN
COMMUNITY COLLEGES IN CALIFORNIA:
RESULTS FROM A LONG-TERM
RESEARCH PARTNERSHIP
The Steinhardt Institute for Higher Education Policy,
New York University, October 19, 2015
Tatiana Melguizo
Associate Professor, University of Southern California
[email protected]
This research was funded by a grant from the U.S. Department of
Education’s Institute of Education Sciences (IES).
Overview




Educational & professional background
Key results from a federally funded long-term
research collaboration with a large urban community
college district in California
Other research interests
Q&A
Inequalities in college access and outcomes
by socioeconomic status in Colombia
Only 25% of students from the lowest
income strata enter college compared
to 60% of students from the highest
income strata
(Melguizo, Sanchez & Velasco, 2015)
Los Andes University
Bogota-Colombia
Increasing inequalities in educational
outcomes in the U.S.
Banning of Affirmative Action
Policies in the U.S. in the mid 1990s
I started my Ph.D. at the
time that Affirmative
Action policies were
banned in California
That major policy shift motivated me to study the impact of attending more
selective colleges and universities for students of color
Findings suggested that students of color both in the 1980s and in the 1990s
had higher graduation rates at the most selective colleges and universities
(Melguizo, 2007)
Findings cited as part of Amicus Brief to the Supreme Court by AERA
Creating a Research Partnership to Study
Assessment and Placement in Developmental Math

The purpose of the partnership is to inform developmental
education research, policies, and practices
 Site: The Los Angeles Community College District
 Research Focus: To examine the impacts and
implementation of test-based, alternative placement policies,
and delivery methods for developmental math
 Research approaches: Experimental and quasiexperimental, qualitative, & descriptive
Strength of partnership built on
commitment & unique expertise of
partners


A joint commitment to improve achievement in dev.
ed. through policy and practical change
An firm belief that each partner is an equal; brings
unique contextual, technical expertise
 LACCD:
Knowledge about local context, access to
administrative records, faculty, and administrators
 USC: A strong record of community college research
 AIR: Technical expertise in conducting experimental and
quasi-experimental research
Problem Statement
•
Every year about 80 percent of community college students in
California are placed into preparatory mathematics. This percentage
is higher than the national average
• Community college students have widely varying initial skills levels
• Colleges have to offer classes to meet these levels and have to keep
heterogeneity in the classrooms manageable
• Placing students incorrectly can reduce the likelihood that students
succeed
Why LACCD?
Los Angeles Community College District a natural laboratory





Diverse student population that varies by college.
Nine colleges with 130,000 plus students.
“Common data system.”
Large number of observations.
Presumption of representativeness—likely to capture
the wide variation across community colleges in the
United States.
Literature on Impacts of Remedial
Education
•
•
Proponents argue that remedial education
provides the preparation necessary for
students to succeed in college (Boylan,
Bliss, & Bonham, 1994; 1997; Lazarik,
1997)
Critics contend that the benefits that
students obtain are not clear (Calcagno &
Long, 2008; Martorell & McFarland, 2011;
Scott-Clayton & Rodriguez, 2015).
Developmental Math Sequence
Arithmetic
Pre-Alg.
Elem. Alg.
Developmental Math
Int. Alg.
TransferLevel
Remediation needs of LACCD students
Five Key Findings
Finding 1: Establishing an effective A&P system is complex.
More support and training is needed for faculty and administrators charged
with
this task. (Melguizo, Kosiewicz, Prather & Bos, 2014).
Finding 2: The largest barrier for developmental math students is attempting their
initial course (Fong, Melguizo, & Prather, 2015).
Finding 3: Community college faculty and administrators have the opportunity to
improve placement and success in developmental math by engaging in a systematic
process of calibration of the cut scores of assessment and placement tests
(Melguizo, Bos, Ngo, Mills & Prather, 2015).
Finding 4: The diagnostic test places students more accurately than the computeradaptive test (Ngo & Melguizo, 2015).
Finding 5: The inclusion of multiple measures in the placement process can increase
access to higher-level math without decreasing students’ chances of success (Ngo
& Kwon, 2014; Fong & Melguizo, 2015).
F2: Over 30 percent of students are NOT
attempting the assigned courses
F3: Systematic Process of Calibration of the Cut
Scores




Math faculty set the cut points between the
different levels based on who applies and
how their course offerings are distributed
If the cut points are too high, too many
students languish in remedial courses
If the cut points are too low, too many
students fail higher-level courses and present
a challenge to the instructors
Getting the cut points just right is important
F3: Different Pathways to
Success
(Arithmetic vs. Pre Algebra)
Placed in
Enroll in
Success
PreAlgebra
PreAlgebra
Failure
No
enrollment
Test
Placed in
Arithmetic
Enroll in
Arithmetic
Success
Failure
Next
course
F3: Ideal Regression Discontinuity
Situation



Regression discontinuity analysis is the strongest
non-experimental method to estimate causal
effects
It depends on a continuous forcing variable and an
exogenously established cut point
Those two conditions are present in this situation
Outcome
Impact
Arithmetic
Pre-Algebra
Placement Test Score
F3: Negative Impact of Starting in AR
Time Dimensions is Key
35
College A
College B
30
Impact on cumulative % of students passing pre-algebra
College C
College D
25
College F
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10
-5
-10
Quarter after Initial Placement Test
11
12
13
14
15
16
F4: Placement Accuracy:
Diagnostic versus Computer-AdaptiveTests
Do diagnostics improve placement accuracy?
Methods
I.
Logistic Regressions

II.
Placement Accuracy

III.
Predicting EA outcomes with skill-specific math
information
Sum of accurate placements
Regression Discontinuity


Traditional (with single-score)
Binding-score (with multiple criteria)
Placement Accuracy
Table 3. Percent of accurate placements by level of developmental math.
AR vs PA
PA vs EA
EA vs IA
College A
73.2
75.3
72.9
College B
-
75.5
48.4
College C
19.8
22.6
24
College D
77.8
47.2
71.4
College G (w/subscores)
51.6
76.3
56.1
College G (w/o subscores)
49.9
68.2
44.9
College H
90.0
65.5
54.1
ACCUPLACER Colleges
MDTP Colleges
Note: Placement accuracy calculated using method described by Scott-Clayton (2012). Percentage shown is the sum of the proportion of students predicted to pass higher-level course
and placed there plus the proportion of students not predicted to pass the higher-level course and placed in the lower level course.
F4: Diagnostic tests are Placing Students more
Accurately than Computer-Adaptive Tests



Students placed using results from computeradaptive tests were more negatively impacted by the
placement decision than prior cohorts placed by
MDTP.
Students were less likely to enroll and persist onto
the next math course after the placement test switch.
Consistent with other studies, we found that the
diagnostic test can provide information on student
proficiency on a range of subtopics such as fractions,
exponents, and reasoning which can improve math
placement decisions and/or tailor instruction in math
F5: The Inclusion of Multiple Measures can Increase
Access w/out Decreasing Student Success
F5: Findings



Only 6% of the students benefitted from multiple
measures at the LACCD
Major benefits for African American and Latino
students who could enroll in higher-level math
courses
We found no evidence that “boosted” students were
less likely to complete the course
Conclusions



Engaging in research partnerships with large districts
are a way of increasing research capacity of the
district while gaining a much more nuanced
understanding of the context for the researcher
Research findings not only contribute to the
knowledge but are also policy relevant and
actionable
Researcher-practitioner partnerships can be
conducive to high-quality and high-impact research
Other Current Projects



Mixed-methods evaluation of the Susan Thompson
Learning Communities. Buffett Foundation
Using high school transcript information to refine A&P
in math. LAUSD-LACCD-USCNSF: EAGER
Accurately estimating Student Learning Outcomes
(SLOs) in higher education
 Brazil
(Melguizo & Wainer, under review)
 Colombia (Melguizo, Zamarro, Sanchez & Velasco, 2015)

Proposed an RCT evaluation to test student selfplacement in Dev Ed Math
THANK YOU!
Questions
Tatiana Melguizo
[email protected]
http://www.uscrossier.org/pullias/research/projects/s
c-community-college/