Multiple Measures Assessment Project:
Math & English Models from Phase II
Multiple Measures Pilot College
Convening - North
Cañada College
Terrence Willett
October 28, 2015
MMAP variables for predicting CC success
English
•
•
•
•
Cumulative HS GPA
AP English class grade
Score on English CST
Grade in last English class
Math
• Cumulative HS GPA
• Grade in Algebra I,
Algebra II, Statistics,
Trigonometry,
Calculus
• Taking a challenging
CST
• Score on math CST
Why Decision Trees?
•
•
•
•
•
Provide easy-to follow and easy-to-code rule sets
Create discrete categories
No distributional or relational assumptions
Visual presentation of decisions/decision-making
They’re fun!
Increasing
Homogeneity
with each split
ABZ
ZAB
Root Node
BZA
Branch
Internal Node
AAA
ABA
ZZZ
BAB
BBB
Leaf Node
Statistics Tree – Direct Matriculants
Statistics Tree – NDM
English Decision Rules
Direct matriculants
Non-direct matriculants
Transfer-level
(criterion = 0.70 / 0.70)
HS 11 GPA >= 2.6
HS 12 GPA >= 2.6
One-level below
(criterion = 0.65 / 0.65)
HS 11 GPA >= 2.3
HS 12 GPA >= 2.2 &
HS 12 English course GPs >= 1.8
Two-levels below
(criterion = 0.60 / 0.55)
HS 11 GPA >= 2.0
HS 12 GPA >= 1.8 &
HS 12 English course GPs >= 1.1
HS 12 GPA >= 1.8 & CST >= 288
Three-levels below
(criterion = 0.55 / 0.55)
HS 11 GPA >=1.4
HS 12 GPA >=1.5 & CST>=268
HS 12 GPA >=1.7
Four-levels below
Everyone else
Everyone else
Level (CB21
code, criterion)
Hardcoded
PreRequisite
Algebra II
Pre-Calculus
(criterion = 0.70) (or better)
Direct Matriculants
(Up through 11th grade,
no CST)
Non-Direct
Matriculants
With CST
Non-Direct
Matriculants
Without CST
HS GPA>=3.4
HS GPA>=3.3
HS GPA>=3.3
HS GPA>=2.6 AND Calculus HS GPA>=3 AND
Statistics
(criterion=0.70)
Algebra I
(or better)
HS_11_GPA >= 3.0
CST>=340
HS GPA>=3 AND Calculus
C or better
HS_12_GPA >= 3.0
HS_12_GPA >= 3.0
HS_11_GPA >= 2.3 AND
Pre-Calculus with C or
better
Algebra I
General
Education Math (or better)
(Transfer Level)
(criterion=0.70)
HS GPA >=3.3
HS GPA>=3.2
HS GPA>=3.2
OR
HS GPA>=2.9 AND Stat C
or better
Algebra II
(or better)
HS GPA >=3.2
HS GPA>=3.2
HS GPA>=3.2
College Algebra
(criterion=0.70)
HS GPA >=2.9
AND
Pre-Calculus C or better
Higher level math rules
Hardcoded Pre- Direct Matriculants
Level (CB21
code, criterion) Requisite
(Up through 11th
grade, no CST)
Non-Direct
Matriculants
With CST
Non-Direct
Matriculants
Without CST
Calculus I
(criterion =
0.70)
HS GPA>=3.1 AND
Calculus C or better
HS GPA>=3.1 AND
Calculus
HS GPA>=3.5
HS GPA>=3.5
Pre-Calculus
or
Trigonometry
(or better)
HS GPA>=3.6
HS GPA>=3.2 AND PreCalculus C or better
Algebra II
(CB21=A,
criterion=0.65)
HS 11 GPA >=2.8
HS 12 GPA >=2.9
HS 12 GPA >=2.9
HS 12 GPA >=2.5
AND
Algebra II CST >= 302
HS 12 GPA >=2.5 AND
Pre-Calculus with C or
better
Algebra I
HS 11 GPA>=2.4
(CB21=B,
criterion = 0.55)
HS 12 GPA>=2.5
HS 12 GPA>=2.3
AND
Algebra II C or better
Pre-Algebra
(CB21=C,
criterion 0.50)
HS 11 GPA>=2.0
HS 12 GPA >=2.1
Arithmetic
(CB21=D)
Everyone else
Basic Skills Math Decision Rules
HS 12 GPA>=2.0
AND
Algebra I CST >= 302
HS 12 GPA >=2.1
HS 12 GPA >=1.6
AND
Algebra 1 Cluster 1
(Number Properties,
Operations, and
Linear Equations) >= 8
Everyone else
Everyone else
Correlations with College Grade:
11th grade Cumulative GPA vs. Accuplacer
English
11th Grade GPA
Accuplacer
Transfer
0.27
0.24
0.25
0.18
0.21
0.10
0.12
0.12
0.12
0.07
1 level below
2 levels below
3 levels below
4 levels below
Math
Transfer - STEM
Transfer – Stats
Transfer – LAM
1 level below
2 levels below
3 levels below
4 levels below
11th Grade GPA
Accuplacer
0.24
0.31
0.26
0.28
0.26
0.23
0.19
0.19
0.16
0.09
0.21
0.11
0.11
0.05
Placement into Transfer-level with
Current Placement Systems vs. Strong Multiple Measures (Statewide)
100%
90%
80%
70%
61%
60%
50%
40%
42%
38%
31%
30%
20%
10%
0%
English
(n=103,510)
Current
Math
(n=143,253)
Disjunctive MM
Transfer Level English Placement
Current
Disjunctive MM
100%
90%
80%
74%
73%
70%
60%
50%
41%
40%
40%
30%
53%
51%
24%
30%
20%
10%
0%
Afr Am
Latino
Asian
White
Transfer Level Math Placement
Current
Disjunctive MM
100%
90%
80%
70%
65%
60%
53%
50%
41%
40%
32%
30%
20%
51%
22%
21%
15%
10%
0%
Afr Am
Latino
Asian
White
How long is High School GPA good for?
Correlation between HSPGA and 1st CC English grade
Decay function for the predictive utility of HSGPA on English grade
0.4
0.35
y = -0.0076x + 0.342
R² = 0.6201
0.3
0.25
0.2
0.15
0.1
0.05
y = -0.0116x + 0.3631
R² = 0.8336
0
0
1
2
3
4
5 6 7 8 9 10 11 12 13 14 15 16 17 18
Semesters of delay (approx. 6 months each)
HS 11 GPA
HS 12 GPA
Linear (HS 11 GPA)
Linear (HS 12 GPA)
Accuplacer
MMAP Research Team Contacts
Mallory Newell
The RP Group
[email protected]
John Hetts
Educational Results Partnership
[email protected]
Terrence Willett
The RP Group
[email protected]
Ken Sorey
Educational Results Partnership
[email protected]
Craig Hayward
The RP Group
[email protected]
Daniel Lamoree
Educational Results Partnership
[email protected]
http://www.rpgroup.org/projects/multiple-measures-assessment-project
Predicting transfer-level grades
Test
Correlation with
GPs
Corrected
correlation
CTEP Grammar
.10
.16
CTEP Reading
.06
.09
CTEP Syntax
.08
.12
Math Assessment Test
.21
.29
11th Grade GPA - English
.38
.39
11th Grade GPA – Math
.37
.38
Cost
Old model
Semesters
Units
Cost to get all to Engl 1A
Placement at Transfer level
Acceleration
MIH + Acceleration
3
2
1
10
6
4
$1,677,552
$1,084,682
$208,851
18%
29%
57%