ASSESS
FOR STUDENT
S U CC E S S
PROFESSIONAL
DEVELOPMENT
Multiple Measures
High School
Variables Model
Summary – Phase II
Revis e d Feb ruar y 2 0 17
M M AP Re s earch Team
www.rpgroup.org
www.ed results.org
EDUCATIONAL
RESULTS
PARTNERSHIP
3536 Butte Campus Dr.
Oroville CA 95965
530-879-4170
CCCAssess.org
A S S E S S F O R S T U D E N T S U CC E S S
Table of Contents
Purpose of This Report............................................................................................................................................................................................. 3
The Importance of High School/College Collaboration............................................................................................................................ 3
Summary of Methods............................................................................................................................................................................................... 3
Table 1. Crosswalk of Variable Names for Math Decision Rules for Transfer-Level Mathematics
Courses for Science, Technology, Engineering, and Mathematics (STEM) Pathways........................................................ 5
Table 2. Math Decision Rules for Transfer-Level Mathematics Courses for Science, Technology,
Engineering, and Mathematics (STEM) Pathways........................................................................................................................................6
Table 3. Crosswalk of Variable Names for Math Decision Rules for Transfer-Level Mathematics
Courses for General Education (General Education/Liberal Arts) Pathways............................................................................. 7
Table 4. Math Decision Rules for Transfer-Level Mathematics Courses for General Education
(General Education/Liberal Arts) Pathways........................................................................................................................................................8
Table 5. Crosswalk of Variable Names for English Decision Rules ..................................................................................................9
Table 6. English Decision Rules.................................................................................................................................................................................10
A Note on the Reading and ESL Decision Rules.....................................................................................................................................10
Table 7. Crosswalk of Variable Names for Reading Decision Rules................................................................................................ 11
Table 8. Reading Decision Rules for Schools with a Top Reading Level of One-Level Below Transfer.............. 12
Table 9. Crosswalk of Variable Names for ESL Decision Rules........................................................................................................... 13
Table 10. ESL Decision Rules for Schools with a Top ESL Level of One-Level Below Transfer.................................... 13
Feedback and Questions...................................................................................................................................................................................... 14
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
A S S E S S F O R S T U D E N T S U CC E S S
Purpose of This Report
This document summarizes the rule sets from Phase II of the Multiple Measures Assessment
Project (MMAP), part of the Common Assessment Initiative (CAI). Included in this report are a set of
rules for predicting success in community college English, math, reading, and ESL courses based
upon high school transcript data. These rules can be used to inform placement of students for
whom high school performance data is available.
The Importance of High School/College Collaboration
Please note that while every effort has been made to create valid predictive models, improving
the college success of former high school students need not be approached as a statistical
problem alone. Collaboration among high schools and colleges leading to articulation of course
sequences and pathways is highly likely to outperform predictive analytics in increasing success
and throughput rates for our students. This document is intended to be a final draft of the Phase II
high school performance multiple measures rule sets.
Summary of Methods
This document provides decision tree output used for the creation of high school transcriptbased placement rules for Phase II of MMAP. Interpretation of these decision trees benefits from
knowledge of high school and community college course sequencing, database codes, and policy
and practice contexts. This handout is not intended to be a complete guide to all of these issues.
The Phase I progress report provides historical context leading up to the Phase II effort and can
be found here: http://rpgroup.org/Our-Projects/All-Projects/Multiple-Measures/Publications. An
example of how to interpret the CB21 codes used for each tree is included here:
Highest Level English Course Offered
Level of Decision Tree
Transfer Level One-Level Below Two-Levels Below
Transfer Level (CB21=Y)
L0 Y
One-Level Below (CB21=A)
L0 A
L1 A
Two-Levels Below (CB21=B)
L0 B
L1 B
L2 B
Three-Levels Below (CB21=C)
L0 C
L1 C
L2 C
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
3
A S S E S S F O R S T U D E N T S U CC E S S
Key differences between Phase I and Phase II decision tree creation include:
Phase I Phase II
Predicted grade points in target college course
Predicted success rates in target college course
Used ANOVA-based classification and
regression trees
Used Poisson-based classification and
regression trees
Each college course level modeled
independently
Each college course level modeled recursively
Independent sets of decision trees and decision rules were developed for each level, using the same
recursive Poisson method as is used to grow the math, English, reading and ESL trees. In order to
prevent overlapping or redundant decision rules, the Phase II decision trees were run recursively,
meaning that once a group of students has been placed in a higher level course, they are no longer
eligible to be included in models developing placement rules for subsequent course levels in the
discipline (since they’d already effectively been assigned to a higher level course). This recursive
process was repeated for each progressively lower level of the sequence. Another innovation in
Phase II involved changing the model estimation method from ANOVA, which estimates the mean
grade points that would be expected of groups of students with a given set of characteristics, to the
Poisson method, which estimates the proportion of students with a given set of characteristics who
would pass any given college class with a C grade or higher.
The proportion of students who are predicted to pass can then be used as a criterion level to
identify which decision rules would qualify students for placement into a given course or course
level (e.g., a group had to have at least 70% of its members pass - a proportion of 0.70 - in order to
be included as a decision rule for placement into transfer level, 65% for placement into one-level
below, etc.). This means the groups of students identified as eligible to be placed at that level would
have an average success rate higher than the criterion level.
Decision rules are formed by selecting leaves from the decision tree that meet or exceed a certain
minimum average probability of success (i.e., the criterion level). The criterion represents the
minimum average successful completion rate (i.e., success rate) of groups of students that the
model places into that level of courses. Importantly, because it selects groups of students whose
average minimum successful completion rate is at the criterion or higher, the actual average for
students placed at each level will often be above that rate, sometimes a fair amount above the
average because it will include students whose success rate is substantially above the criterion.
Typical criterion levels for math decision trees are as follows:
Transfer-Level (L0) criterion 0.70
One-Level Below Transfer (L1) criterion 0.65
Two-Levels Below Transfer Level (L2) criterion 0.60
Three-Levels Below Transfer Level (L3) criterion 0.55
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
4
A S S E S S F O R S T U D E N T S U CC E S S
Three sets of trees were developed based on the availability of data from the senior year of high school
as well as CST scores: one set of trees for direct matriculants (DMs) used data through 11th grade,
another for non-direct matriculants (NDMs) used data through 12th grade, and a third for non-direct
matriculants with available CST scores (NDMs with CST).
• Direct Matriculant Models (11th grade): These models utilize cumulative high school grade point
average (GPA) as of the completion of 11th grade and high school course enrollments and grades up
to 11th grade as predictors of successful completion of courses in each discipline at the community
college. This model would be used with most students who are transitioning directly to college from
high school, as they will be assessing, matriculating, and even enrolling before the results from their
12th grade courses are available.
• Non-Direct Matriculant Models (12th grade): These models utilize cumulative high school GPA as
of the completion of 12th grade, high school course grades through 12th grade, California Standards
Test (CST) information as predictors of successful completion of courses in each discipline at the
community college. These models would be used for students who had more than a one semester
gap between their high school exit and community college entry.
It should be noted, that these rules operate under the assumption that colleges will continue to use
additional methods such as a test or informed self-placement to place students in the course section,
as not every community college student can be successfully placed with the current method. For
example, colleges that use methods like the EAP or the SAT/ACT will likely need to continue to use
these additional methods to make sure they maintain the capacity to place every student.
Table 1. Crosswalk of Variable Names for Math Decision Rules for Transfer-Level Mathematics
Courses for Science, Technology, Engineering, and Mathematics (STEM) Pathways
Variable Name
Variable Description
hs_11_gpa /
hs_12_gpa
Cumulative high school (hs) grade point average (gpa) up through
11th (hs_11_gpa) or 12th grade (hs_12_gpa) as indicated.
Pre-Calculus C (or better) / PreCalculus C+ ) or better) / Calculus /
Algebra II B (or better) / Statistics
(with C or better)
Completed Pre-Calculus (or a higher level course) in hs with a C or
better / Completed Pre-Calculus (or better) in hs with a C+ or better /
Took Calculus in hs (did not need to pass) / Completed Statistics (or
better) in hs with a C or better
CST
Scaled score on the Math California Standards Test (CST) typically
taken in 11th grade of high school. The CST is no longer being offered
as of 7/1/2013.
A full list of variable names used in the models and their definitions can be found here: http://bit.ly/2citDJW.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
5
A S S E S S F O R S T U D E N T S U CC E S S
Table 2. Math Decision Rules for Transfer-Level Mathematics Courses for Science, Technology,
Engineering, and Mathematics (STEM) Pathways
Level
(Minimal final HS
course level required
for placement)
Direct Matriculants
(up through 11th grade transcript
available)
Non-Direct Matriculants
(up through 12th grade transcript
available)
Calculus I
Passed Pre-Calculus or
Trigonometry (or higher)
with a C or better
High school 11th grade cumulative
GPA of 3.6 or better
OR
High school 11th grade cumulative
GPA of 3.2 or better AND Pre-Calculus
course grade of C or better
High school 12th grade cumulative
GPA of 3.1 or better AND enrolled in a
Calculus course
OR
High school 12th grade cumulative
GPA of 3.5
Pre-Calculus
Passed Algebra II (or
higher) with a C or better
High school 11th grade cumulative
GPA of 3.4 or better
OR
High school 11th grade cumulative
GPA of 2.6 or better AND enrolled in a
Calculus course
High school 12th grade cumulative
GPA of 3.3 or better
OR
High school 12th grade cumulative
GPA of 3.0 or better AND Algebra II
CST score of 340 or higher
OR
High school 12th grade cumulative
GPA of 3.0 or better AND Calculus
course grade of C or better
Trigonometry
Passed Algebra II (or
higher) with a C or better
High school 11th grade cumulative
GPA of 3.4 or better
OR
High school 11th grade cumulative
GPA of 3.0 or better AND
Pre-Calculus course grade of C+ or
better
OR
High school 11th grade cumulative
GPA of 3.0 or better AND
Algebra II course grade of B or better
High school 12th grade cumulative
GPA of 3.3 or better
OR
High school 12th grade cumulative
GPA of 2.8 or better AND
Pre-Calculus course grade of C or
better
College Algebra
Passed Algebra II (or
higher) with a C or better
High school 11th grade cumulative
GPA of 3.2 or better
OR
High school 11th grade cumulative
GPA of 2.9 or better
AND Pre-Calculus course grade of C or
better
High school 12th grade cumulative
GPA of 3.2 or better
OR
High school 12th grade cumulative
GPA of 3.0 or better AND
Pre-Calculus or Statistics course
grade of C or better
Colleges
103
103
Students
31,286
31,286
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
6
A S S E S S F O R S T U D E N T S U CC E S S
Table 3. Crosswalk of Variable Names for Math Decision Rules for Transfer-Level Mathematics
Courses for Math Decision Rule for Transfer-Level Mathematics Courses for General Education
(General Education/Liberal Arts) Pathways
Variable Name
Variable Description
hs_11_gpa /
hs_12_gpa
Cumulative high school (hs) grade point average (gpa) up through
11th (hs_11_gpa) or 12th grade (hs_12_gpa) as indicated.
Pre-Calculus C (or better) / Statistics
C (or better)
Student completed Pre-Calculus in hs with a C or better, or a higher
level course or completed Statistics in hs with a C or better, or a
higher level course.
CST
Scaled score on the Math California Standards Test (CST) typically
taken in 11th grade of high school. The CST is no longer being offered
as of 7/1/2013.
A full list of variable names used in the models and their definitions can be found here: http://bit.ly/2citDJW.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
7
A S S E S S F O R S T U D E N T S U CC E S S
Table 4. Math Decision Rules for Transfer-Level Mathematics Courses for General Education
(General Education/Liberal Arts) Pathways
Level
(Minimal final HS
course level required
for placement)
Direct Matriculants
(up through 11th grade transcript
available)
Non-Direct Matriculants
(up through 12th grade transcript
available)
Statistics
Passed with a C or better
Algebra I (or higher)
High school 11th grade cumulative
GPA of 3.0 or better
OR
High school 11th grade cumulative
GPA of 2.3 or better AND Pre-Calculus
course grade of C or better
High school 12th grade cumulative
GPA of 3.0 or better
OR
High school 12th grade cumulative
GPA of 2.6 or better AND Pre-Calculus
course grade of C or better
General Education
Math+
Passed with a C or better
Algebra I (or higher)
High school 11th grade cumulative
GPA of 3.3 or better
High school 12th grade cumulative
GPA of 3.2 or better
OR
High school 12th grade cumulative
GPA of 2.9 or better AND Statistics C
or better
High school 11th grade cumulative
GPA of 2.8 or better
High school 12th grade cumulative
GPA of 2.9 or better
OR
High school 12th grade cumulative
GPA of 2.5 or better AND Algebra II
CST score of 302 or higher
OR
High school 12th grade cumulative
GPA of 2.5 or better AND PreCalculus course grade of C or better
Algebra I2
(CB21=B)
(No minimal final HS
course level required for
placement)
High school 11th grade cumulative
GPA of 2.4 or better3
High school 12th grade cumulative
GPA of 2.5 or better
OR
High school 12th grade cumulative
GPA of 2.0 or better AND Algebra I
CST score of 302 or higher3
Pre-Algebra4
(CB21=C)
(No minimal final HS
course level required for
placement)
High school 11th grade cumulative
GPA of 2.0 or better
High school 12th grade cumulative
GPA of 2.1 or better*
Arithmetic
(CB21=D)
(No minimal final HS
course level required for
placement)
Everyone Else
Everyone Else
Colleges
111
111
Students
117,201
117,201
Algebra II1
(CB21=A)
Passed with a C or better
Algebra I (or higher)
+ Transfer Level Course (CB21 = Y) such as “MATH 100. MATHEMATICS FOR GENERAL EDUCATION”
Criterion = 0.70 for transfer level courses unless otherwise indicated
1
Criterion = 0.65
2
Criterion = 0.60 unless otherwise indicated
3
Criterion = 0.56
4
Criterion = 0.54
* Rule for L0 C PreAlgebra NDM CST in previous versions was removed due to inability to replicate
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
8
A S S E S S F O R S T U D E N T S U CC E S S
Table 5. Crosswalk of Variable Names for English Decision Rules
Variable Name
Variable Description
hs_11_gpa /
hs_12_gpa
Cumulative high school (hs) grade point average (gpa) up
through 11th (hs_11_gpa) or 12th grade (hs_12_gpa) as indicated.
hs_11_course_grade_points /
hs_12_course_grade_points
Grade points in high school (hs) English course in 11th (hs_11_
course_grade_points) or 12th (hs_12_course_grade_points) grade
as indicated.
A = 4 grade points
B = 3 grade points
C = 2 grade points
D = 1 grade point
F = 0 grade points
CST
Scaled score on the English California Standards Test (CST)
typically taken in 11th grade of high school. The CST is no longer
being offered as of 7/1/2013.
CB21 Value
Y = Transfer Level
A = One-Level Below Transfer
B = Two-Levels Below Transfer
C = Three-Levels Below Transfer
A full list of variable names used in the models and their definitions can be found here: http://bit.ly/2citDJW.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
9
A S S E S S F O R S T U D E N T S U CC E S S
Table 6. English Decision Rules
1
Level
Direct Matriculants
(up through 11th grade
transcript available)
Non-Direct Matriculants
(up through 12th grade transcript
available)
Transfer level
High school 11th grade
cumulative GPA of 2.6
High school 12th grade cumulative
GPA of 2.6 or better
One-level below
High school 11th grade
cumulative GPA of 2.3
High school 12th grade cumulative
GPA of 2.2 or better
AND
High school 12th grade English
course grade of C or better
Two-levels below
High school 11th grade
cumulative GPA of 2.0
High school 12th grade cumulative
GPA of 1.8 or better AND high school
12th grade English course D
(or better)1
OR
English CST score of 288 or higher1
Three-levels below
High school 11th grade
cumulative GPA of 1.4
High school 12th grade cumulative
GPA of 1.7 or better
OR
High school 12th grade cumulative
GPA of 1.5 or better AND English CST
score of 288 or higher
Colleges
111
111
Students
214,394
214,394
Criterion = 0.55
A Note on the Reading and ESL Decision Rules
The Reading and ESL decision trees in this document were created by first organizing all
California Community Colleges (CCC) into sets of colleges that correspond to the different
ways in which the respective curricula are realized across the system. Colleges were grouped
according to the highest CB21 level of their curriculum. For example, colleges with transferlevel ESL that serves as the final course completing the developmental sequence in ESL and/
or are required for satisfaction of the ESL competency are grouped together. An example of
how to interpret the CB21 codes used for each tree is displayed above.
A college will need to determine which level rule set applies to their Reading or ESL sequence.
For example, if a college has an ESL course that is transferable and students place into it, then
they would likely use the top level rule set of transfer-level ESL. But, if a college has a transferlevel ESL course, but a course at that level is not required to satisfy the locally determined
requirements for demonstrating competency in ESL and, typically, students do not place
into it, rather, they place into one-level below and can later voluntarily elect to enroll in the
transfer-level ESL course, the college would be recommended to use the rule set for onelevel below as their top level course. To use the tables in this document, select the set that
corresponds to the highest level or top level ESL or Reading course available at your college.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
10
A S S E S S F O R S T U D E N T S U CC E S S
Table 7. Crosswalk of Variable Names for Reading Decision Rules
Variable Name
Variable Description
hs_11_gpa /
hs_12_gpa
Cumulative high school (hs) grade point average (gpa) up
through 11th (hs_11_gpa) or 12th grade (hs_12_gpa) as indicated.
hs_11_course_grade_points /
hs_12_course_grade_points
Grade points in high school (hs) English course in 11th (hs_11_
course_grade_points) or 12th (hs_12_course_grade_points) grade
as indicated.
A = 4 grade points
B = 3 grade points
C = 2 grade points
D = 1 grade point
F = 0 grade points
CST
Scaled score on the English California Standards Test (CST)
typically taken in 11th grade of high school. The CST is no longer
being offered as of 7/1/2013.
CB21 Value
Y = Transfer Level
A = One-Level Below Transfer
B = Two-Levels Below Transfer
C = Three-Levels Below Transfer
A full list of variable names used in the models and their definitions can be found here: http://bit.ly/2citDJW.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
11
A S S E S S F O R S T U D E N T S U CC E S S
Table 8. Reading Decision Rules for Schools with a Top Reading Level of One-Level
Below Transfer1
Level
Direct Matriculants
(up through 11th grade
transcript available)
Non-Direct Matriculants
(up through 12th grade transcript
available)
Test Out
High school 11th grade
cumulative GPA of 2.8 or better
High school 12th grade cumulative
GPA of 2.5 or better
One-level below
High school 11th grade
cumulative GPA of 2.3 or better
High school 12th grade cumulative GPA of 2.0 or better AND High
school 12th grade English course of
C+ or better
OR
High school 12th grade cumulative
GPA of 2.0 or better AND
English CST score of 274 or higher
Two-levels below
High school 11th grade
cumulative GPA of 1.9 or better
English CST score of 310 or higher+*
Three-levels below
Placement via test or other
locally determined method
English CST score of 234 or higher+
Four-levels below
Everyone else
Everyone else
Colleges
50
50
Total Students
28,157
28,157
Note: placement via test or other locally determined method = too few cases available to grow decision trees or rules.
Note: 102 colleges and 57,552 students were included in the overall Reading models.
+ Based on a minimum of three years of high school data; to increase sample size, some cases are missing 9th or 10th grade data.
1
Criterion = .57
* Previous version included a non CST rule for placement. This rule was removed due to inability to replicate the rule.
1
For additional Reading rule sets see: http://rpgroup.org/Our-Projects/All-Projects/Multiple-Measures/PilotCollegeResources.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
12
A S S E S S F O R S T U D E N T S U CC E S S
Table 9. Crosswalk of Variable Names for ESL Decision Rules
Variable Name
Variable Description
hs_11_gpa /
hs_12_gpa
Cumulative high school (hs) grade point average (gpa) up
through 11th (hs_11_gpa) or 12th grade (hs_12_gpa) as indicated.
hs_11_course_grade_points /
hs_12_course_grade_points
Grade points in high school (hs) English course in 11th (hs_11_
course_grade_points) or 12th (hs_12_course_grade_points) grade
as indicated.
A = 4 grade points
B = 3 grade points
C = 2 grade points
D = 1 grade point
F = 0 grade points
Y = Transfer Level
A = One-Level Below Transfer
B = Two-Levels Below Transfer
C = Three-Levels Below Transfer
CB21 Value
A full list of variable names used in the models and their definitions can be found here: http://bit.ly/2citDJW.
Table 10. ESL Decision Rules for Schools with a Top ESL Level of One-Level Below Transfer2
Level
Direct Matriculants
(up through 11th grade
transcript available)
Non-Direct Matriculants
(up through 12th grade transcript
available)
One-level below
High school 11th grade
cumulative GPA of 2.7 or better
High school 12th grade
cumulative GPA of 2.6 or better
Two-levels below
High school 11th grade
cumulative GPA of 2.2 or better
High school 12th grade
cumulative GPA of 2.4 or better
Three-levels below
Remedial or Non-remedial
English course in 9th, 10th, or
11th grade with a C+ or better
High school 12th grade
cumulative GPA of 1.5 or better1
Four-levels below
Placement via test or other
locally determined measure
Placement via test or other
locally determined measure
Five-levels below
Everyone else
Everyone else
Colleges
30
30
Students
2,752
2,752
Note: placement via test or other locally determined measure = too few cases available to grow decisions trees or rules.
Note: 98 colleges and 8,680 students were included in the overall model for ESL.
1
2
Criterion = 0.59
For additional ESL rule sets see: http://rpgroup.org/Our-Projects/All-Projects/Multiple-Measures/PilotCollegeResources.
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
13
A S S E S S F O R S T U D E N T S U CC E S S
Feedback and Questions
Any feedback and/or questions related to the content of this document can be sent to one or more
of the individuals listed below.
• Mallory Newell
Supervisor, Institutional Research and Planning, De Anza College; The RP Group
[email protected]
• Craig Hayward
The RP Group
[email protected]
• Dan Lamoree
Senior Research and Systems Analyst, Education Results Partnership
[email protected]
• John Hetts
Senior Director of Data Science, Educational Results Partnership
[email protected]
• Loris Fagioli
Research and Planning Analyst, Irvine Valley College; The RP Group
[email protected]
• Peter Bahr
Associate Professor of Education, University of Michigan; The RP Group
[email protected]
• Terrence Willett
Director of Planning, Research, and Knowledge Systems, Cabrillo College; The RP Group [email protected]
• Ken Sorey
Executive Vice President, Educational Results Partnership
[email protected]
Multiple Measures—High School Variables Model Summary, Phase II | RP Group | February 2017
14