Presentation Outline
• The Future of Institutional Research (IR)
& Technology in improving first-year
students’ success.
• Example 1: Demonstration of an IR
innovation.
• Example 2: Demonstration of a
Technology innovation.
The Future of IR and Technology
• IR’s future is moving beyond reporting to
analysis. This means converting data into
‘actionable’ information that FYE
personnel can use.
• Technology’s future is moving beyond
data management to production of tools
that directly facilitate and improve
student success.
Example 1:
Student-at-Risk Prediction Model
• Also known as a predictive model, or enrollment
forecasting model.
• Helps answer questions like:
– Which student variables are most useful for
predicting freshmen retention?
– What is the “best” combination of variables to
optimize predictions?
– How useful is this combination for identifying at-risk
students?
Relevant Previous Research
Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco:
Jossey-Bass.
Bean, J. P. (1985). Interaction effects based on class level in an explanatory model of
college student dropout syndrome. American Educational Research Journal, 22(1), 35–64.
Caison, A. L. (2006). Analysis of institutionally specific retention research: A comparison
between survey and institutional database methods. Research in Higher Education, 48(4),
435-451.
Herzog, S. (2006). Estimating student retention and degree-completion time. Decision
trees and neural networks vis-à-vis regression. New Directions for Institutional Research,
131, 17-33.
Pascarella, E., and Terenzini, P. (2005). How College Affects Student: Volume 2, A Third
Decade of Research. San Francisco: Jossey-Bass.
Sujitparapitaya, S. (2006). Considering student mobility in retention outcomes. New
Directions for Institutional Research, 131, 35-51.
Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research.
Review of Educational Research, 45(1), 89-125.
4 Steps to Modeling Retention
1. Get
Freshmen
Data.
(i.e. We used fall
2009 & 2010
data to build our
“training” data
set.)
2. Build Model.
RETENTION
3. Apply
model
parameter
s to new
data.
(i.e. model
validation,
scoring)
4. Check the actual 2011 retention outcomes to
see how well the model performed.
Examples of Student Variables Analyzed
DEMOGRAPHICS
PRECOLLEGE
ACADEMIC
Gender
Age
Ethnicity
Residency
Geographic Origin
High School GPA & Rank
SAT
AP CLEP
Educational Goals
Transfer GPA
# Transfer Credits
On Campus Employment
Housing
Student Life Activities
Athletics
STAR Usage
Average Class Size
Need Based Aid
Non-need Based Aid
Pell Grant
Work Study
% of Aid Met
PERSISTENCE
Major
Credit Load
Credits Earned
First Term GPA
Distance Education
Dual Enrollment
High Failure Rate Courses
Courses Taken (including
Math & English)
CAMPUS
EXPERIENCE
FINANCIAL
NEED
Credits earned
Credits attempted
Credit Completion Ratio
Math/English Enrollment/Completion
Continuous Enrollment
Milestone metrics
Ethnicity by Geographic Origin
Employment by Housing
High School GPA by First Term GPA
Residency by Need Based Aid
Ratio of Successful Adds to Drops
INTERACTIONS
MILESTONES
7 Strongest Predictors of Retention
Stronges
t
Continental
US
High School
GPA
% Need Met
Educational
Goals
26
(.0 2.8
00 0 4
)*
10
1.3
(.0
00 6 8
)*
20.2
9
(.00 2
0)*
13.486
(.000)*
7.817 (.005)*
AP/CLEP
Credit
08
4 (.0
3
1
.
7
FYE Class
15 Credits
On Campus
Work
Weakest
)*
19
4 .4 6 ) *
3
( .0
1
79 2)*
.
3 5
(.0
These variables account for
approximately 39% of the variance in a
student’s likelihood of returning for a
third semester (Pseudo R Square = .
387).
RETENTION
IN YEAR 1
*Wald statistic (sig.)
The Wald test statistic was used to indicate
strength of the variable instead of the coefficient,
standardized beta. Because of the nature of the
logistic regression, the coefficient is not easily
interpretable to indicate strength.
Predictors in Regression Equation
B
ED GOALS
S.E.
Wald
df
Sig.
Exp(B)
.575
.157
13.486
1
.000
1.778
1.761
.175
101.368
1
.000
5.817
-2.544
.157
262.804
1
.000
.079
.393
.147
7.134
1
.008
1.481
1.021
.227
20.292
1
.000
2.777
ON CAMPUS
WORK
.411
.211
3.791
1
.052
1.508
FIFTEEN
CREDITS
.267
.127
4.419
1
.036
1.306
AP/CLEP
.453
.162
7.817
1
.005
1.573
.000
.001
HS GPA
CONTINENTAL
US
FYE CLASS
Step 1a FIN NEED MET
Constant
-6.623
.628
111.200
1
a. Variable(s) entered on step 1: EDGOALS, HSGPA, MAINLAND, CAS110, FINNEED, EMPLOY,
FIFTEENCREDITS, APCLEP.
Pseudo Rsquare = .387
Scoring Students
• Scoring of relative dropout/retention risk
p = exp(a+b1x1+b2x2+b3x3+b4x4….)
1 + exp(a+b1x1+b2x2+b3x3+b4x4….)
Where:
p = probability of enrollment/non-enrollment
exp = base of natural logarithms (~ 2.72)
a = constant/intercept of the equation
b = coefficient of predictors (parameter estimates)
Example: John is at risk of dropping
• John:
– is from the continental U.S. (0)
– has a below average high school GPA (2.65)
– is enrolled in 9 credits (9)
– has a low % of financial need met (.45)
– isn’t not working on campus (0)
– isn’t enrolled in CAS 110 (0)
– didn’t specify any educational goals in survey (0)
• Probability of Dropping: 0.77
Sample Data for FYE Advisors
LAST
UH ID
NAME
HS WORK
FIRST
CURRENT
AP/
EMAIL
RESIDENT
GP
ON
NAME
CREDITS
CLEP
A CAMP
1st YR
EXP
CLASS
% FIN
ADVISOR
STAR
NEED
PREVIOUS
LOGINS
MET
CONTACT
001
12
HI
6
3.80
Y
Y
77%
0
Y
002
15
HI
0
3.13
N
N
43%
3
N
003
16
CA
6
2.59
Y
Y
65%
2
N
UH ID
AGE
GENDER ETHNICITY COLLEGE
DEPT
MAJOR
Ed Goal
Specified
Relative Risk
Value
Risk Level
001
18
F
CH
CA&H
ART
BA
Yes
14.92
LOW
002
18
F
HW
CSS
SOC
BA
Yes
36.88
MEDIUM
003
19
M
AA
CENG
EE
BS
No
89.18
HIGH
Impact on Campus
• 407 freshmen from 2011 dropped out
in year one.
• Retaining just 22 students from 2011
would have improved Mānoa’s overall
retention rate from 78.8% to 80%.
• Additional Revenue from Tuition and
Fees = $210,000 (for 16 HI, 6 WUE,
excludes out-of-state!).
• Are there 22 students in this group that
we can help/retain?
Example 2: ‘STAR’ Technology
Gary Rodwell
Director of Advanced Technology &
Lead Architect of ‘STAR’
University of Hawaii at Manoa
Essential engagement
Essential engagement
Mahalo
Reed Dasenbrock
Vice Chancellor for Academic Affairs
John Stanley
Institutional Analyst
Gary Rodwell
Director of Advanced Technology
University of Hawaii at Manoa
Questions: [email protected]