Predictive Analytics:
Goals, Means, & Managing
Expectations
Tony Scinta
Nevada State College
BIG Goals
Practical insights into
the development of an
analytics platform
Of your faculty
About your
students
Key questions to ask:
About your
institution
About your
philosophy
Nevada middle tier of
higher education
- circa 1999
Nevada middle tier of
higher education
- circa 2016
Inadequate Academic
Preparation
Institutional
Challenges
100%
Commuter
Firstgeneration/No
n-cognitive
challenges
Work
obligations/Poo
r finances
Watch entire season
of Breaking Bad in
one weekend
Institutional Policy . . .
Enforcement of prerequisites . . .
Gateways
to
Completion
Course scheduling . . .
Best practices in
teaching & learning . . .
Assessment techniques. . .
Academic support
services . . .
Fail Often . . .
Fail Early . . .
1st semester
GPA below 2.0
Fail Often . . .
Fail Early . . .
4-Year Rate
100 Percent
Fail Often . . .
Fail Early . . .
5-Year Rate
Graduate
Don't Graduate
100 Percent
Graduate
Don't Graduate
Fail Early . . .
6-Year Rate
Graduate
Don't Graduate
100 Percent
Silver Lining
Year-to-Year Retention
Fail Often . . .
16%
Higher
Used Advising
Year-to-Year Retention
HS GPA < 3.0
25%
Higher
19%
Higher
Used Advising
GOAL:
Help students before it is
too late
MEANS:
Early identification of
at-risk students
Assistance from academic
support services
Institutional
Challenges
19%
Higher
Used Tutoring
BIG QUESTIONS
First Question:
What should I ask
before proceeding
with an analytics
effort?
What is your goal?
Second Question:
Third Question:
What predicts that goal?
How can we know if students
are on track to reach the goal?
72
Class Add Date
Credits
Passed
Transfer
GPA
Pass
Ratio
Academic
Year
Pell Eligible
Semester
Credits Taken
Gender
Ethnicity
HS GPA
Expected Family
Contribution
1st Term
Major
Cumulative
GPA
Remedial
Math
1st
Generation
Attended
Orientation
Instruction
Mode
Academic
Level
Enrollment
status
Distance
Predictive Model
Predicts the probability that a student will earn a grade
of C or better in the course
GREAT, RIGHT?
Usability Pitfalls
1. Student feedback was inadequate
o SOLUTION: Added new dashboards
Usability Pitfalls
1. Student feedback was inadequate
o SOLUTION: Added new dashboards
2. Faculty wanted more data and they wanted it to be
more accessible
SOLUTIONS:
o More data on student cards (e.g., time since last log in)
o Emails to faculty
o Ability to view grades as raw points or percentages
Modeling Pitfalls
Modeling Pitfalls
1. Bad predictions
1. BAD
PREDICTIONS!
Modeling Pitfalls
1. Bad predictions
o Anomalous predictions
o Too lenient
Low Risk
76-100%
Type II Error?
Modeling Pitfalls
1. Bad predictions
Number of Students
3500
3000
o Anomalous predictions
o Too lenient
2500
2. Faculty vs. Gradebook
3207
2000
1500
1000
752
356
500
0
Green
Yellow
Low Risk
51-75%
Red
Low Risk
0-50%
Modeling Pitfalls
Modeling Pitfalls
SOLUTIONS
SOLUTIONS
Low Risk
90-100
Low Risk
75-89.9%
Low Risk
0-74.9%
Characteristics/History
Grades
Philosophical Pitfalls
“Non-cognitive”
Concerns
No solution yet
Structural Pitfalls
Structural
Pitfalls
Small Class
Size
Structural Pitfalls
InsufficientAdvisors
New Comprehensive
Dashboard
Structural Pitfalls
SOLUTIONS
Take Home Lessons
1. Analytics platforms are not easy to do right
o Clarify roles BEFOREHAND
o Modeling is not magic
• Manage expectations BEFOREHAND
• Choose the right parameters
2. One size does not fit all
o Our experience – faculty need it less in small courses
o Advising may be critical
3. Even done right, there are concerns
o Belonging/efficacy should be accounted for
o Advising may be critical
4. If it works, it is worth the effort