Annual First Year Experience Conference February 7-­‐10, 2015 Dallas, TX Examining Research from a First-Year Student
Math Early Warning Pilot
Dr. Greg Budzban, Chair and Professor Math Department Amber Manning-­‐Ouelle?e, Director of Enrollment Management, College of Business Southern Illinois University Carbondale Session Agenda •  Ins$tu$onal Profile •  SIU Gateway Math Course Structure •  Math Data and Predic$ve Value of Week 8 Vector Analysis and Markov models •  Predic$ve Analysis •  Pilot data and Outreach Efforts •  Lesson Learned •  Q & A InsQtuQonal Profile •  4-­‐year Public Research University and Open Access •  Undergraduate 13,461 •  Graduate 4,485 •  48% First-­‐genera$on •  Over 85% on some type of financial aid •  Average 22.2 ACT •  Minority Enrollment 28% •  Female 46% Male 54% •  103 Bachelors degrees, 78 Masters, 34 Doctoral programs Gateway Math Courses Trends •  22% of students require mathematic s
remediation
•  Remedial courses not always effective
•  Research indicates that the more required
developmental courses students take, the less
likely they are to do so.
SIU Math Courses •  Math 101 (Non-STEM majors, satisfies Core
Curriculum requirement)
•  Math 107 (Includes STEM/Business majors, no
credit towards degree, “remedial” course)
•  Math 108 (Includes STEM/Business majors,
credit towards degree, satisfies UCC)
Early Warning IntervenQon PlaUorm Week 3
25%* (Preparation) + 25%* (Motivation) + 50%* (Demonstration) • 
• 
• 
• 
RED:
ORANGE:
YELLOW:
GREEN: 0% to 55%
56% to 65%
65% to 75%
76% to 100%
•  Students also receive an intervention score in week 8 and week 12 that
is simply their course grade at that time.
Intermediate Algebra Data Fall 2013 Week'3'to'Week'8''Math'107'FALL'2013
Warning'level
Week'3'
Totals
Green'at'
week'8
Yellow'at'
week'8
Green'at'wk'3
Yellow'at'wk'3
Orange'at'wk'3
Red'at'wk'3
Not'on'Wk'3'list
'Week'8'totals
%ABC
176
67
36
52
4
335
55.52%
122
17
5
1
2
147
90.47%
38
27
11
2
0
78
57.69%
Not'
Orange'at' Red'at'week'
enrolled'
week'8
8
week'8
6
8
2
16
6
1
10
9
1
4
40
5
1
1
0
37
64
9
10.80%
6.25%
0%
#ABC
%ABC
137
36
9
3
1
186
77.84%
53.73%
25.00%
5.77%
25.00%
55.52%
College Algebra Data Fall 2013 Week'3'to'Week'8''Math'108'FALL'2013
Warning'level
Week'3'
Totals
Green'at'wk'3
Yellow'at'wk'3
Orange'at'wk'3
Red'at'wk'3
Not'on'Wk'3'list
'Week'8'totals
%ABC
303
112
79
144
9
647
60.90%
Not'
Green'at' Yellow'at' Orange'at' Red'at'
enrolled'
week'8
week'8
week'8 week'8
week'8
245
32
16
10
0
36
43
21
12
0
11
24
22
22
0
4
12
34
91
3
2
3
1
3
0
298
114
94
138
3
94.90%
60.53%
30.85%
9.35%
0%
#ABC
%ABC
263
73
29
25
4
394
86.80%
65.18%
36.71%
17.36%
44.44%
60.90%
Intermediate Algebra Data Spring 2014 Warning'level
Green'at'wk'3
Yellow'at'wk'3
Orange'at'wk'3
Red'at'wk'3
'Week'8'totals
#ABC
%ABC
Week'3'
Totals
55
33
29
49
166
68
40.96%
Week'3'to'Week'8''Math'107'SPRING'2014
Green'at' Yellow'at' Orange'at' Red'at'week' Withdrew'
week'8
week'8
week'8
8
by'week'12
26
16
6
7
0
5
17
8
3
3
0
4
17
8
2
0
2
5
42
13
31
39
36
60
18
29
21
14
4
93.55%
53.85%
38.89%
6.67%
0%
#ABC
%ABC
37
18
9
4
68
67.27%
54.55%
31.03%
8.16%
40.96%
College Algebra Data Spring 2014 Warning'level
Green'at'wk'3
Yellow'at'wk'3
Orange'at'wk'3
Red'at'wk'3
Not'on'Wk'3'list
'Week'8'totals
#ABC
%ABC
Week'3'to'Week'8''Math'108'Spring'2014
Week'3' Green'at' Yellow'at' Orange'at' Red'at' Withdrew'
Totals week'8 week'8
week'8
week'8 by'week'12
177
116
41
15
5
3
79
15
31
21
11
8
55
1
14
19
20
10
87
5
6
8
59
33
3
2
0
0
0
1
401
139
92
63
95
54
210
134
53
16
7
0%
52.37% 96.40% 57.61%
25.40%
7.31%
0%
#ABC
%ABC
141
38
16
13
2
210
79.66%
48.10%
29.09%
14.94%
66.67%
52.37%
PredicQve Value of Week 8 Grades •  Intermediate Algebra : Success rate of Week 8 metric (C or be?er) Fall 2013 – 
– 
– 
– 
Red
Orange Yellow
Green
4/64 (6.25%) 4/37 (10.8%) 45/78 (57.7%) 133/147 (95.3%) •  College Algebra : Success rate of Week 8 metric (C or be?er) – 
– 
– 
– 
Red
Orange
Yellow
Green
13/138 (9.3%) 29/94 (30.8%) 70/114 (60.5%) 283/298 (94.9%) Markov Models of Student Performance 0.81 0.11 0.38 0.32 0.05 0.19 0.03 0.08 0.14 0.03 0.28 0.28 0.11 0.02 0.24 0.63 Markov Models of SIU College Algebra Fall 2013 – Week 8 to Final grade 0.95 0.61 0.39 0.31 0.09 0.05 0.69 0.91 Feature Vector Data Analysis •  The structure of the performance data permits a fine grain analysis to
optimize student support resources. •  Example: Analyze the transition behavior of two “yellow” students
in College Algebra in Spring 2014.
•  Student 1 :
•  .25*(68) + .25*(68) + .5*(80) = 17 + 17 + 40 = 74
•  Final Grade:F
MOTIVATION!
•  Student 2 : •  .25*(55) + .25*(98) + .5*(72) = 13.75 + 24.5 + 36 = 74. 25
•  Final Grade: B
•  Prediction? Which student succeeded?
Markov Models of SIU College Algebra Fall 2013 – Week 8 to Final grade 0.95 0.61 0.39 0.31 0.09 0.05 0.69 0.91 EW Pilot Fall 2014-­‐COS/COB •  The EW data suggests that “geeng green” by week 8 is the pathway to success. •  Colleges of Science/Business pilot will target “yellow alert” students in Fall 14 –  Goal: 50% “Yellow-­‐to-­‐Green” by week 8. Outreach Efforts Academic Affairs •  Upper level Administra$on •  Support and understanding of goal •  Faculty involvement • IntenQonal le?ers to students on effort • Reframe to posiQve • Invest in student Experience in classroom Outreach Efforts Student Affairs •  College-­‐level Reten$on Staff •  Devised protocol for Qmeline of intervenQons •  Set tracking methods to collaborate across departments •  Shared common data with key consQtuents at the university •  Followed up with feedback survey at the beginning of the spring semester •  First-­‐year advisors •  Contacted the students via phone, email •  Tracked responses in EAS •  Housing staff •  RA involvement with study sessions Using the transiQon matrix to make predicQons ⎡.81 .11
⎢.32 .38
[298 132 72 158]∗ ⎢
⎢.14 .3
⎢
⎣.03 .08
Actual Week 3 distribuQon (Fall 2014) [ Gr Y Or R] .05 .03⎤
.19 .11⎥
⎥ = [299 118 99 144]
.28 .28⎥
⎥
.24 .63⎦
Week 3 to 8 transiQon matrix From Fall 2013 Predicted Week 8 distribuQon [ Gr Y Or R] Markov Models of SIU College Algebra Fall 2014 Pilot 0.38→0.425 0.32→0.425 0.19→0.09 0.11→0.06 Increased the students in the top two categories from 70% to 85% . An increase of 21.5% in one semester! Markov Models of SIU College Algebra Fall 2014 – Week 8 to Final grade 0.94 0.69 0.29 0.08 0.06 0.31 0.71 0.92 College Algebra Data Fall 2014 Week 3 to Week 8 Math 108 FALL 2014
Week 3 Green at Yellow at Orange at Red at Withdrew #ABC
Warning level
Totals week 8 week 8 week 8 week 8
Green at wk 3
297
256
22
9
10
0
271
Yellow at wk 3
132
57
49
16
10
0
95
Orange at wk 3
68
17
23
15
13
0
33
Red at wk 3
132
7
21
24
76
4
26
Not on Wk 3 list
0
0
0
0
0
0
0
Week 8 totals
629
337
115
64
109
4
425
%ABC
67.57% 94.10% 69.57% 29.70% 8.25%
0%
%ABC
91.2%
72.0%
48.5%
19.7%
0.0%
67.6%
Using the transiQon matrix to make predicQons ⎡.81 .11 .05
⎢.32 .38 .19
[298 132 72 158]∗ ⎢
⎢.14 .3 .28
⎢
⎣.03 .08 .24
Actual Week 3 distribuQon (Fall 2014) [ Gr Y Or R] .03⎤
.11⎥
⎥ = [299 118 99 144]
.28⎥
⎥
.63⎦
Week 3 to 8 transiQon matrix From Fall 2013 Actual Week 8 distribuQon (Fall 2014) Predicted Week 8 distribuQon [ Gr Y Or R] [ 340 121 70 129] RecommendaQons •  Campus-­‐wide direc$on and communica$on •  Success because of so many partnerships (Advising Council, math department, EAS, retenQon staff, Deans, Chairs, and student affairs staff •  Iden$fy the math course “needs” of your campus •  What are DWF rates? •  Demographic of students? •  Does “remedial” math work? •  Major-­‐specific math courses •  Prerequisite courses •  Tenured vs. adjunct faculty RecommendaQons •  Seek data as support for curriculum changes •  NaQonal trends •  Campus advising •  InsQtuQonal data on course failures, drop, and repeats •  Department collabora$on •  Assess what departments are already doing early warning •  Set protocol for outreach and Qmeline •  Frequent and consistent meeQngs RecommendaQons •  Select pla_orm that supports your students and faculty pedagogy •  One system that a course coordinator oversees •  User-­‐friendly and pulls the data into manageable informaQon •  Assess your outreach efforts •  Target what worked •  Feedback from students •  Reframe to posiQve – moQvaQon is already there •  Track tutoring and instructor office hours •  Early intervenQon means students find the correct major! Questions?
Dr. Greg Budzban, [email protected]
Amber Manning-Ouellette, [email protected]