MOOC Completion and Retention in the Context of Student Intent
Justin Reich
Harvard University
Abstract
MOOC critics have raised concerns about low overall completion rates, but these
rates should not be evaluated without accounting for student intentions. Using
survey and log data from nine 2013-2014 HarvardX courses, this article investigates
how completion and attrition rates differ based on students’ self-reported
intentions towards course participation. Across these courses, on average 22
percent of students who intended to complete a course earned a certificate, and
these students are estimated to be 4.5 times more likely to earn a certificate than
students who only intended to browse a course. The median lifetime—the time at
which half of students have stopped participating—of intended-completers was 35
percent of a course, compared with a median lifetime of 10 percent of a course for
intended-browsers. Efforts to personalize MOOCs based on self-reported intentions
should be conducted with care: many students who do not intend to complete a
MOOC do so, and the majority of students who intend to complete a MOOC are not
successful.
Introduction
The signature critique of Massive Open Online Courses (MOOCs) is that they have
low certification rates. From the very first mega-courses offered at Stanford and
MIT, commentators have observed that relatively low percentages of students who
ever register for a course go on to earn a certificate. Across hundreds of MOOCs now
completed, certification rates typically range between 2% and 10%, when these
rates are calculated by dividing the number of certificate earners by the total
number of students who have ever registered for a course.
One response to these concerns has been to argue that this way of calculating MOOC
certification rates is misleading because it does not account for student intention.
Students register for MOOCs for many reasons, and many students have no intention
of completing the courses in which they enroll. For most MOOCs, the only way to
“shop” is to sign up, so many students who register in a course do so just to evaluate
it. Others who register for a MOOC only intend to audit the course or complete a
section of the course, so they register with an intent to participate and learn but not
an intent to complete. As some have argued, if residential college course completion
rates were calculated in the same manner as MOOC completion rates, one would
have to divide the number of people who pass a residential course not by the
number enrolled in the course at the add/drop deadline, but by the number of
people who ever applied to enter the university. A better approach, however, may
1
be to calculate MOOC completion rates as a percentage of students who enrolled in a
course with the intention to complete the course and earn a certificate.
Koller, Ng, Do, and Chen were among the first to make this rebuttal, in their
Educause Review Online article Intention and Retention in Massive Open Online
Courses. In the article, Koller and colleagues investigated some of the underlying
mechanisms of certification rates by examining attrition rates, the rate at which
students stop participating in a course. They examined MOOC attrition by analyzing
the number of hours of lecture video watched by students in Coursera courses. They
argued that the distribution of hours of video viewed in these courses could be
modeled with a two-component mixture model where students are hypothesized to
come from one of two groups, a high-retention or a low-retention group. In this
model, each group is estimated to have a consistent attrition rate: a high attrition
rate for the low-retention group and a low attrition rate for the high-retention
group. These differences in attrition over time then explain, by the end of the course,
the differences in certification rate.
Koller and colleagues then suggest that researchers and course developers might be
able to identify whether MOOC registrants belong in the high-retention or lowretention group by asking them about their intentions. They shared limited early
data that suggested that students who indicated an intent to complete a course did
so at higher rates. In one course, “Writing in the Sciences,” students were asked to
complete a pre-course survey with questions about their intended commitment to
the course. Of those who completed the survey, 63 percent indicated that they
intended to earn a certificate, and of those 24 percent did so. Among all other
students—those who did not complete the survey and those who submitted a
survey but did not intend to complete the course—only 2 percent earned a
certificate. The evidence to warrant this line of reasoning, however, was limited by
having few courses where student intentions were solicited. To better explore these
issues, more systematic survey data were needed.
In the 2013-2014 academic year, HarvardX—an initiative at Harvard University to
create online courses accessible to the public—instituted a common pre-course
survey in their open online courses that probed students in four dimensions:
intention (how much of the course they intended to complete), motivation (why
they enrolled in the course), preparedness (their familiarity with the course content
and with online learning), and demographics (e.g. country of residence and parental
education). These data allow a detailed investigation of how attrition and
completion in HarvardX open online courses differ by a student’s stated intentions.
Two research questions guide this inquiry into completion, attrition, and intention:
First, across multiple courses, what are the completion rates of students who intend
to complete a course compared with other students? Second, what are patterns of
attrition among students who intend to complete a course compared with other
students? Addressing the first question responds to those who have argued that
analyses of MOOC completion rates must be conditioned on student intentions.
2
Addressing the second question illuminates some of the underlying mechanisms
that lead to differential completion rates. Answers to these questions can help
policymakers better understand the opportunities and challenges that open online
courses pose for higher education, and they can help course developers better
understand how learners engage with these courses.
Survey and Log Data from Nine HarvardX Courses
To address these research questions, I draw on data from nine HarvardX courses
with 290,606 registrants and 79,525 survey responses.1 As shown in Table 1, these
courses ranged in size from 11,000 to 92,000 students. As has been reported
previously, many registrants never enter the courseware and a small percentage of
students engage with problems and assignments. Using an unweighted course
average (ignoring the number of registrants in each course), 65 percent of students
take at least one action within a course, 21 percent earn a grade greater than zero,
and six percent earn a certificate.
This study included all HarvardX courses that met four criteria: (1) offered
between September 2013 and June 2014, (2) hosted on the edX platform, (3) issued
certificates to students who earned a sufficient grade, (4) if in a series of modules,
the first module in that series. Criteria (2) excluded Fundamentals of Neuroscience,
which operated on a proprietary platform. Criteria (3) excluded two poetry courses
which did not offer certificates. Criteria (4) excluded four modules of the ChinaX
course, Parts 2 through 5. I only include students who enrolled before the “wrap
date,” the final date in which course materials were due in order to earn a
certificate.
1
3
Table 1: Descriptive statistics for nine 2013-2014 HarvardX courses
Actions Grade
Course
Launch
Wrap
Days Enrollees > 0
>0
14420
2638
Ancient Greek Hero
9/3/13
12/31/13 119
17915
(80%)
(15%)
60924 10280
Science and Cooking 10/8/13
3/15/14
158
92045
(66%)
(11%)
18400
6440
Clinical Trials
10/14/13 2/14/14
123
28102
(65%)
(23%)
20067
8276
China: Part I
10/31/13 12/23/13
53
37238
(54%)
(22%)
19947
9190
Health and Society 11/15/13 2/14/14
91
34775
(57%)
(26%)
22166
5295
Letters of Paul
1/6/14
3/5/14
58
32663
(68%)
(16%)
8335
3227
Global Health
2/25/14
5/27/14
91
11514
(72%)
(28%)
Data Analysis in
13114
4747
4/7/14
6/30/14
84
20315
Genomics
(65%)
(23%)
8893
3054
US Health Policy
4/7/14
6/30/14
84
16069
(55%)
(19%)
All HarvardX
Students
Nine Course Average
290606
96
32293
186266
(64%)
20623
(65%)
53147
(18%)
5905
(21%)
Earned
Cert.
730
(4%)
1795
(2%)
2408
(9%)
2022
(5%)
3321
(10%)
1547
(5%)
1266
(11%)
653
(3%)
761
(5%)
14503
(5%)
1611
(6%)
At the launch of each course, students were directed to a pre-course survey in the
early parts of the courseware. Reminder emails were sent to students who did not
complete the pre-course survey. When examining survey data, it is important to
examine what proportion of a target population responds to a given survey. As
expected, response rates to the pre-course survey were higher among more active
students in the course. On average across courses, 27 percent of all registrants
completed the survey, 42 percent of students with at least one action completed the
survey, and 68 percent of students with a non-zero grade completed the survey.
Findings from these analyses of survey data generalize best to students who
complete at least one action in the course or earn a non-zero grade.
4
Measures
The analysis here includes three kinds of measures: survey measures of selfreported intentions, survey measures of student demographic characteristics, and
measures of persistence and course completion computed from course event logs.
Before turning to these analyses, I briefly describe how I obtain and define these
measures.
Intentions
To ascertain student intentions, the HarvardX pre-course survey asked the following
question:
“People register for HarvardX courses for different reasons. Which of the following best
describes you?”
 Here to browse the materials, but not planning on completing any course
activities. [coded as Browse]
 Planning on completing some course activities, but not planning on earning a
certificate. [coded as Audit]
 Planning on completing enough course activities to earn a certificate. [coded
as Complete]
 Have not decided whether I will complete any course activities. [coded as
Unsure]
Table 2 shows the distribution of these intentions by course, ordered by the
percentage of students intending to complete the course. The percentage of students
stating that they intended to earn a certificate in each course ranges from 40
percent in U.S. Health Policy to 78 percent in Clinical Trials.
5
Table 2: Distribution of students self-reported course engagement intentions in nine
2013-2014 HarvardX courses.
Total
Response
Unsure Browse Audit Complete Responses Rate of Reg.
(%)
(%)
(%)
(%)
(n)
(%)
Data Analysis in
Genomics
US Health Policy
Science and
Cooking
China: Part I
Letters of Paul
Ancient Greek Hero
Global Health
Health and Society
Clinical Trials
All Students
Course Average
13
12
5
5
42
37
40
46
5657
3845
28
24
17
17
14
18
11
11
9
3
5
2
4
2
2
2
30
27
30
21
18
12
10
49
51
53
57
69
75
78
27610
7901
8563
7577
3966
5514
8892
30
21
26
42
34
16
32
15
14
3
3
26
25
56
58
79525
27
28
Demographics
HarvardX collected demographic data from two sources. All students who register
for the edX site are asked questions about their year of birth, gender, and their level
of education (converted here into an 8-point scale ranging from none to doctorate).
The HarvardX pre-course survey asked additional questions about students’ country
of residence, English fluency (on a scale from 1-5), and familiarity with the course
topic (on a scale from 1-5). These additional demographic questions were assigned
to a random half of survey takers. There are fewer responses to these questions, but
responses generalize to both random halves.
Persistence and Completion
Students who earn a course’s minimum passing grade can earn a certificate of
completion. I describe those who do not earn a certificate as “stopping out” of the
course. The neologism is unusual, but if a student never intended to complete an
entire course, the phrase “dropping out” has an inappropriately pejorative
connotation.
To measure student persistence, I use event logs to determine a student’s first date
of activity and last date of activity in a course. For all students who do not earn a
certificate, I define their last date of activity as their stop out date. Students who
6
earn a certificate do not ever stop out of a class according to this definition,
regardless of when they finish activity.
What are the completion rates of students who intend to complete a course
compared with other students?
In the nine 2013-2014 HarvardX courses, certificate rates—defined as the
percentage of all registrants who earned a certificate—ranged from 2 to 11 percent,
aligning with common findings from all MOOCs. Table 3 shows the completion rates
for different subpopulations within each course. Among all students, certification
rates ranged from 2.0 to 11.2 percent, with the average across courses of 5.9
percent. The certification rates of survey respondents are much higher than the
certification rate of all students, ranging from 5.7 percent to 31.2 percent, with an
average of 16.5 percent over the nine courses. Presumably, a student who is willing
to complete a survey is more willing to do everything else to complete a course,
aligning with the commonplace finding that students who engage with any part of a
MOOC are more likely to engage with other parts of a MOOC.
With that caveat, certification rates varied substantially among students with
different self-reported intentions. Those planning to browse had the lowest rates,
ranging from .7 percent to 14.3 percent, with an average across courses of 5.9
percent. These certificate earners are potentially an important population: they
represent students who explicitly stated an intention to just “shop” the course, but
then went on to earn a certificate. Of those who intended to earn a certificate,
between 9.1 and 35.7 percent were successful in doing so. The average across
courses was 22.1 percent, quite close to what Koller and colleagues reported for
“Writing in the Sciences.” Certification rates among those who intended to complete
the course were higher than for students with other intentions and higher than the
rate for all students in the course. That said, the majority of students who intend to
complete a course—in some cases the overwhelming majority of students—were
not successful in doing so.
7
Table 3: Certification rates of all students, survey respondents, and respondents
with different levels of self-reported course engagement intentions from nine 20132014 HarvardX courses.
All
Respondents Unsure Browse Audit
Complete
(%)
(%)
(%)
(%)
(%)
(%)
Data Analysis in
Genomics
3.2
8.9
5.8
3.3
5.1
14.6
US Health Policy
4.7
15.8
9.5
5.5
8.8
24.1
Science & Cooking
2.0
5.7
2.7
0.7
1.5
9.8
China: Part I
5.4
19.4
12.4
3.6
9.0
28.8
Letters of Paul
4.7
13.7
7.5
6.6
5.5
20.3
Ancient Greek Hero
4.1
6.8
4.2
1.7
3.5
9.1
Global Health
11.0
25.8
14.1
8.3
10.8
32.2
Health and Society
9.6
31.2
21.3
14.3
15.5
35.7
Clinical Trials
8.6
21.4
10.6
9.0
8.0
24.8
All Students
Course Average
5.0
5.9
13.3
16.5
6.9
9.8
3.7
5.9
4.9
7.5
19.5
22.1
One potential limitation of comparing these descriptive statistics is that, as Table 2
shows, some courses have higher proportions of students intending to complete
than others. Thus, I use logistic regression modeling to more concisely estimate the
effect of intending to complete on the odds that a student completes a course.
Table 4 shows a taxonomy of logistic regression models predicting certification rate
based on stated intentions, controlling for student demographic characteristics and
the fixed effects of course. In these models, the reported value in each cell is an odds
ratio. Odds ratios higher than one indicate a higher probability of certification
associated with a given predictor, assuming all other predictors can be held
constant. Odds ratios lower than one indicate a lower probability of certification
associated with a given predictor, assuming all other predictors can be held
constant. In all three models, the “browse” category of student intention is the
reference category. Later, I report adjusted risk ratios, the ratio of the mean
predicted probabilities of certification, for the final model.
As a baseline, Model A includes only the student intention predictors. There is a
strong positive association between the self-reported intent to complete a course
and course completion. In Model B, I add demographic characteristics as control
predictors, and in Model C I add the fixed effects of courses. The strong positive
association between intent to complete and completion persists in these subsequent
models.
8
Computing adjusted risk ratios from the odds ratios in Model C, I estimate that an
intended-completer is 4.5 times more likely to earn a certificate (p<.001)than an
intended-browser, holding constant demographic characteristics. An intendedcompleter is 3.5 times more likely to earn a certificate (p<.001) than an intendedauditor.
A student’s stated intention is a stronger predictor of course completion than any of
the demographic predictors. Older students, students with more education, and
students with greater initial familiarity with the course material all had higher odds
ratios of course completion, and female students and U.S. residents had lower odds
ratios of completion than others. While these estimates are statistically significant,
they are substantively modest.
Table 5: Taxonomy of logistic regression models predicting course completion in
nine 2013-2014 HarvardX courses.
Model A
Model B
Model C
Intercept
.038***
.021***
.007***
(.004)
(.003)
(.001)
Unsure
Audit
Complete
1.94***
(.213)
1.35**
(.147)
6.36***
(.666)
Age
Level of Education
Female
English Fluency
Subject Familiarity
US resident
Course Fixed
Effects
Not Included
2.05***
(.291)
1.29
(.182)
6.54***
(.882)
2.11***
(.304)
1.33*
(.189)
5.96***
(.814)
1.02***
(.001)
1.15***
(.014)
.886***
(.025)
.884***
(.017)
1.06***
(.014)
.738***
(.024)
1.02***
(.001)
1.12***
(.015)
.861***
(.026)
.993
(.022)
1.10***
(.016)
.782***
(.027)
Not Included
Included
n
79525
41187
41187
Pseudo r2
.060
.077
.13
Note: Cell contents are odds ratios and (standard errors). * p<.05 ** p<.01 *** p<.001
9
Do students who intend to complete a course persist longer than students who
do not intend to complete?
Attrition rates can illuminate the underlying mechanisms of completion rates and
provide a more nuanced picture of student persistence. Indeed, nearly one-quarter
of all survey respondents claim that they intend to only complete part of a course, so
efforts to increase the persistence of these students, who still may never earn a
certificate, are important to measure. Koller, Ng, Do, and Chen modeled persistence
by examining the fraction of a course’s video that a student watches. While this is
one useful perspective, students in HarvardX courses learn not only through videos,
but through texts, problems, and other course elements. Therefore, I model
persistence as a function of the percentage of total course days in which students
are active during the course.
In Figure 1, I display Kaplan-Meier survivor functions for five groups of students
from the nine HarvardX courses: those who stated an intention to (1)browse,
(2)audit or (3)complete the course, those who stated an (4)unsure intention, and
those who (5)did not respond to the survey. On the x-axis I plot the percentage of
the course time elapsed from 0 percent to 100 percent. On the y-axis I plot the
observed proportion of the cohort remaining. The plotted curves are called survivor
functions, and higher survivor functions indicate higher levels of persistence on
average in the group.
10
Figure 1: Kaplan-Meier survivor functions for students who did not respond to the
pre-course survey (n=211,081), intended-browsers(n=2,636), intendedauditors(n=20,709), intended-completers(n=44,354), and students with unsure
intentions (n=11,826) in nine 2013-2014 HarvardX courses.
These survivor curves show that intended-completers persist longer than other
registrants, and several features of Figure 1 are of particular interest. First, all five
cohorts of students experience sharp drops in the first 1 to 2 percent of a course. At
all time points, those intending to complete are most likely to remain, followed by
intended auditors, the unsure, and intended browsers. Among all four groups of
survey respondents, the shape of these curves is quite similar. While the magnitude
of the initial drop is different, the subsequent rate of decline is fairly similar. One
common way of summarizing and comparing survivor functions is by examining the
median lifetime of different groups, in this case the time at which half of a cohort has
stopped out. For non-respondents to the survey, the median lifetime occurs before 1
percent of the course is complete. For intended browsers, the median lifetime is 10
percent of the course, and for intended completers it is 35 percent of the course.
As a complement to these survivor functions, Figure 2 shows hazard functions for
the four groups of students by stated intentions, and these functions provide
another view of attrition. Hazard probability, the proportion of each cohort stopping
out in any given period of time, is on the y-axis. Course time elapsed, in bins for
11
every 2% of course time, is on the x-axis. I constrain these observations to the first
96 percent of the course, since hazard rapidly approaches 1 after that. Notice the
large difference in hazard rate in the beginning of courses, ranging from .18 among
intended browsers to .09 among intended completers. (Though not shown, for
survey non-respondents, hazard rate in the first two percentile units would be over
.5.) Soon however, the hazard rates of all four groups stabilize at roughly the same
level through the first half of the course. Intended-completers have a lower hazard
rate at any given time in the course than other groups. In the second half of the
course, hazard rates for all groups begin to climb again, though more slowly for
intended-completers.
Figure 2: Hazard rates by stated intention for nine 2013-2014 HarvardX Courses.
Recall that Koller and colleagues, after examining video-watching behavior,
hypothesized that their two latent groups of students—the high-retention and lowretention groups—had different, constant attrition (or hazard) rates. If this model
extended to this examination (where attrition is a function of active course time),
we would expect the hazard functions of intended-completers and intendedbrowsers to differ substantially in rate across all times: students who intend to
complete the course should have a constant low hazard rate and students who do
not intend to complete the course should have a constant high hazard rate.
12
Instead, it appears that no stated-intention group has a constant hazard rate over
time; rather hazard rates vary considerably across the early, middle, and end parts
of the course. All students are likely to stop out in the first few days of a course, and
all students are less likely to stop out in the middle of the course. Between the 15th
and 50th percentile unit of time in a course, hazard rates are below 5 percent in any
2 percent unit of course time. Instead, it is hazard ratio—the ratio of hazard rates
among stated intention groups—that is roughly constant over time. Though the
baseline hazard rate varies over time, at any given time intended-browsers are
more likely to stop-out than intended-completers. In time periods where hazard
increases, it increases more sharply among intended-browsers than among
intended-completers. Though perhaps a subtle point, this distinction between ratio
and rate has important substantive implications, discussed below. The behavior of
students with different self-reported completion intentions is, in some respects,
more alike than different.
Discussion: Those Who Intend to Complete a Course are More Likely to Do So.
Now What?
Key Findings and Recommendations for Course Developers
Students who intend to complete a MOOC are more likely to do so than other
students. Of HarvardX students who responded to the pre-course surveys, 56
percent reported that they intended to complete their course. An average of 22
percent of these intend-completers across courses went on to actually earn a
certificate. I estimate that a student who intends to earn a certificate is 4.5 times
more likely to do so than a student who intends to browse a course and 3.5 times
more likely to do so than a student who intends to audit a course, controlling for
demographics and the fixed effects of course. Students who enroll in a course with
the intent to finish do so at much higher rates than students who enroll in a course
without intending to complete it. Still, the majority of those who intended to earn a
certificate did not do so.
My intuition is that these findings are probably not of a great enough magnitude to
change most people’s prior beliefs about MOOCs. For those inclined to be concerned
with MOOC attrition rates, these findings may do little to ameliorate concerns that
MOOCs are not suitable learning environments for most students. For those inclined
to be excited about the opportunities that MOOCs afford, these findings indicate that
completion rates among people committed to completing a course are several times
higher than the rates that have been widely reported to date. As policymakers and
university leaders debate the costs, benefits, and future of MOOCs, these findings
provide a useful reference point, even if they are unlikely to settle key policy
questions.
For course developers, there are several useful insights to be drawn from these
findings. First, certification rates conditioned on intention can be used by course
teams as one indicator among many—ranging from course satisfaction to
13
performance on learning assessments to persistence through course material to
engagement on forums and in social media—that help characterize the success of a
course. The finding that 22 percent of students who intend to complete a course go
on to do so may prove to be a useful benchmark for course teams.
[[CALLOUT BOX]]
Key Takeaways for Course Developers:
A New Benchmark for Completion Rates: Across nine HarvardX courses, 22
percent of students who intended to complete a course went on to earn a certificate.
Other MOOC course teams may find this a useful benchmark, in the context of all the
particular details of every unique course, to help characterize the success of a
course.
Learner Intentions Can Change: On average across courses, 6 percent of intendedbrowsers, 7.5 percent of intended-auditors and 10 percent of students with unsure
commitments to the course earned a certificate. Some of the people who did not
initially intend to complete a course were convinced to do so. These “intention flips”
may be a greater indicator of course success that the students who inevitably attrite.
Attrition Happens Early: Course Beginnings are Important: Regardless of a
student’s stated intentions, attrition rates are highest in the early part of the course.
Course developers should recognize that for many students, the first unit of the
course is the only part someone will see. Course teams should consider allocating
resources to making that beginning unit inviting and compelling.
[[CALLOUT BOX]]
One group of students that proves interesting are those who did not intend to
complete a course but did so. On average across courses, 6 percent of intendedbrowsers, 7.5 percent of intended-auditors and 10 percent of students with unsure
commitments to the course earned a certificate. For courses whose purpose is to
bring new people into a field, these statistics might be even more important that the
percentage of intended-completers who complete a course. It has been obvious from
high dropout rates and anecdotal evidence that some people who intend to
complete a course dropout, and these statistics suggest that the opposite appears to
be also true: people go on to complete courses that they never intended to finish.
These findings also compel caution among those considering personalizing learning
experiences according to students’ self-reported intentions. Many students who did
not intend to earn a certificate in these nine courses did so, and the majority of
students who intended to earn a certificate did not. As more options for conditional
logic and personal learning pathways become available in course platforms, these
14
insights suggest that course developers should think carefully about giving students
with modest intentions a lesser experience.
Course developers should pay particular attention to the characteristics of how
attrition occurs over time in courses. As has been observed in other studies, attrition
is very high early in courses, and then soon levels out at a relatively low level. One
contribution of this paper is to observe that these patterns hold across all stated
intention groups. Nearly 10% of intended completers leave in the first days of a
course. Students who intend to complete a course do not have low constant attrition
rates and students who do not intend to complete do not have high constant
attrition rates. Rather attrition rates vary considerably over time within courses, but
the ratio among these hazard rates for students from different stated-intention
groups remains fairly constant. These findings suggest that instructors concerned
about attrition should consider focusing their efforts on building community and
engagement in the first days of a course when attrition is highest. These efforts have
the potential to benefit students at all levels of stated intentions. Again, however,
these findings raise cautions about developing adaptive or personalized approaches
based on students’ stated intentions. Retention patterns among students with
different levels of stated intention look quite similar.
One other important caution to emerge from this study, useful to anyone following
MOOC research, is that students who express any intention at all are more likely to
complete a MOOC than those who do not complete a pre-course survey. Across a
sample of nine HarvardX courses from 2013-2014, the completion rate among all
registrants averaged 6 percent compared with an average of 16.5 percent for survey
respondents. This difference highlights an important caveat in examining
completion rates across courses: students who complete an optional pre-course
survey can be very different from those who do not. Comparisons between students
who state an intention to complete a course and all other students can be
misleading; a more fair comparison should be between those who state an intention
to complete a course and students who state a different intention.
What Exactly is the Problem with Low Completion Rates? Opportunities for Future
Research
This research provides more precise estimates of MOOC certification and attrition
rates conditioned on student intention, but one fundamental question remains:
What would be a good certification rate for a MOOC? If 22% of students who intend
to complete do so, is that still a disaster? A resounding success?
In the public discourse in blogs and news articles about MOOC completion rates, low
completion rates of MOOCs have often been presented as self-evidently problematic.
Few commentators or researchers have delineated exactly why low MOOC
completion rates are of concern. But these exact reasons are important, because
course developers and policy makers could respond to different problems with
different approaches.
15
Low passing rates might be an indicator that the quality of MOOCs is low, since
many students choose not to persist through the entire course. They might indicate
that MOOCs are too difficult and offer insufficient support for students’ learning
needs. MOOC certification rates might be low because registering for a MOOC is
often a precondition to evaluating its worthiness, and students might typically shop
multiple courses before settling on a smaller number to commit to. They might be
low because many people intend only to complete part of a course. They also might
be low because well-intentioned people get overwhelmed by other commitments,
and stop out of a course even though they enjoyed it and were progressing well.
This research based on pre-course surveys provides a better, though still imperfect,
sense of the intentions that students bring to a course. These findings can resolve
some questions concerning how best to understand certification rates. For instance,
if only slightly more than half of survey respondents intend to complete a course, it
seems unlikely that open-enrollment MOOCs could ever expect to see certification
rates higher than 50% for survey respondents, no matter how wonderful the quality
of the course. And if substantial numbers of people that do not intend to finish a
course go on to do so, then that might be a useful and novel indicator of how well
MOOCs engage their audiences.
To better understand the meaning of MOOC attrition rates, researchers need to
learn much more about why people stop out of MOOCs. If students leave because
they are overwhelmed with other commitments or because they have learned
everything they needed to learn, then that has very different implications than if
they leave because they are disappointed in the course or are finding it too difficult.
This research has provided better answers to the question, “why did people come to
these MOOCs?” The next challenge is to get better answers to the question, “why did
they leave?”
In the 2013-2014, HarvardX also attempted to survey students at the end of courses,
and the effort was mostly unsuccessful. Response rates to these end-of-course
surveys were very low. Most of the students who complete end-of-course surveys
were very satisfied with HarvardX courses, but these were the students who have
already persisted through months of voluntary challenges. The typical approach to
course evaluation—asking all students to evaluate a course when it ends—is unlike
to work in the MOOC context. A great innovation to MOOC research would be to
develop mechanisms to predict student stop-out behavior very soon after a
student’s final action, and then survey students about their reasons for leaving at
that moment. A deeper understanding of the causes of student attrition would
clarify what could be done to address issues of MOOC attrition. If researchers can
discern how many students leave MOOCs because of life’s other commitments, that
might help estimate a reasonable ceiling on retention rates in voluntary, free, open
online courses. Uncovering how many students leave because they are dissatisfied
might help better estimate the level of attrition in MOOCs that course developers
could realistically address through better instructional design.
16