EdPolicyWorks
Working Paper:
Never Put Off ‘Till Tomorrow?
Ignacio Martinez1
This paper identifies the causal effect of procrastination on achievement in a MOOC. I use two approaches, instrumental variables (IV) and a randomized control trial. I show that rain and snow affect when a student takes a quiz,
and therefore can be use as an IV. I find that taking the course first quiz on the day it is published, rather than
procrastinating, increases the probability of course completion by 15.4 percentage points. For the randomized
control trial, I send an email (directive nudge) encouraging a randomly selected group of students to procrastinate
less. As an example of the magnitude of the effects, Germans assigned to the treatment group were 167% more
likely to obtain the course certificate. This shows that very low-cost intervention can increase student achievement.
I also find that the effects are heterogeneous across countries, suggesting that it may be advisable to customize
nudges to country characteristics. This online experiment may provide valuable lessons for traditional classrooms.
1
PhD Candidate, University of Virginia, Department of Economics, [email protected]
Updated June 2014
EdPolicyWorks
University of Virginia
PO Box 400879
Charlottesville, VA 22904
EdPolicyWorks working papers are available for comment and discussion only. They have not been peer-reviewed.
Do not cite or quote without author permission. Working paper retrieved from:
http://curry.virginia.edu/uploads/resourceLibrary/28_Martinez_MOOCs_Procrastination.pdf
Acknowledgements: I am grateful to Sarah Turner, Leora Friedberg, Kristin Palmer, James McMath, Jacalyn Huband,
and Katherine Holcomb. I am solely responsible for any errors. The most recent version of this paper is available at
http://goo.gl/QPJpDo
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Working Paper © 2014 Rector and Visitors of the University of Virginia. For more information please visit www.curry.virginia.edu/edpolicyworks or contact [email protected]
Martinez – MOOCs Procrastination
Never put off ’till tomorrow?
Ignacio Martinez
1
Introduction
Thomas Jefferson famously said, “Never put off ’till tomorrow what you can do to-
day.”1 Conventional wisdom establishes that procrastination is bad for student achievement: without deadlines students may procrastinate on their work, and this may reduce
learning.
Research on procrastination is difficult to measure and endogenous to most outcomes of interest. Most papers rely on self-reported procrastination, which may cause
a Hawthorne effect: students may change their behavior and procrastinate less because
they are asked to record their behavior. Also, low ability students may be more likely to
procrastinate, and changing their work habits might not improve their outcomes.
Massive Online Open Courses (MOOCs) provide an ideal laboratory to study procrastination. The rich data that MOOC providers collect include when the course material is published and when the students interact with it. Therefore, researchers can observe procrastination directly without relying on self reported measures. Using data from
Coursera, a MOOC provider, Martinez and Diver (2014) provide descriptive evidence of
the strong negative correlation between procrastination and achievement, as shown in
Figure 1: students who attempt Quiz 1 for the first time later, perform on average worse
than those who do not procrastinate.
In this paper, I use two approaches to estimate the causal effect of procrastination on
achievement. First, because MOOCs collect information on individual IP addresses, I
can use weather data for an instrumental variables (IV) approach. Second, I use directive
nudges for an experimental approach.
1
http://www.monticello.org/site/jefferson/canons-conduct
2
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Weather shocks provide a source of variation that predicts procrastination. I show
that rain and snow affects when a student takes a quiz, and therefore can be use as an IV.
For example, a student is 2.3 percentage points less likely to attempt the first quiz the day
it is published on a day with with rainfall, but 5.0 percentage points more likely on a day
with snowfall.
Next, I show that a directive nudge can affect students choices and help them improve
their achievement. These results are more important than the weather IVs because they
can be replicated in all types of classrooms. Students randomly assigned to the treatment
group received an email in which I provide them with information about the negative
correlation between procrastination and achievement. These students were 17% more
likely (relative to a very low base rate) to successfully complete the course than students in
the control group. Additionally, I show that the effect of the treatment is heterogeneous
among different countries. For example, Germans assigned to the treatment group were
167% more likely to obtain the course certificate, Spaniards 67%, and Indians 40%.
The remainder of this paper is organized as follows. In the next section, I discuss the
relevant literature. In Section 3, I present the economic model. Section 4, describes the
data. In Section 5, I show that weather can be use as an instrument. Section 6, describes
the randomized control trial and its results. In Section 7, I conclude.
2
Literature
Psychologists have been studying procrastination since the seventies. Ellis and Knaus
(1977) claim that, “Procrastination constitutes an emotional hang-up that does you considerable damage.” They also claim that, based on their work as psychotherapists, about
ninety-five percent of college-level individuals procrastinate. They never consider that
they are basing their “guesstimate” from a highly selected sample (i.e., their patients).
Neither did they consider that procrastination could be correlated with some other un3
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
observed characteristic which is the real cause for their patients problems. Knaus (2001)
describe procrastination as our “ancient nemesis”. He claims procrastination may have
originated as early as 2.5 million years ago when our ancestors first grouped into small
clans and someone decided to needlessly put off doing something beneficial for the clan.
These hypotheses are founded with small surveys that rely on indirect measures of procrastination. Moreover, none of these studies addresses the problem of procrastination
being an endogenous choice. Chun Chu and Choi (2005) are the first to consider that “active” procrastination could be good: some people choose to procrastinate because they
know they will do better under pressure. For their study, they invited students to respond
to a questionnaire entitled “Survey of University Students‘ Time Use.” 230 undergraduate students filled out the questionnaire, but the paper does not mention how these
students compared to their peers who chose not to participate in the study. This raises
concerns about selection bias and external validity. In this paper, I use a direct measure
of procrastination and both an instrumental variable and experimental approach to deal
with the endogeneity concerns.
The first paper to address procrastination in the economic literature is Akerlof (1991).
Akerlof argues that although procrastination might initially appear to be outside the appropriate scope of economics, it affects the performance of individuals and institutions in
the economics and social domain. He proposes an economic model in which procrastination occurs when present costs are unduly salient in comparison with future costs, leading
individuals to postpone tasks until tomorrow without foreseeing that when tomorrow
comes, the required action will be delayed yet again. This model challenges the common
assumption in economics that individuals are rational maximisers. Anderson and Block
(1995) argues that the examples Akerlof offers can be explained within the framework of
the standard economic model. They argue that Akerlof confuses later regret with prior
irrationality, which is parallel to confusing ex post with ex ante. O’Donoghue and Rabin
(2001) develop a model where a person chooses from a menu of options and is partially
4
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
aware of her self-control problems. Their model predicts that additional options can induce procrastination, and a person may procrastinate more in pursuing important goals
than unimportant ones. They argue that their second result arises because the greater the
effort a person intends to incur, the more likely she is to procrastinate in executing those
plans. Instead of using the standard economic assumption that preferences are timeconsistent (i.e., a person’s relative preference for well-being at an earlier date over a later
date is the same no matter when she is asked), they model individuals with present-biased
preferences. Finally, Siegfried (2001) argues that combating procrastination is essential in
order to have a successful undergraduate economics honors program. He argues that getting students to work on their thesis early is the key to success. In order to achieve this his
university uses a series of short-term deadlines and the “fear of personal embarrassment”
One approach that I use to control for the endogeneity of procrastination is to use
weather as an instrument. The method of instrumental variables is a signature technique
in the econometrics toolkit, as discussed in Angrist and Krueger (2001). Connolly (2008)
links the American Time Use Survey to rain data from the National Climatic Data Center
(NCDC). She finds that, on rainy days, men shift on average 30 minutes from leisure to
work, suggesting that rain raises the marginal value of work. In this paper, I link data
from the NCDC on rain and snowfall to data from Coursera. Coursera collects student’s
IP addresses. Using this, I geolocate students and assign them a NCDC weather station.
Assuming that procrastinators do not choose their location in response to the weather,
rain and snow are an exogenous source of variation for procrastination and allows me to
identify the causal effect of procrastination on achievement.
MOOCs offer a new learning environment. Martinez and Diver (2014) explore the
opportunities and challenges that MOOCs generate for research. Using data from Coursera, they show a strong negative correlation between procrastination and achievement.
In this paper, I go beyond studying the correlation to determine the causal effect of procrastination on achievement by using both an instrument variables and experimental ap5
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
proach. The experimental approach consists of nudging students by sending information
in a email, as in Martinez (2014). The term “nudge” was first used by Thaler and Sunstein
(2008) to describe “Any aspect of the choice architecture that alters people’s behavior in a
predictable way without forbidding any option or significantly changing their economic
incentives.”
3
Model
In this section, I present a simple model in which a student with ability a chooses
whether to take a Quiz in the first period, procrastinate and take it in the second, or not
take it at all.2 In the second period, if the student’s best grade is greater than g, the student
gets a payoff equal to W . Not attempting the Quiz in a given period yields a grade equal
to zero. On the other hand, attempting the Quiz yields an uncertain grade determined
by a function of his ability and an unobserved random variable ϵt :
gt = f (a, ϵt ) .
Each period the student gets utility from non-Coursera activities (i.e., leisure, and
work), ℓt . The marginal utility of these activities is given by µt . If students like to procrastinate, µ1 > µ2 .
The utility of attempting the Quiz in period 2 for a student with a grade from period 1 equal to g1 and ability a is given by the utility from non-Coursera activities when
attempting the quiz, plus the expected payoff of succeeding in the course:
V2T (g1 , a) = µ2 ℓ2,T + Pr [ max (g1 , g2 ) > g| a] W.
2
In this model, the student is rational and has time consistent preferences. The student will choose to
procrastinate if that is the optimal choice given his information set. An alternative model could generate
procrastination by assuming agents with time inconsistent preferences, as in Choi et al. (2003).
6
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
The utility of not attempting the quiz in period 2 for a student with grade from period
1 equal to g1 and ability a is given by the utility from non-Coursera activities when not
attempting the quiz, plus the payoff of succeeding in the course if the grade in period 1
was greater than the threshold:
V2N T (g1 ) = µ2 ℓ2,N + 1(g1 >g) W.
Therefore, the period 2 problem can be written as:
{
}
V2 (g1 , a) = max V2T , V2N T .
The period 1 problem, in which the student decides whether to attempt the quiz or procrastinate, can be written as:
V1 (a) = max {µ1 ℓ1,N + E [V2 (0, a)] , µ1 ℓ1,T + E [V2 (g1 , a)]} ,
where grades are a function of student ability and an unexpected shock, and the utility
from non-Coursera activities is greater when non attempting the quiz:
gt = f (a, ϵt )
ℓt,N > ℓt,T
This problem can be easily solved recursively. A student with g1 > g will not attempt
the quiz in period 2.3 A student with g1 < g will attempt the quiz if
Pr [g2 > g] W > µ2 (ℓ2,N − ℓ2,T )
3
Adding direct utility from grades would allow this model to explain students attempting the quiz
eventhough they do not need to do it in order to obtain W .
7
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
In the first period, a student will attempt the quiz if
µ1 ℓ1,T + E [V2 (g1 , a)] > µ1 ℓ1,N + E [V2 (0, a)]
E [V2 (g1 , a)] − E [V2 (0, a)] > µ1 (ℓ1,N − µ1 ℓ1,T )
If the expected value of attempting the Quiz in period 1 increases, ↑ E [V2 (g1 , a)], or the
expected value of going into period 2 with a grade of zero decreases, ↓ E [V2 (0, a)], then
students who were at the margin of attempting the Quiz in period 1 will take it, that is,
they will procrastinate less. We can interpret the empirical strategies which I employ in
terms of changes in key parameters in the model.
The intention of the informational experiment described in section 6 is to increase
the expected value of attempting quizzes earlier. Similarly, if the marginal utility of nonCoursera activities decreases, ↓ µ1 , students at the margin will procrastinate less. In section 5, I show that rain and snow affect procrastination.
4
Data
I use data from two sources. The first is Coursera, an education platform that part-
ners with top universities and organizations worldwide to offer free online courses to
anyone. The second source, is the National Climatic Data Center (NCDC) of the National Oceanic and Atmospheric Administration (NOAA).4 Coursera data provides me
with a measure of procrastination, and a platform to run an experiment. I link these data
with NCDC data which provides weather conditions (rain and snow) for each Coursera
participant in the geography identified by IP addresses.
4
This weather data is publicly available at http://goo.gl/x2UrXq.
8
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
4.1
Coursera
I use data from the third edition of Foundations of Business Strategy (FBS) by Michael
J. Lenox of the University of Virginia. 75,180 students were initially enrolled in this
course. The course ran from January 13, 2014 to February 25, 2014. FBS. explores the
underlying theory and frameworks that provide the foundations of a successful business
strategy. The class is divided into weekly modules. Each module consists of an introductory video, a reading from the strategist’s toolkit, a series of video lectures, a quiz, and a
case study to illustrate points in the lectures. Students wishing to receive a Statement of
Accomplishment must satisfy the following criteria:
1. Complete the 6 quizzes. Students can take each quiz three times. The recorded
score is the best score on each quiz. Quizzes have 10 questions with 4 choices each.
2. Submit a final project: a strategic analysis for an organization of their choosing.
3. Assess five peers’ strategic analysis using the peer assignment function.
Final grades are out of 100 points. Out of the 100 points of the final grade, 50 points
are for the final project, 42 points for the quizzes (spread evenly over the 6 quizzes), and 8
points for post and comment up-votes. Those who receive 70 points or more and assess
five peers’ strategic analysis receive a Statement of Accomplishment.
In this paper I use the 24,122 students who expressed, at the time of enrollment, their
intention to complete all the course work necessary to obtain the Statement of Accomplishment.5 These students were randomly assigned to treatment and control groups to
test whether a directive nudge affects their behavior and achievement. Only 1,212 of these
students received a Statement of Accomplishment.
5
At the time of enrollment students had to answer the following question: “How many of the assignments and quizzes do you intend to do?” Only students who selected all were considered for this study.
Other students may have little or no intention of taking the quizzes.
9
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Figure 2 shows when each of the quizzes were published, and the timing of the directive nudge intervention, which occurred on the same day that Quiz 6 was published.
4.2
Weather Data
The NCDC has data from more than 90,000 weather stations around the world.6
The data includes maximum and minimum daily temperatures, rain, and snowfall. Using
stations and students, latitude and longitude, I can match each student to their nearest
weather station.
Connolly (2008) shows that, for men, a rainy day shifts about half an hour from
leisure and home production to work, suggesting that rain raises the marginal value of
work.7 As in her paper, I define a rainy day as a day with at least 0.10 inches of rain.
Additionally, I take snow into consideration, and I define a snowy day as a day with at
least 0.10 inches of snow. Rain and snow might have different effects on student choices.
For example, in a rainy day a student may stay at work until later and be less likely to do
Coursera work when he gets home. On the other hand, on a snowy day a student may
have to stay home instead of going to work, increasing the likelihood of doing Coursera
work.
5
The Effect of Procrastination Using Weather as an Instrument
I estimate linear regressions using rain and snow as IVs for procrastination. The
course started on Monday January 13, and the first quiz was published that day, (Figure 2). The OLS estimate in the first column of Table 1 shows that a student who takes
6
7
In this paper I use their daily data, but it is possible to access hourly records
Result for women are weak. A rainy day is associated with 3 more minutes at work.
10
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
the Quiz on day 1 is 15.4 percentage points more likely to obtain the statement of accomplishment than a student who does not, relative to a very low base of 0.6% who obtain
the statement.
If procrastination is correlated with some other unobserved characteristic (e.g., ability), this estimate would be biased, probably upward. To deal with this endogeneity bias,
I estimate a first stage relationship that shows that a student is 2.3 percentage points less
likely to take the Quiz on the first day if it is raining, and 5 percentage points more likely
if it is snowing. Connolly (2008) suggests that on a rainy day people are more likely to
spend more time at work. If this is so, they would have less time to do their Coursera
activities. On the other hand, on a snowy day people are more likely to stay home and do
their Coursera work.
The second stage regression shows that being induced not to procrastinate in take the
Quiz on day one increases the probability of obtaining the statement of accomplishment
by 13.8 percentage points. Thus, the estimate shrinks because some good students selfselect.8
I can further investigate the impact of procrastination by consider how long it takes
a student to attempt the quiz, rather than simply looking at whether they take it on the
first day. However, I cannot observe this measure of procrastination for students who
never took the Quiz. In order to have another estimate of the effects of procrastination
on achievement I define an upper and lower bound for those students who did not take
the Quiz. In Table 2, I assume that students who did not take the Quiz would have pro8
Note that Table 1 uses a linear probability model. Wooldridge (2012) says “…the linear probability
model is useful and often applied in economics. It usually works well for values of the independent variables that are near the average in the sample”. Angrist and Pischke (2008) give several empirical examples
where the marginal effects of a dummy variable estimated by LPM and probit are indistinguishable. IV
Probit can not be used here because the endogenous regressor is discrete. Vytlacil and Yildiz (2007) propose nonparametric identification and estimation of the average effect of a dummy endogenous regressor
in models where the regressors are weakly but not additively separable from the error term. A dynamic discrete choice model with 41 periods and new choices appearing every week could be a better specification.
But that specification is more costly to solve and does not provide any clear advantage over this simple
specification.
11
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
crastinated until one minute after the deadline. In Table 3, I assume that students would
have procrastinated for a year after the deadline. The IV estimate using the lower bound
implies that an additional hour (week) of procrastination decreases the probability of obtaining the certificate by 0.01 (1.68) percentage points. The upper bound assumption, implies that a week of procrastination decreases the probability of obtaining the certificate
by 0.336 percentage points. The fact that these estimates for procrastination are much
smaller than the estimate from Table 1 shows that the effects of procrastination are nonlinear. That is, procrastinating the day the Quiz is published has much larger effects than
procrastinating an additional day after 10 days of procrastination.
6 Randomized Email Nudge
I divided up the group of 24,122 students into randomly assigned treatment and control groups. On February 17, 2014, the day Quiz 6 was published, I sent an email to the
students in the treatment group, as shown in Figure 3. This email is a low-cost directive
nudge encouraging students not to procrastinate.9
Table 4 shows the effect of the treatment on Quiz 6 outcomes. In order to deal with
attrition, I assign a grade of 0 to students who did not attempt the quiz. “Took Q6”
shows that students in the treatment group were 0.8 percentage points, or 7.74%, more
likely to take the quiz. “maxGrade” shows that students assigned to the treatment group
scored 7 points, or 8.92% better on their best attempt on Quiz 6 than students in the
control group. “Procrastination” suggests that, conditional on taking the quiz at some
point, students assigned to the treatment group procrastinated 2.1 fewer hours on average.
This difference is not statistically significant.
9
I can confirm that 43.02% of these emails were open. To know this, a unique white pixel is inserted in
the body of the email. When the white pixel loaded from the server, I know the email was open. Because
images are blocked by default on some email clients, these numbers are lower bounds for the open rates.
The open rate is crucial to calculating the effect of the treatment on the treated.
12
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
The nudge not only affected students’ behavior in Quiz 6, but also their choices to
go back and take quizzes 1 to 5 for the first time. For example, Table 5 shows that students receiving the email nudge were about a half percentage point more likely to attempt Quizzes 1-5 for the first time after the intervention than students in the control
group. The students who were nudged into taking the quizzes are increasing my measure of procrastination in the treatment group.10
Students assigned to the treatment group were not only more likely to complete Quiz
6, but also to obtain a Statement of Accomplishment. Table 6 shows that students assigned to the treatment group were 0.8 percentage points, or 16.85% more likely to obtain
the course certificate. For an intervention with negligible cost, this is an important effect.
Table 7 shows that the treatment has heterogeneous effects across countries. Italians
assigned to the treatment group procrastinated, on average, 63.22 fewer hours than those
assigned to the control group. Nigerians and Indians assigned to the treatment group
procrastinated more than those in the control group. This is explained by the fact that
those assigned to the treatment group still procrastinated but were more likely to take the
quiz at all.
Finally, Table 8 shows that the effect on outcomes is also heterogeneous across countries. Germans assigned to the treatment group were 167% more likely to obtain the certificate, Spaniards 67%, Indians 40%, and there were not statistically significant effects
for other countries.
10
For example, imagine that there are only 2 students in the treatment and 2 in the control group. One
student in each group took Quiz 1 before the intervention. After the intervention, the student in the treatment group decides to stop procrastinating and take the Quiz, but the student in the control group keeps
on procrastinating. This will increase my measure of procrastination for the treatment group.
13
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
7 Conclusions
This paper examines the role of procrastination on achievement. Understanding
whether or not there is a causal relationship, so that inducing procrastinators to take action improves their learning, is fundamental in order to design a course with incentives
that maximize student achievement.
First, I show that a student is less likely to take a quiz on a rainy day and more likely to
take a quiz on a snowy day. Using weather as an instrument, I show that attempting the
first quiz on the day it is published increases the probability of obtaining the Statement
of Accomplishment by 13.8 percentage points.
Second, I show that a directive nudge can strongly increase achievement. For example, students in the treatment group were 0.8 percentage points, or 16.85% more likely
to obtain the course certificate. For an intervention with negligible cost, this is an important effect. Moreover, the effects are heterogeneous across countries. For example,
Germans assigned to the treatment group were 167% more likely to obtain the course
certificate, Spaniards 67%, and Indians 40%. There were no statistically significant effects in achievement for students from other parts of the world, especially as sample sizes
got smaller. In order to understand the causes of this heterogeneity, more data is needed.
For example, it is possible that the level of education are different, or employment status,
or simply cultural differences. Currently, Coursera survey data is not good enough to
address this question, but Coursera is constantly improving their platform.
Future research should explore other directive nudges to improve achievement. For
example, evidence Martinez (2014) shows that telling students how they are performing
relative to their peers can improve ranking, on average, by 8.43 percentage points. Future
intervention could test what is the impact of telling students how students in the top of
the class are using their time before attempting a quiz.
14
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
References
Akerlof, George A. 1991. “Procrastination and obedience.” American Economic Review
81 (2):1–19.
Anderson, Gary M and Walter Block. 1995. “Procrastination, Obedince, and Public
Policy: The Irrelevance of Salience.” American Journal of Economics and Sociology
54 (2):201–215.
Angrist, Joshua D and Alan B Krueger. 2001. “Instrumental Variables and the Search for
Identification: From Supply and Demand to Natural Experiments.” The Journal of
Economic Perspectives 15 (4):69–85.
Angrist, Joshua D and Jörn-Steffen Pischke. 2008. Mostly harmless econometrics: An
empiricist’s companion. Princeton university press.
Choi, James J, David Laibson, Brigitte C Madrian, and Andrew Metrick. 2003. “Optimal
defaults.” American Economic Review :180–185.
Chun Chu, Angela Hsin and Jin Nam Choi. 2005. “Rethinking procrastination: Positive effects of” active” procrastination behavior on attitudes and performance.” The
Journal of Social Psychology 145 (3):245–264.
Connolly, Marie. 2008. “Here comes the rain again: Weather and the intertemporal substitution of leisure.” Journal of Labor Economics 26 (1):73–100.
Ellis, Albert and William J Knaus. 1977. Overcoming procrastination: Or how to think
and act rationally in spite of life’s inevitable hassles. Institute for Rational Living.
Knaus, William J. 2001. “Procrastination, blame, and change.” Journal of Social Behavior
and Personality 15 (5):153–166.
15
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Martinez, Ignacio. 2014. “Nudges, framing, effort and student performance: Lessons
from a MOOC.”
Martinez, Ignacio and Paul Diver. 2014. “MOOCs as a Massive Research Laboratory:
Opportunities and Challenges.”
O’Donoghue, Ted and Matthew Rabin. 2001. “Choice and procrastination.” The Quarterly Journal of Economics 116 (1):121–160.
Siegfried, John J. 2001. “Principles for a successful undergraduate economics honors program.” The Journal of Economic Education 32 (2):169–177.
Thaler, Richard H and Cass R Sunstein. 2008. Nudge: Improving decisions about
health, wealth, and happiness. Yale University Press.
Vytlacil, Edward and Nese Yildiz. 2007. “Dummy endogenous variables in weakly separable models.” Econometrica 75 (3):757–779.
Wooldridge, Jeffrey. 2012. Introductory econometrics: A modern approach. Cengage
Learning.
16
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Figure 1: Procrastination and Achievement
Note: Extracted from Martinez and Diver (2014).
Figure 2: Time Line
Jan 13 2014
Jan 20 2014
Jan 27 2014
Feb 3 2014
Feb 10 2014
Feb 17 2014
Feb 22 2014
Quiz 1
Quiz 2
Quiz 3
Quiz 4
Quiz 5
Quiz 6 & Intervention
Course ends
17
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Figure 3: Treatment
Subject: [Foundations of Business Strategy] Don’t Leave for Tomorrow What You
Can Do Today
Dear [name],
Our analysis of the previous iteration of this course shows that students who choose
to do the quizzes late perform worse than those who do them earlier.
We encourage you to try the next quiz earlier. Keep in mind that you can retake the
quizzes 3 times.
Best,
University of Virginia MOOC Research Team
18
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Table 1: Regression Results, taking Quiz 1 the day is published and achievement
Dependent variable:
took Quiz 1 the 13
Certificate
took Quiz 1 the 13
Certificate
OLS
First Stage
Second
Stage
(1)
(2)
(3)
0.154∗∗∗
(0.003)
0.138∗∗
(0.063)
−0.023∗∗∗
(0.006)
rain the 13a
0.050∗∗
(0.020)
snow the 13b
−0.0002∗∗∗
(0.00004)
longitude
0.006∗∗∗
(0.002)
Constant
Observations
R2
Adjusted R2
Residual Std. Error
F Statistic
23,463
0.102
0.102
0.206 (df = 23461)
2,678.940∗∗∗ (df = 1; 23461)
0.291∗∗∗
(0.004)
23,463
0.002
0.002
0.451 (df = 23459)
17.446∗∗∗ (df = 3; 23459)
0.011
(0.018)
23,463
0.101
0.101
0.206 (df = 23461)
Notes: ∗ p<0.1; ∗∗ p<0.05; ∗∗∗ p<0.01
dummy variable equal to 1 if it rained more than 0.1 inches the day Quiz 1 was published.
b
dummy variable equal to 1 if it snowed more than 0.1 inches the day Quiz 1 was published.
a
19
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Table 2: Regression Results, Procrastination Q1 and Achivement
Dependent variable:
ProcrastinationLowa
Certificate
ProcrastinationLow
Certificate
OLS
First Stage
Second
Stage
(1)
(2)
(3)
−0.0002∗∗∗
(0.00000)
−0.0001∗∗
(0.0001)
18.949∗∗∗
(5.243)
rain the 13b
−46.341∗∗∗
(17.248)
snow the 13c
0.220∗∗∗
(0.034)
longitude
0.196∗∗∗
(0.003)
Constant
Observations
R2
Adjusted R2
Residual Std. Error
F Statistic
23,463
0.102
0.102
0.206 (df = 23461)
2,676.202∗∗∗ (df = 1; 23461)
816.126∗∗∗
(3.266)
23,463
0.003
0.003
389.703 (df = 23459)
20.741∗∗∗ (df = 3; 23459)
0.160∗∗∗
(0.055)
23,463
0.096
0.096
0.207 (df = 23461)
Notes: ∗ p<0.1; ∗∗ p<0.05; ∗∗∗ p<0.01
how many hours pass between the publishing of the Quiz and their first attempt at it. For students who did not attempt
the Quiz, I assume they procrastinated until a minute after the deadline.
b
dummy variable equal to 1 if it rained more than 0.1 inches the day Quiz 1 was published.
c
dummy variable equal to 1 if it snowed more than 0.1 inches the day Quiz 1 was published.
a
20
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Table 3: Regression Results, Procrastination Q1 and Achivement
Dependent variable:
Certificate
ProcrastinationHigh
Certificate
OLS
First Stage
Second
Stage
(2)
(3)
(1)
ProcrastinationHigh
a
∗∗∗
−0.00002∗∗
(0.00001)
−0.00002
(0.00000)
243.716∗∗∗
(57.891)
rain the 13b
−457.968∗∗
(190.432)
snow the 13c
2.023∗∗∗
(0.374)
longitude
0.167∗∗∗
(0.002)
Constant
Observations
R2
Adjusted R2
Residual Std. Error
F Statistic
23,463
0.118
0.118
0.204 (df = 23461)
3,138.606∗∗∗ (df = 1; 23461)
6, 673.100∗∗∗
(36.055)
23,463
0.002
0.002
4,302.763 (df = 23459)
17.307∗∗∗ (df = 3; 23459)
0.153∗∗∗
(0.044)
23,463
0.116
0.116
0.204 (df = 23461)
Notes: ∗ p<0.1; ∗∗ p<0.05; ∗∗∗ p<0.01
how many hours pass between the publishing of the Quiz and their first attempt at it. For students who did not attempt
the Quiz, I assume they procrastinated for a year.
b
dummy variable equal to 1 if it rained more than 0.1 inches the day Quiz 1 was published.
c
dummy variable equal to 1 if it snowed more than 0.1 inches the day Quiz 1 was published.
a
Table 4: Quiz 6
maxGrade
firstGrade
Attempts
Took Q6
Procrastination
n
Treatment
0.8506
0.5278
0.2461
0.1002
102.9237
12061
Control 90% Confidence Interval
0.7809
(0.0154, 0.124)
0.4966 (-0.005, 0.0673)
0.2241
(0.0059, 0.0381)
0.093
(0.001, 0.0135)
105.0334 (-6.8086, 2.5892)
12061
p-value
0.0347
0.1561
0.0247
0.058
0.4601
21
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Table 5: Attempting Quizzes 1-5
Quiz 1
Quiz 2
Quiz 3
Quiz 4
Quiz 5
n
Treatment
0.0198
0.0273
0.0343
0.0424
0.0585
12061
Control 90% Confidence Interval
0.017
(0, 0.0057)
0.024
(0, 0.0067)
0.0296 (0.001, 0.0085)
0.0383
(-1e-04, 0.0082)
0.0522
(0.0015, 0.0111)
12061
p-value
0.1034
0.1031
0.0369
0.1088
0.0323
Table 6: Achievement
Treatment Control
Certificate 0.0541
0.0463
n 12061
12061
90% Confidence Interval p-value
(0.0032, 0.0124)
0.0056
Table 7: Procrastination by Country
Country
Italy
Nigeria
India
Mexico
Netherlands
Portugal
Canada
United States
Australia
Russia
United Kingdom
China
Spain
Colombia
France
Germany
Ukraine
Brazil
treatment control 90% Confidence Interval p-value n Treatment n Control
62.95
126.17 (-96.4735, -29.9799)
0.00
61
53
115.57
87.28 (7.753, 48.8257)
0.03
79
65
117.28 103.78 (1.7414, 25.258)
0.06
215
194
112.27
150.03 (-70.4457, -5.074)
0.06
63
55
57.34
94.35 (-73.2042, -0.8067)
0.09
58
63
113.85
66.98 (-1.1174, 94.861)
0.11
59
53
97.89
117.06 (-41.7352, 3.3982)
0.16
102
82
99.96 107.54 (-17.3334, 2.1589)
0.20
316
314
97.61
81.27 (-11.7926, 44.4721)
0.33
74
59
89.41
114.47 (-68.7584, 18.6417)
0.34
58
62
97.72
114.48 (-47.8758, 14.3557)
0.37
90
78
82.74
96.13 (-45.1692, 18.3865)
0.48
62
60
95.39
85.30 (-15.8501, 36.0327)
0.52
90
78
88.31 100.23 (-58.5295, 34.6981)
0.66
56
57
104.11
97.24 (-19.9504, 33.6842)
0.67
76
70
100.89
95.85 (-26.5307, 36.6173)
0.79
69
54
82.34
87.62 (-47.6962, 37.1344)
0.83
58
54
113.93
112.98 (-14.9745, 16.8918)
0.92
123
110
22
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia
Martinez – MOOCs Procrastination
Table 8: Course completion
Country
India
Germany
Spain
Portugal
Brazil
France
Nigeria
Netherlands
Canada
Russia
United Kingdom
Italy
Ukraine
Colombia
Mexico
Australia
China
United States
treatment control 90% Confidence Interval p-value n Treatment n Control
0.07
0.05 (0.0041, 0.0338)
0.04
1749
1677
0.08
0.03 (0.01, 0.0954)
0.04
482
506
0.10
0.06 (0.004, 0.0788)
0.07
615
602
0.06
0.03 (-0.0089, 0.0769)
0.19
455
455
0.04
0.03 (-0.0027, 0.0233)
0.19
1334
1336
0.10
0.08 (-0.0242, 0.0754)
0.40
509
506
0.08
0.06 (-0.0211, 0.0601)
0.43
545
521
0.08
0.11 (-0.0889, 0.0362)
0.49
450
437
0.09
0.08 (-0.0206, 0.0443)
0.55
725
697
0.03
0.04 (-0.0378, 0.0177)
0.55
584
559
0.05
0.04 (-0.0149, 0.0291)
0.60
784
827
0.07
0.06 (-0.0366, 0.0667)
0.63
453
440
0.04
0.03 (-0.0231, 0.0414)
0.64
508
485
0.04
0.05 (-0.0574, 0.0322)
0.64
448
446
0.03
0.02 (-0.0204, 0.0336)
0.69
499
538
0.06
0.05 (-0.0275, 0.0418)
0.73
598
528
0.03
0.03 (-0.0182, 0.0261)
0.77
627
623
0.05
0.05 (-0.0088, 0.0096)
0.94
3226
3200
23
EdPolicyWorks Working Paper Series No. 28. June 2014.
Available at http://curry.virginia.edu/edpolicyworks/wp.
Curry School of Education | Frank Batten School of Leadership and Public Policy | University of Virginia