Does students’ active involvement increase academic achievement?
Beat Affolter, Marius Gerber, Sven Grund, and Alexander F. Wagner*
September 2015
Abstract
This paper investigates the impact of student involvement on academic achievement. Our
analysis takes into account that a potential endogeneity problem exists in this relationship. A
large university course where significant student discussions (and coaching by tutors) occur
online presents a unique opportunity to address this problem. We first observe that students
living geographically further away from campus participate more in online discussions. However,
distance of a student from the university is unlikely to be related to academic achievement in the
course other than through involvement in online activities; this hypothesis can be substantiated by
observing that distance from the university is not correlated with academic performance in
another course that does not entail such online activities. Using this identification strategy (and a
second, independent source of exogenous variation in involvement), we find that active
involvement significantly positively contributes to academic achievement.
*
Affolter: University of Zurich, Department of Banking and Finance, Plattenstrasse 14, CH-8032 Zurich,
Switzerland, [email protected].
Gerber: Kienbaum Consultants International, Leutschenbachstrasse 95, CH - 8050 Zurich, Switzerland,
[email protected].
Grund: doc.- Grund - dynamic organizational consulting, Renggerstrasse 3, CH-8038 Zurich, Switzerland,
[email protected].
Wagner (corresponding author): University of Zurich and CEPR. Mail: University of Zurich, Department of
Banking and Finance, Plattenstrasse 14, CH-8032 Zurich, Switzerland, [email protected].
We are grateful to Florian Eugster, Martin Halla, Christoph Wenk and several conference and seminar participants
for helpful comments. We thank the University of Zurich and the Department of Banking and Finance for providing
data used in this study.
1
Introduction
Two and a half thousand years ago, Confucius is reputed to have said: “Tell me and I will forget;
show me and I may remember; involve me and I will understand.” In this paper, we provide
evidence for (the third part of) this Confucian Conjecture. Specifically, we document a causal
impact of a student’s active involvement on academic achievement.
There are several reasons for economists to be interested in the impact of involvement on
learning. First, given the arguably high individual costs of active involvement, knowing whether
the basic tenet of microeconomics that effort is positively related to output also holds for
involvement and academic achievement is naturally of interest.
Second, most academic
economists (and academics in other fields) are also involved in teaching, and as such many
faculty members wonder “what works” in course design.
In particular, they are regularly
confronted with the desire of students of being able to participate in a class, but it can be costly
for a university and the faculty to design courses (especially those with a significant size) that
offer this opportunity. Whether or not resources of educational institutions and society more
generally should be devoted to developing course settings that facilitate active involvement in
part depends on whether individual academic achievement is actually fostered by involvement.
Third, on the macro level, human capital is vital for the success of modern economies. While
economic models of the role of human capital development for economic growth abound,
economics as a discipline is relatively silent on the impact of specific features of the learning
process on the outcome. Educational psychology does offer some insights into this matter. It
suggests that knowledge is gained within a communication process in which active involvement
1
of the individual is critical for learning success; 1 pure class attendance, acquiring factual
knowledge or spending time studying are insufficient for deep understanding.
theoretically and intuitively, the Confucian Conjecture is appealing.
Therefore,
However, modern
econometric methods have rarely been employed to test the hypothesis that involvement is
important for academic achievement.
To make progress on these issues, we use unique data from a large university course which
offers students the opportunity (but does not require them) to engage in online activities. The
course uses a learning concept in which group exercises (“involving activities” which consist of
creating a range of deliverables, such as programming a tool to estimate “betas” of real
companies and building a valuation spreadsheet for real companies) and online discussions (the
“forum” which is facilitated by online tutors) play a central role. In particular, we are able to
monitor activities of students on an online discussion board where students share opinions, raise
questions, and post answers relating to their group tasks. The quantity of posts by students
provides our measure of involvement.
Our primary interest is in whether greater involvement in the online group forum drives
student learning and performance in the final exam. We expect online involvement to have a
positive effect on student achievement because students who involve themselves in the forum
benefit from two major advantages: first, their communication is documented in writing, arguably
requiring additional thoughts when asking or answering a question and facilitating recall later on;
second, tutors only support online interactions of groups.
The key empirical challenge when relating involvement to academic achievement is that
involvement is endogenous.
We identify the effect of active involvement on academic
1
See, for example, the classic contributions of Dewey (1916), Vygotsky (1978), and Bruner (1996) for this
“constructivist” view of learning. 2
achievement by first noting that online involvement is powerfully predicted by the geographic
distance of a student’s residence to the university; students living further away are more likely to
involve themselves online. At the same time, pre-college education in the country where the
university is located (Switzerland) is arguably unrelated to geographic location (contrary, for
example, to the United States). Therefore, it appears plausible that there is no direct effect of the
place of residence on academic achievement.
Consistent with this exclusion restriction
conjecture, we document the distance to university is uncorrelated with the academic
achievement in a previous course that does not offer the online involvement option. Observing
the correlation of our instrumental variable with this “parallel” academic achievement measure
(where the causal mechanism we posit – the effect of online involvement on academic
achievement – cannot play a role) offers an important advantage of this analysis over the typical
instrumental variables setting where evidence in favor of the exclusion restriction is necessarily
more indirect. Geographic distance is also uncorrelated with a number of observables that could
be related to academic achievement.
Additionally, we use a second, independent instrument for involvement, the student’s
attitude towards online communication boards.
We document that this instrument, too, is
uncorrelated with achievement in a parallel course and is as such unlikely to be correlated with
omitted variables that drive academic achievement in the course we investigate.
Our main finding is a positive causal impact of student involvement (driven by geographic
distance or the student’s attitude towards online communication boards) on academic
achievement. Importantly, the two instrumental variables are essentially uncorrelated with each
other, but the estimated size of the effect of involvement on achievement is very similar with
these two different sources of identification. The result survives many robustness checks.
3
Our findings complement the literature that has established the role of several important
factors for academic achievement of students.2 A small number of papers has directly considered
student effort. De Fraja et al. (2010) measure student effort by students’ attitudes towards school
and the opinion of their teachers about student laziness, and they document that effort of students,
teachers and parents boosts children’s achievement in school. Arguably, opinions of teachers are
subjective; we use an objective measure of involvement. Stinebrickner and Stinebrickner (2008)
establish a causal impact of the amount of study time invested by the student on academic
achievement. Involvement is related to but distinct from the amount of time invested; students
can invest time without being involved in the subject. Finally, some papers (mostly in the
psychological and education literature) deal with correlations of activity and learning
performance, 3 but these papers have not considered the endogeneity of student activity – the
central task of this paper.
The paper is organized as follows. Section 2 discusses the empirical strategy. Section 3
introduces the data. Section 4 presents the results. Section 5 concludes.
2
For example, a significant body of work illuminates the effect of class size on performance (e.g., Angrist and Lavy
(1999), Fredriksson and Öckert (2008), Leuven et al. (2008), Heinesen (2010), Hoxby (2000), Bandiera et al. (2010),
De Giorgi et al. (2012), and Fredriksson et al. (2013)). Other work considers the influence, on student achievement,
of the quality of professors (Carrell and West 2010), teachers’ performance incentives (Lavy 2002, Barlevy and Neal
2012), teacher’s attendance incentives (Duflo et al. 2012), and financial incentives for students (Leuven et al. (2010),
Fryer (2011), Levitt et al. (2011)). Neal (2011) provides a survey of the use of performance pay in education. Yet
other studies have considered the role of student attendance (Romer (1993), Durden and Ellis (1995), Arulampalam
et al. (2007)). Teaching practices also have an effect on social capital (Algan et al. 2011).
3
Shea et al. (2001) and Swan et al. (2000) investigate the link between perceived level of interaction and perceived
student learning achievement, while Jiang and Ting (2000) and Rovai and Barnum (2003) consider the link between
actual online activity and perceived learning achievement. Perceptions of achievement can be deceiving, of course.
Gerber et al. (2008) document a positive correlation of online activity in terms of postings and academic
achievement measured as the result of the final exam; by contrast, Picciano (2002) finds no general evidence that
online activity correlates positively with academic achievement. None of these studies addresses the possible causal
relationship between involvement and achievement.
4
2
Empirical strategy
Consider a university course where students exhibit different degrees of involvement. We wish
to test the hypothesis that higher involvement leads to greater academic achievement. Assume
for the moment that suitable measurements have been found for both involvement and
achievement. Still, an important concern arises in a regression of achievement on involvement.
One cannot assume that involvement levels are randomly assigned to students. For example,
particularly capable students may be more active, leading to an upward bias in the estimate of the
effect of activity on learning outcomes. Alternatively, these students might bother less with
participating because they are doing fine on their own, leading to a downward bias. Either way, a
naïve regression of achievement on involvement will yield biased results. 4 In the empirical
analysis, we include a proxy for previous academic achievement, but it is possible that
unobserved heterogeneity remains that may bias the results.
Specifically, we need to compare the academic achievement of students with a given
involvement level with the counterfactual outcome that would have been observed had this
involvement not taken place. In observational data, this causal effect can be identified using an
instrumental variable, i.e., a variable that significantly affects active involvement of students, but
is unrelated to academic achievement.
To motivate the choice of our instrumental variables and the identification strategy, we next
describe the institutional context in which we conduct our analysis.
4
Further below we comment on the potential complication that arises if students substitute between different modes
of involvement. Fortunately, in our setting, the involvement mode we study, online involvement, is the by far most
important active involvement mode.
5
2.1 Institutional context
We study student involvement in a large Intermediate Corporate Finance course (“the course”
from now on) taught at the University of Zurich, Switzerland in the second year of the bachelor
program in banking and finance. Essentially all courses in this second year are mandatory, and
so is this course. The course builds on an introductory 5 week Basic Corporate Finance course
taught in the first year (“the first-year course”). The course runs from September to December,
with a final exam in January. The course employs a blended learning concept, based on four
building blocks: self-study, coaching, lectures, and involving activities.
First, students acquire the basic concepts on their own following an online “study trail,” a
textbook, multimedia material, interactive worksheet exercises, multiple choice questions, and so
on.
Second, online coaches supervise discussion boards in order to support students in
understanding the concepts. Third, the mostly theoretical content of self-study is accompanied
by lectures where the professor links theory with practice or uses examples to bring theory into
relevant context. Lectures are recorded and students can watch them online or on their mobile
devices.5 Physical attendance of the lectures is possible but neither a course requirement, nor is it
monitored.
Fourth, the central feature we focus on in our study are group exercises, the involving
activities (IA). Before the course starts, each student chooses to analyze one out of twenty real
public corporations. Thereafter, she or he is randomly assigned to teams of three to five students
within these company groups. For each company there are between two and four teams. Each
team solves three case studies using real data of their respective firm. Each group exercise takes
5
A special software makes it possible to see what the professor writes on the screen just like students attending class
physically.
6
between one and two weeks, for a total of five weeks. Our focus is on the activity of students in
these five weeks. 6 The activities require more than just study effort; they require actually
producing something new, namely, a write-up of a case study and building a spreadsheet that
allows them to conduct a company valuation.7
For the group exercises, student groups use several virtual interaction tools (discussion
boards, chat, file storage) integrated in the online course platform. Each group discussion board is
supervised by a tutor (“coach”) who supports the students and answer questions within 24 hours;
there are as many coaches as companies. These tutors have passed the course with a very high
grade in the past year(s). A “coaching handbook” guides their activities and suggests some
milestones for a minimal set of tutor actions (including motivational words here and there). It
also asks tutors not to simply hand out answers to group questions but to induce students to think
about the issues at hand and to try to engage them in the course topic. Tutors can themselves
contact a “headcoach” and ultimately the faculty member responsible for the course. Tutors
monitor and can guide the discussions on the board.
All these discussion threads remain
available for team members for the duration of the course. Students use the online discussion
board to organize themselves, pose questions, discuss their findings, and form a group opinion.
Thus, involvement is seen in the extent of activity on the group discussion board. We
expect online involvement to have a positive effect on student achievement because students who
involve themselves in the online discussion board benefit from two major advantages compared
6
The first involving activity takes place in September and requires students to write a memo on the company they
study. The second involving activity takes place in October and requires students to calculate the cost of equity
capital for their company. The third involving activity takes place in November and asks students to conduct a
valuation of their company.
7
While our experience with the course setting strongly makes us believe in the validity of this conceptualization of
involvement, we recognize that our analysis inevitably presents a joint hypothesis test, namely, that online
involvement as we measure it is an actual measure of involvement, and that involvement contributes to academic
achievement. 7
to students working and meeting offline. First, their communication is documented in writing.
This arguably requires more careful thoughts when asking or answering a question than verbal
communication does. Also, it facilitates future recall of lessons previously learned. Second,
tutors only support online interactions, implying that only online involvement yields immediate
reinforcement and critical feedback. These features enhance learning opportunities for students.
2.2 Instrumental variables and empirical model
We argue that (1) the distance of the student’s domicile from university and (2) the attitude a
student has towards the usefulness of online discussion boards provide two valid instrumental
variables for a student’s online involvement. We discuss the reasoning underlying each of these
two independent instrumental variables separately.
The motivation for the first, geographic instrument derives from the following simple
microeconomic argument. In principle, students can involve themselves both through online
activities – posts on the discussion board – and through personal, physical group interactions.
The decision in which form to engage depends on the relative marginal benefits and marginal
costs of the two activities. Both online and offline involvement offer some marginal academic
achievement benefit, but these marginal benefits are unrelated to the distance of the student to the
university. The marginal costs of online involvement also do not vary with distance. By contrast,
for students living further away from the university, the opportunity cost to physically come to
meetings outside of class can be significant. Indeed, given that the lectures of the course we are
studying are recorded and are available for convenient viewing on computers and mobile devices,
many students do not come to the lectures at all. It follows that geographic distance from the
university should be positively correlated to activity on the online discussion board. The first
8
stage of our regressions tests this hypothesis. When employing this instrument, we quantify the
impact of involvement on achievement driven by geographic distance.8
The identifying assumption is that geographic distance has no direct impact on achievement
in class other than through online activity. This assumption is by definition not testable, but we
provide several pieces of evidence in favor of this conjecture in the empirical analysis below.
Specifically, we show in Section 3.4 that not only is distance uncorrelated with several
observables (such as demographics), but it is also uncorrelated with a “parallel” academic
achievement measure, namely, academic achievement in an unrelated course that does not offer
similar online involvement opportunities. The latter finding strongly corroborates the assumption
that geographic distance is uncorrelated with potentially unobserved determinants of academic
achievement. Moreover, we argue that in Switzerland concerns regarding geographical
heterogeneity in pre-college education are unlikely to play an important role.
Our second instrument draws on the attitudes of students towards online learning. In their
first year of studies, students have access to a general online discussion board so that they have
some idea of the potential benefits of this medium for them. It is natural to expect that students
who regard the online discussion board as ex ante more useful will indeed use it more actively.
At the same time, the final exam is a written, closed-book exam. Thus, for the preparation for
and the performance in this exam attitudes towards computer use and online learning can play no
role.
This suggests the validity of the identifying assumption that preferences for online
8
To the extent that we find a positive impact of involvement driven by distance on achievement, a natural question
to ask is: Why do not all students involve themselves more? One answer is that the students living close by are likely
to regard online participation as unnecessary. This paper only concerns online involvement, and the paper does not
exclude the possibility that other forms of involvement are equally or more effective. We note that it is likely that
those who engage more in online activities due to distance are less likely to engage in other involvement; in
particular, they are less likely to discuss with other students in person. We cannot monitor these other meetings. If
such personal interactions are also ultimately beneficial to success in the course, but occur less for those with more
online involvement, we are underestimating the impact of online involvement on course performance.
9
discussions are not correlated with academic achievement in the course other than through online
involvement. We provide evidence in favor of this assumption in Section 3.4.
We use a standard 2SLS setup, with standard errors clustered on the student group level.
3
Data
3.1 Dependent variable
Our main outcome variable, ACHIEVEMENT, is the percentage of points earned in the final
exam in the course out of the number of possible points. In our setting, test scores provide a
good measure of students’ performance and learning for similar reasons as they do in Bandiera,
Larcinese and Rasul (2010). First, test scores are not curved.9 Second, half of the exam consists
of multiple-choice questions where there is no discretion in grading. Points for the other half are
given independently by three assessors; the professor responsible for the course is not directly
involved at this stage. Third, the faculty teaching the course do not have any incentive to boost
the number of students taking the course; the course is mandatory for all students in the banking
and finance bachelor program.10
3.2 Explanatory variables
Our key explanatory variable is INVOLVEMENT, which we define as the number of discussion
board entries a student makes for the involving activities. The quantity of posts is a simple and
intuitive measure of the degree of online involvement.
Our focus, thus, is on individual
9
Final grades, building on test scores, are also not curved, but there is some degree of discretion of faculty where to
set cutoffs for top and bottom grades.
10
Moreover, even if students cared about points, not grades, raising student evaluations would not be a concern
because the exam takes place after evaluations have been filled out (but before the professor learns the evaluation
outcome).
10
activities. 11 We note that posts have different contents – some are more knowledge-related;
others are of a more interpersonal nature; yet others refer to organizational and logistic topics.
Our analysis cannot distinguish these types. While this presents a limitation, we note that it is
quite possible that all three forms of posts can contribute to the overall achievement. Imprecise
measurement of true involvement will imply measurement error in the explanatory variable and,
thus, attenuation bias.
Our instrumental variables are unlikely to be correlated with this
measurement error, making it attractive to use the IV approach also for this errors-in-variables
problem.
Our first instrumental variable measures the opportunity cost of time for a student to come
to physical meetings at university. Using data on the place of residence of students and the
online time table of the Swiss public transportation system (essentially nobody drives to school
due to very limited parking), we calculate RAW DISTANCE, which is the number of minutes it
takes to get to the main location of the university. In our main regressions, we use a binary
indicator variable, DISTANCE, which is equal to 1 if travel time is above the median and 0
otherwise.12
The second instrumental variable is ATTITUDE. In an online entry survey, before students
get exposed to the actual course, they are asked (besides other questions): “As how useful do you
regard the following study methods? … [Online discussion boards].” Higher scores (on the scale
of 1 to 5) indicate more positive attitudes towards the usefulness of the discussion board.
11 Individual activity is a necessary (but not sufficient) condition for interactions with others to take place. Studying
actual interactions is outside the scope of this paper. 12
By far the biggest jump in the extent of online involvement occurs around the median of distance, making our
main specification both economically and statistically appealing. Similar results hold with either a continuous
distance variable that enters linearly or with multiple quantiles. With more quantiles, one obtains a finer fit in the
first stage, but at the price of a lower first-stage F-statistic.
11
As for control variables, more able students are expected to do better on the exam. We proxy
for previous knowledge of corporate finance and possibly for individual cognitive abilities
through PREVSCORE, the percentage of reachable points a student obtained on the final exam of
the first-year course. We also control for age and the number of semesters a student has been
studying.
Because it is possible that some tutors are more effective than others, all regressions include
student group fixed effects.
We also collect additional information used in robustness checks. This includes, for example,
COMPUTER USE, which is the number of years a student has been using computers. Moreover,
we obtain data on the number of discussion board postings each student reads (READING). The
INVOLVEMENT-READING RATIO is the ratio of INVOLVEMENT and READING.
3.3 Sample and descriptive statistics
Descriptive statistics of our sample are presented in Table 1. The main analysis is based on a
sample of between 158 and 172 students.13 There is broad heterogeneity in terms of academic
achievement.14 The average number of postings per student in the five weeks during which the
group exercises take place is 17, with a standard deviation of 16, indicating substantial
heterogeneity in behavior of students. In total, we recorded 6082 postings, including responses by
13
We have in principle data on 218 students. We impose two sample restrictions in the main analysis, but the results
do not depend on these restrictions. First, we exclude 6 students above the age of 29 to get a consistent sample of
typical students. Second, we note that while even students failing the first-year course are allowed to take the
intermediate course (which we focus on), they do need to take the first-year course again concurrently. If they fail
the first-year course again, they are excluded from any further university studies in the economic field in Switzerland.
Thus, their incentives in the intermediate course we focus on are typically quite different from those of students who
have passed the first year course. The sample size in the regressions is finally determined by the fact that we do not
have domicile location and demographic data for all students. We verify that the sample of those students where we
do not have all data used in our analysis is no different in observable variables than the sample used in the analysis. 14
The final grades range from 6 (excellent) to 1 (very poor). The passing grade is 4 (sufficient). Grades are assigned
in steps of quarters. A grade which is one quarter better than another grade requires approximately 4 additional
exam points. Thus, grade 5 needs 16 more exam points than grade 4.
12
coaches. Thus, there is an overall fairly high degree of involvement online. Excluding students
with very high participation does not affect the results. Median (average) travel time to the
university is 28 (37) minutes, with significant variation.
TABLES 1 and 2 ABOUT HERE
Correlations of all variables are in Table 2.
Interestingly, ACHIEVEMENT and
INVOLVEMENT do not show a high correlation; as the regression analysis reveals, this is likely
due to endogeneity of INVOLVEMENT.
3.4 Correlates of instrumental variables
In this section, we document several attractive properties of our instruments; see the correlations
in Table 2 and the descriptive statistics for the sample split according to distance, shown on the
right side of Table 1. We recognize that few, if any, instrumental variables are perfect, but the
institutional setting considered here offers an opportunity to directly address several conjectures
regarding potential correlations of the instruments with unobserved variables that are also
correlated with academic achievement.
First, DISTANCE and demographics are uncorrelated. Older students (who might have
worked before beginning their studies), students who take the class in a later term than is
standard, and male students do not live closer by or further away than younger students, students
who take the class in the standard (third) term of studies, and female students, respectively.
Similarly, age, term of studies and gender are uncorrelated with ATTITUDE.
Second, we note that while geographic location is unlikely to be correlated with a general,
innate degree of engagement or energy that could explain the academic achievement of students,
13
there are some more specific aspects of this instrumental variable that one needs to be mindful of.
Consider these three possibilities: (i) It is in principle conceivable that students from suburbian
areas have lower or higher cognitive abilities than students living in the city center (where the
university is located). (ii) A related concern might be that individuals living further away are less
likely to attend university in the first place; Card (1995) provides evidence of this for the U.S. (iii)
Another potential concern is that students living further away live with their parents, whereas
those living in the center of Zurich might need to earn money by also working part-time jobs,
leaving less time for studies. Similarly, it is possible that those living further away generally have
more time at their disposal (or that those living in the center are more prone to enjoy a coffee
with friends).
To address these concerns, we first note that heterogeneity of neighborhoods in terms of
wealth is not an important characteristic of the Swiss social landscape. Moreover, pre-college
education is of essentially uniform quality in Switzerland (contrary, for example, to the U.S).
Adding to this conceptual argument, we note that if cognitive abilities were indeed
heterogeneous across the communities surrounding the university, or if selection into university
attendance were indeed taking place based on distance from the university, or if students living at
home spend more time studying, we would necessarily observe a correlation of distance also with
the academic achievement in other courses.
Fortunately, our setting provides us with a “parallel” academic achievement variable,
namely, performance in a previous course, which does not offer online involvement. Thus, we
can directly assess the mentioned concerns.
As can be observed in Tables 1 and 2, geographic distance is, in fact, unrelated to academic
achievement in the first-year course, PREVSCORE. Also, t-tests show that the mean values of
14
PREVSCORE do not differ in the two DISTANCE subsamples.
(Moreover, Kolmogorov-
Smirnov tests do not reject the hypothesis that PREVSCORE has the same distributions in these
two subsamples.)
Overall, while geographical distance from the university probably does not induce perfectly
random assignment of online activity levels to students, this evidence supports the intuition that
distance is unlikely to be correlated with innate ability, available resources for studying, and
other unobserved variables that are correlated with academic achievement.
Finally, experience with using the computer and the internet might be correlated with
ability and other skills that might affect academic achievement. However, attitudes towards
online learning and distance are not correlated with experience with computers, eliminating this
concern.
Lastly, ATTITUDE and DISTANCE are uncorrelated, thus providing us with two
independent sources of identification.
4
Results
4.1 Determinants of involvement
Table 3 presents the first-stage regressions. Students living further away from the university are
indeed more likely to involve themselves online than those living closer by, see Column (1). The
quantitative effects are sizable: Students living further away than the median make two thirds of a
standard deviation more posts. Similarly, the results in Column (2) imply that students with more
positive attitudes towards the usefulness of the discussion board as a learning tool in fact use the
posting opportunity more during the term.
15
TABLE 3 ABOUT HERE
In Column (3) we include both DISTANCE and ATTITUDE. Importantly, both variables
enter highly significantly, and with a similar quantitative size as independently. This confirms
the impression obtained from the descriptive statistics: these two candidate instrumental variables
capture different aspects of the motivation for involvement.15
Overall, these results establish the relevance of the instruments and alleviate weak
instrument concerns.
4.2 The effect of involvement on achievement
Our main results are presented in Table 4. The baseline OLS regression in Column (1) shows
that INVOLVEMENT is positively and significantly associated with achievement.
INSERT TABLE 4 HERE
Our main inference is based on the 2SLS regressions presented in the remainder of the table.
Using DISTANCE as the instrument, the first-stage F-statistic is almost 10, indicating that the
instrument is strong.16
15
We also note (in untabulated results) that the distance of other group members is not significantly associated with a
student’s online involvement. The average and the total involvement of other group members is also positively
associated with own involvement (but the exclusion restriction does not hold for these group-level variables which is
why we do not use them as instruments).
16
Stock, Wright, and Yogo (2002) suggest a threshold of 8.96 in the case of one instrument. This threshold applies
for the case of iid errors. As we cluster standard errors by groups, we report the Kleibergen-Paap rk statistic. Here,
the Staiger and Stock (1997) threshold, 10, is in practice considered as a good rule of thumb.
16
Consistent with the Confucian Conjecture, involvement has a strong, positive effect on
achievement. A one standard deviation increase in involvement (driven by geographic distance)
implies an achievement score that is higher by 9 (=15.57*0.58) percentage points, or about eight
tenths of a standard deviation. Note that the coefficient in the instrumental variables regression is
substantially larger than in the basic regressions, indicating that endogeneity had introduced a
downward bias in the relationship between active involvement and achievement in those earlier
regressions. (A Hausman test rejects the use of OLS instead of IV.)
Column (3) uses ATTITUDE as the instrument. Again, the first stage is strong, and the
second-stage estimate is again highly significant. It is noteworthy that with this rather different
instrumental variable we obtain a virtually identically-sized quantitative impact of involvement
on achievement when involvement is driven by attitudes as we did when involvement is driven
by geography. (Recall that ATTITUDE and DISTANCE are essentially uncorrelated.) This is
reassuring.
Finally, Column (4) uses both instruments. In this over-identified regression, the first stage
is somewhat weaker, as is expected when using more instruments. However, once again the size
of the coefficient on INVOLVEMENT is consistent with the two other 2SLS regressions. 17
LIML is recommended as a robustness check for the case of an overidentified model because in
such a model LIML and two-stage least squares can have quite different finite-sample
distributions (Angrist and Pischke 2008, p. 213). Interestingly, under LIML, the coefficient on
INVOLVEMENT is identical to the one obtained in 2SLS up to three decimals.
Overall, the results establish active involvement as an economically and statistically
powerful driver of academic achievement.
17
The Hansen-Sargan J statistic yields a p-value of above 0.9, indicating that the overidentifying restrictions are not
rejected.
17
4.3 Additional Results and Robustness
In terms of control variables, we find that younger students perform significantly better during
the course. This may be because they are motivated or also because younger students have left
high school more recently and are, therefore, still better trained in studying for exams than those
who have worked for a few years before resuming their studies. Students who have done well in
the past (because they are cognitively more able and/or because they have acquired more subject
matter knowledge) also do better in the present course.
Table 5 reports some additional results.
First, Column (1) includes some additional
individual-level control variables. Gender and experience with the use of computers do not
explain achievement, and the other results remain very similar as before.
Second, Columns (2) and (3) consider the role of group-level activity. In Column (2), we
find that uneven degrees of involvement are not helpful for group members. Column (3) seems to
show that, ceteris paribus, the more information there is to process for an individual, that is, the
more involved the other group members are, the lower the ultimate achievement. However, the
coefficient on individual involvement in Column (3) is much higher than in the other regressions.
Naturally, if a student is more involved, this induces responses by others. This balances out the
group-level effect. Note also, however, that the first stage is weak in this particular regression.18
Third, Columns (4) and (5) consider the role of active involvement relative to “consumption”
activity on the discussion boards, namely, reading of posts of others. If active involvement is no
more conducive to academic achievement than passive observation on the discussion board (or
“lurking”) we would expect no relationship between the INVOLVEMENT RATIO and
18 The size of the coefficients nonetheless makes sense. Intuitively, if a student post on average engenders two
responses, the total effect of an individual post is +1.25-2*0.3 = 0.65, which is approximately the size of the
coefficient in regressions not controlling for activity of the other group members. 18
achievement. Our results reject this hypothesis. Column (4) shows that the INVOLVEMENT
RATIO enters significantly in an OLS regression. Because reading itself is also endogenous, we
proceed by instrumenting the INVOLVEMENT RATIO. For this ratio, ATTITUDE provides the
more powerful instrument.19 In Column (5), we again find a positive and significant impact of
this “net” measure of involvement on achievement.
5
Conclusion
One of the central questions for any economy and any organization is how its members learn
effectively. This paper provides evidence that involvement fosters learning and leads to better
learning outcomes.
We have studied a specific setting, and naturally a transfer of the insights from this online
learning setting to more traditional settings is not guaranteed. Broadly speaking, our results
imply that where there is a way for a student to involve herself or himself in a documented
fashion, then involvement in this way – whether driven by opportunity cost of time
considerations or preferences for this mode of learning – will lead to higher performance.
Overall, in terms of policy implications the findings of the paper vindicate efforts to offer
students opportunities for involvement.
19
Students who regard the discussion board as more useful may be more active posters as well as more active
readers, but the data show that the effect on involvement dominates. Similarly, while distance from university has
relatively small explanatory power for the variation in reading patterns of students, it drives active involvement.
19
References
Algan, Yann, Pierre Cahuc and Andrei Shleifer. Teaching Practices and Social Capital, CEPR
Working Paper No. 8625, 2011.
Angrist, Joshua D., and Victor Lavy. Using Maimonides' Rule to Estimate the Effect of Class
Size on Scholastic Achievement. The Quarterly Journal of Economics, 1999, 114(2), 533575.
Angrist, Joshua D., and Jörn-Steffen Pischke. Mostly Harmless Econometrics: An Empiricist‘s
Companion. Princeton University Press, 2008, Princeton.
Arulampalam, Wiji, Robin A. Naylor and Jeremy Smith. Am I missing something? The Effects
of Absence from Class on Student Performance. Working paper, University of Warwick,
2007.
Bandiera, Oriana, Valentino Larcinese and Imran Rasul. Heterogeneous Class Size Effects: New
Evidence from a Panel of University Students. The Economic Journal, 2010,
120(549), 1365–1398.
Barlevy, Gadi and Derek Neal. Pay for Percentile. American Economic Review, 2012, 102(5),
1805–31.
Bruner, Jerome S. The culture of education. Harvard University Press, 1996, Cambridge, MA.
Card, David. Using geographic variation in college proximity to estimate the return to schooling.
In: Louis N. Christofides, E. Kenneth Grant and Robert Swidinsky, eds., Aspects of labour
market behaviour: Essays in honour of John Vanderkamp, University of Toronto Press, 1995,
Toronto, 201-222.
Carrell, Scott E., and James E. West. Does Professor Quality Matter? Evidence from Random
Assignment of Students to Professors. Journal of Political Economy, 2010, 118(3), 409-432.
De Fraja, Gianni, Tania Oliveira, and Luisa Zanchi. Must Try Harder: Evaluating the Role of
Effort in Educational Attainment. The Review of Economics and Statistics, 2010, 92(3),
577–597.
De Giorgi, Giacomo, Michele Pellizzari, and William Gui Woolston. Class Size and Class
Heterogeneity. Journal of the European Economic Association, 2012, forthcoming.
Dewey, John. Democracy and Education: An Introduction to the Philosophy of Education.
MacMillan, 1916, New York.
Duflo, Esther, Hanna Rema, and Stephen P. Ryan. Incentives Work: Getting Teachers to Come to
School. The American Economic Review, 2012, 102(4), 1241-1278.
Durden, Garey C., and Larry V. Ellis. The Effects of Attendance on Student Learning in
Principles of Economics. The American Economic Review, 1995, 85(2), 343-346.
Fredriksson, Peter, and Björn Öckert. Resources and Student Achievement – Evidence from a
Swedish Policy Reform. The Scandinavian Journal of Economics, 2008, 110(2), 277-296.
20
Fredriksson, Peter, Björn Öckert, and Hessel Oosterbeek. Long-term effects of class size.
Quarterly Journal of Economics, 2013, 128(1), 249-285.
Fryer, Roland G. Financial Incentives and Student Achievement: Evidence from Randomized
Trials. The Quarterly Journal of Economics, 2011, 126(4), 1755-1798.
Gerber, Marius, Sven Grund, and G. Grote. Distributed Collaboration Activities in a Blended
Learning Scenario and the Effects on Learning Performance. Journal of Computer Assisted
Learning, 2008, 24(3), 232-244.
Heinesen, Eskil. Estimating Class-size Effects using Within-school Variation in Subject-specific
Classes. The Economic Journal, 2010, 120(545), 737–760.
Hoxby, Caroline M.. The Effects of Class Size on Student Achievement: New Evidence from
Population Variation. The Quarterly Journal of Economics, 2000, 115 (4), 1239-1285.
Jiang, Mingming, and Evelyn Ting. A Study of Factors Influencing Students’ Perceived Learning
in an Web-based Course Environment. International Journal of Educational
Telecommunications, 2000, 6(4), 317–338.
Lavy, Victor. Evaluating the Effect of Teachers’ Group Performance Incentives on Pupil
Achievement. Journal of Political Economy, 2002, 110(6), 1286-1317.
Leuven, Edwin, Hessel Oosterbeek, and Marte Rønning. Quasi-experimental Estimates of the
Effect of Class Size on Achievement in Norway. The Scandinavian Journal of Economics,
2008, 110(4), 663-693.
Leuven, Edwin, Hessel Oosterbeek, and Bas van der Klaauw. The effect of financial rewards on
students’ achievement: Evidence from a randomized experiment. Journal of the European
Economic Association, 2010, 8(6), 1243-1265.
Levitt, Steven D., John A. List, Susanne Neckermann and Sally Sado. The Impact of Short-term
Incentives on Student Performance. Working Paper, University of Chicago, September 2011.
Neal, Derek. The Design of Performance Pay in Education. In: Erik A. Hanushek, Stephen J.
Machin and Ludger Woessmann, The Handbook of Economics of Education, 2011, 495-550.
Picciano, Anthony G. Beyond Student Perceptions: Issues of Interaction, Presence and
Performance in an Online Course. Journal of Asynchronous Learning Networks, 2002, 6(1),
21-40.
Romer. David. Do Students go to Class? Should they?. Journal of Economic Perspectives, 1993,
7(3), 167-174.
Rovai, Alfred P., and Kirk T. Barnum. On-line Course Effectiveness: An Analysis of Student
Interactions and Perceptions of Learning. Journal of Distance Education, 2003, 18(1), 57–73.
Shea, Peter, Eric Frederickson, Alexandra Pickett, William Pelz , and Karen Swan. Measures of
learning effectiveness in the SUNY Learning Network. Online Education, 2001, 2.
Staiger, Douglas, and James H. Stock. Instrumental Variables Regression with Weak Instruments.
Econometrica, 1997, 65(3), 557-586.
21
Stinebrickner, Ralph and Todd R. Stinebrickner. The Causal Effect of Studying on Academic
Performance. The B.E. Journal of Economic Analysis & Policy, 2008, 8(1).
Stock, James H., Jonathan H. Wright, and Motohiro Yogo. A Survey of Weak Instruments and
Weak Identification in Generalized Method of Moments. Journal of Business & Economics
Statistics, 2002, 20(4), 518-529.
Swan, Karen, Peter Shea, Eric Frederickson, Alexandra Pickett, William Pelz, and Greg Maher.
Building Knowledge Building Communities: Consistency, Contact and Communication in
the Virtual Classroom. Journal of Educational Computing Research, 2000, 23(4), 389–413.
Vygotsky, Lev S. Mind in society. Harvard University Press, 1978, Cambridge, MA.
22
168
172
Term of Studies
Gender (1: male; 0: female)
COMPUTER USE (years)
63.99
6.47
93.20
16.91
9.94
0.71
4.10
2.98
3.28
36.94
72.73
16.72
4.95
50.51
8.75
2.99
0.45
1.47
1.70
1.07
23.59
9.49
15.57
10.96
STD
87.95
Max
48.57
20.00
1.00
9.00
9.00
5.00
0.50
38.11
23.00 358.00
2.50
0.00
0.00
3.00
0.00
1.00
12.00 105.00
60.00 100.00
1.00 122.00
22.73
Min
6.83
102.51
18.29
10.06
0.71
4.04
2.90
3.35
55.33
72.41
20.47
65.84
Mean
5.57
59.90
8.95
2.47
0.46
1.51
1.62
1.05
20.87
9.67
19.45
10.03
STD
87.95
Max
95.00
48.57
18.00
1.00
9.00
9.00
5.00
0.50
38.11
23.00 358.00
3.70
5.00
0.00
3.00
0.00
1.00
29.00 105.00
60.00
1.00 122.00
41.59
Min
Distance = 1
6.17
83.26
15.17
10.23
0.70
4.08
3.05
3.19
19.01
72.90
12.46
62.11
Mean
4.46
38.69
8.06
3.04
0.46
1.39
1.81
1.10
4.70
9.15
8.93
11.57
STD
42.86
20.00
1.00
9.00
8.00
5.00
28.00
90.00
39.00
81.59
Max
0.50
22.31
33.00 186.00
2.50
0.00
0.00
3.00
0.00
1.00
12.00
60.00
1.00
22.73
Min
Distance = 0
0.41
0.01
0.02
0.70
0.90
0.87
0.58
0.34
0.00
0.74
0.00
0.03
TT
Notes. This table shows the descriptive statistics for the full sample and the sample split by DISTANCE whereby DISTANCE is 1 if travel time to university
by public transport is above the median and 0 if it is below median. Sample statistics include number of obeservations (Obs), mean, standard deviation (STD),
smallest (Min) and largest (Max) observation. The last column shows the result of a T-Test (TT) for equality of the means (DISTANCE 1 and 0). A p-value
of 0.05 or smaller means that distribution equality is rejected on the 5 percent significance level.
DISTANCE is equal to 1 if travel time to university by public transportation is above the median and 0 otherwise. ACHIEVEMENT is the percentage of
points earned in the final exam in the course out of the number of possible points. PREVSCORE is the percentage of reachable points a student obtained on
the final exam of the first-year course. INVOLVMENT is the extent of activity on the discussion board (number of postings). COMPUTER USE is the
number of years a student has been using computers. INVOLVEMENT RATIO is the ratio of INVOLVEMENT and number of postings read (READ).
OTHERS is the sum of the individual levels of involvement of the other team members. DISTRIBUTION is the standard deviation of involvement in the
whole team. NETREAD is READ minus INVOLVEMENT. READPERCENT is READ, scaled by the total number of readable entries posted in the
student’s group. ATTITUDE is the attitude towards the usefulness of discussion boards. In an online entry survey, before students get exposed to the actual
course, they are asked: “As how useful do you regard the following study methods? … [Online discussion boards].” Higher scores (1 to 5) indicate more
positive attitudes towards the usefulness of the discussion board.
172
167
Age above 19
DISTRIBUTION (Std. Dev. of INVOLVEMENT in group)
172
ATTITUDE towards the usefulness of discussion boards
172
167
RAW DISTANCE (travel time to university in minutes)
172
160
PREVSCORE (Score in previous term)
OTHERS (sum of INVOLVEMENT in group)
172
INVOLVEMENT
INVOLVEMENT-READING RATIO
172
172
ACHIEVEMENT
Obs Mean
Variable
Full sample
Table 1: Descriptive Statistics
23
0.17** 0.71*** -0.03
-0.02 0.82*** -0.02
-0.03 0.67***
11 OTHERS (sum of INVOLVEMENT in group)
12 DISTRIBUTION (Std. Dev. Of INVOLVEMENT in group)
0.03
1.00
6
-0.05
0.08
0.08
0.19**
0.06
0.05
1.00
7
8
0.34**
0.13
0.11
0.03
0.02
-0.04
0.07
-0.14*
-0.09
-0.04
0.01
-0.15** -0.21*** 1.00
-0.10 0.36***
-0.04
1.00
5
0.18** 0.26*** -0.02
-0.03
0.01
-0.01
-0.04
0.08
1.00
4
Notes. This table shows pairwise correlations. Variables are defined in table 1. *** p<0.01, ** p<0.05, * p<0.1.
0.07
0.11
10 INVOLVEMENT-READING RATIO
0.03
-0.09
-0.03
-0.11
9 COMUTER USE (years)
0.12
8 Gender (1: male; 0: female)
-0.02
-0.03
-0.00
-0.01
0.19**
0.13
-0.32*** 0.03
7 Term of Studies
6 Age above 19
5 ATTITUDE towards the usefulness of discussion boards
0.17** 0.26*** -0.03
1.00
0.26*** -0.03
1.00
3
4 DISTANCE
2 INVOLVEMENT
2
3 PREVSCORE (Score in previous term)
1
0.11
1 ACHIEVEMENT
1
1
10
1
11
0.13 0.38*** 0.72***
0.08 0.31***
-0.03
1.00
9
Table 2: Correlations
24
Table 3: First-stage regressions: Determinants of INVOLVEMENT
(1)
DISTANCE (1: above median, 0: otherwise)
(2)
(3)
0.365
(0.69)
10.523
(1.64)
yes
3.363***
(3.60)
-0.068
(-0.13)
3.49
(0.44)
yes
7.889***
(3.02)
2.436***
(2.72)
0.307
(0.55)
4.153
(0.58)
yes
160
0.26
167
0.24
158
0.28
8.430***
(3.23)
ATTITUDE towards the usefulness of discussion boards
Age (years above 19)
Constant
Group fixed effects
Observations
R-squared
Notes . This table presents OLS regressions of INVOLVEMENT on two
instrumental variables and control variables. Variables are defined in table 1. Robust tstatistics, based on standard errors clustered at the student group level, are in
parentheses. *** p<0.01, ** p<0.05, * p<0.1.
25
Table 4: Main results: The effect of INVOLVEMENT on ACHIEVEMENT
INVOLVEMENT
PREVSCORE (Score in previous term)
Age (years) above 19
Constant
Group fixed effects
Observations
Endogenous variable
Instrument(s)
F-stat of excluded instrument
p-value of Hausman test
p-value of Hansen-Sargan test
(1)
OLS
(2)
IV
(3)
IV
(4)
IV
0.102***
(2.67)
0.304***
(3.07)
-2.399***
(-4.41)
42.063***
(4.57)
yes
0.576**
(2.40)
0.411***
(3.29)
-2.611***
(-4.50)
29.408**
(2.28)
yes
0.546**
(2.14)
0.370***
(3.13)
-2.492***
(-4.58)
30.604***
(2.65)
yes
0.568***
(2.68)
0.418***
(3.48)
-2.775***
(-4.96)
27.896**
(2.35)
yes
172
160
INVOLVEMENT
DISTANCE
9.90
<0.01
167
158
INVOLVEMENT
INVOLVEMENT
ATTITUDE
DISTANCE, ATTITUDE
12.59
7.45
<0.01
<0.01
0.94
Notes. This table presents OLS and 2SLS regressions of ACHIEVEMENT on INVOLVEMENT and control
variables. Variables are defined in table 1. Robust t-statistics, based on standard errors clustered at the student
group level, are in parentheses. *** p<0.01, ** p<0.05, * p<0.1.
26
35.968***
(3.10)
yes
0.405***
(3.65)
-2.093***
(-3.98)
-1.927**
(-2.04)
0.819**
(2.44)
43.514***
(3.34)
yes
0.367***
(2.79)
-1.942***
(-3.04)
1.246*
(1.78)
-0.313
(-1.61)
(3)
IV
155
160
160
INVOLVEMENT INVOLVEMENT INVOLVEMENT
DISTANCE
DISTANCE
DISTANCE
9.85
15.51
4.68
0.384***
(2.83)
-2.778***
(-4.52)
0.381
(0.44)
-0.306
(-0.14)
-0.023
(-0.06)
28.808**
(2.14)
yes
0.547**
(2.31)
(2)
IV
172
39.760***
(4.35)
yes
0.228***
(2.67)
0.303***
(3.04)
-2.372***
(-4.57)
(4)
OLS
(5)
IV
167
I-RATIO
ATTITUDE
12.08
26.626**
(2.11)
yes
0.809**
(2.26)
0.338***
(2.76)
-2.409***
(-4.60)
Notes. This table presents OLS and 2SLS regressions of ACHIEVEMENT on INVOLVEMENT and control variables.
Variables are defined in table 1. Robust t-statistics, based on standard errors clustered at the student group level, are in
parentheses. *** p<0.01, ** p<0.05, * p<0.1.
Observations
Endogenous variable
Instrument
F-stat of excluded instrument
Group fixed effects
Constant
Computer experience (years)
Gender (1: male, 0: female)
Term of studies
Age (years above 19)
PREVSCORE (Score in previous term)
INVOLVEMENT-READING RATIO
DISTRIBUTION (Std. Dev. of INVOLVEMENT in group)
OTHERS (sum of INVOLVEMENT of others)
INVOLVEMENT
(1)
IV
Table 5: Additional results: The effect of INVOLVEMENT on ACHIEVEMENT
27