Monitoring Affect States During Effortful
Problem Solving Activities
Sidney K. D’Mello1, Blair Lehman1, and Natalie Person2
1
Institute for Intelligent Systems, University of Memphis, Memphis, TN, USA
2
Department of Psychology, Rhodes College, Memphis, TN, USA
Abstract. We explored the affective states that students experienced during effortful problem solving activities.
We conducted a study where 41 students solved difficult analytical reasoning problems from the Law School
Admission Test. Students viewed videos of their faces and screen captures and judged their emotions from a set
of 14 states (basic emotions, learning-centered emotions, and neutral) at relevant points in the problem solving
process (after new problem is displayed, in the midst of problem solving, after feedback is received). The results
indicated that curiosity, frustration, boredom, confusion, happiness, and anxiety were the major emotions that
students experienced, while contempt, anger, sadness, fear, disgust, eureka, and surprise were rare. Follow-up
analyses on the temporal dynamics of the emotions, their contextual underpinnings, and relationships to problem
solving outcomes supported a general characterization of the affective dimension of problem solving. Affective
states differ in: (a) their probability of occurrence as regular, routine, or sporadic emotions, (b) their temporal
dynamics as persistent or random emotions, (c) their characterizations as product or process related emotions,
and (d) whether they were positively or negatively related to problem solving outcomes. A synthesis of our
major findings, limitations, resolutions, and implications for affect-sensitive artificial learning environments are
discussed.
Keywords. Affective states, emotions, problem solving, LSAT, intelligent tutoring systems
INTRODUCTION
Learning is an immersive process that is optimized when accompanied by problem solving activities.
Problem solving serves as a catalyst towards deep learning because it forces students to be active
participants in their learning rather than passive information receivers (Bransford, Brown, & Cocking,
2000). Deep learning involves critically examining facts and ideas, integrating new information into
existing structures, and exploring relationships between ideas. This can be contrasted with shallow
learning that primarily involves the accumulation of information (Entwistle, 1988; Graesser, Jeon, &
Dufty, 2008; Prosser & Trigwell, 1999). The importance of problem solving in promoting deep
learning has not been overlooked by developers of Intelligent Tutoring Systems (ITSs). Several ITSs
have incorporated some aspects of problem solving as part of their learning activities (Aleven &
Koedinger, 2002; Aleven, McLaren, Roll, & Koedinger, 2006; Anderson, 1990; Gertner & VanLehn,
2000; Lesgold, Lajoie, Bunzo, & Eggan, 1992).
The effectiveness of problem solving in promoting learning at deeper levels of comprehension
can be attributed to the deployment of key cognitive and metacognitive processing. Cognitive
processes such as knowledge acquisition, information association, causal reasoning, and inference
generation coupled with metacognitive processes such as planning, re-planning, and monitoring
continually operate throughout the problem solving process (Anderson, Douglass, & Qin, 2005;
Brown, 1987; Chi, Siler, Jeong, Yamauchi, & Hausmann, 2001; Cromley, Azevedo, & Olson, 2005;
Efklides, 2006; Jonassen, Peck, & Wilson, 1999). Consequently, significant attention has been
devoted to the cognitive and metacognitive aspects of learning and problem solving (Anderson et al.,
2005; Biggs, 1996; Boekaerts, Pintrich, & Zeidner, 2000; Graesser, D'Mello, & Person, 2009; Winnie,
2001).
With the exception of anxiety, however, the role of affective processes in deep learning and
problem solving has received considerably less attention (Pekrun, Goetz, Daniels, Stupnisky, &
Raymond, 2010). The affective link is critical because affective processes are inextricably bound to
cognitive and metacognitive processes during learning (Barrett, 2009; Brosch, Pourtois, & Sander,
2010; Frijda, 2009; Izard, 2007; Lazarus, 2000; Moors, 2009; Russell, 2003; Scherer, 2009; Schwarz,
in press; Schwarz & Skurnik, 2003; Stein, Hernandez, & Trabasso, 2008). The importance of affect is
further elevated in problem solving activities because solving problems in mathematics and science is
inevitably accompanied by the natural steps of making mistakes and recovering from them. Students
get confused when confronted by contradictions, misconceptions, and salient contrasts (D'Mello &
Graesser, 2010a; Graesser, Lu, Olde, Cooper-Pye, & Whitten, 2005; VanLehn, Siler, Murray,
Yamauchi, & Baggett, 2003), they get frustrated by obstacles and challenges that block goals (Dweck,
2002; Klein, Moon, & Picard, 2002), and they experience anxiety when failure is attributed to events
outside of their control (Heider, 1958; Weiner, 1986). Persistent failure might eventually lead to
boredom and disengagement and other problematic long term effects such as attrition and dropout
(Csikszentmihalyi, 1990; Larson & Richards, 1991; Mann & Robinson, 2009; Pekrun et al., 2010).
Alternatively, students experience happiness and delight when tasks are completed, eureka moments
when challenges are unveiled and major discoveries are made, and flow-like states when they are so
engaged in problem solving that time and fatigue disappear (Csikszentmihalyi, 1990). Simply put, the
importance of emotions during problem solving cannot be underestimated because student affective
states can impede or facilitate the problem solving process by modulating cognitive processes in
striking ways (Clore & Huntsinger, 2007; Fielder, 2001; Isen, 2008; Schwarz, in press; Spering,
Wagener, & Funke, 2005).
Although we are beginning to make progress towards understanding the role of affect during
academic problem solving, several important questions still remain unanswered. These include: (a)
what are the affective states that accompany problem solving activities? (b) how does affect interact
with cognitive, metacognitive, and motivational processes?, (c) how do contextual factors such as task
demands, interruptions, and other perturbations impact affect?, (d) how do individual differences (e.g.,
affective traits, motivation, aptitude) moderate affective processes?, (e) what are the critical
interactions among affect, cognition, metacognition, motivation, context, and individual differences
that predict problem solving performance? There is currently no model or theory that addresses this set
of questions, although some important insights have emerged over the past few decades.
Theories of Affect during Problem Solving
One of the first theories linking affect to performance is the classic Yerkes-Dodson law (1908). The
law proposes a curvilinear inverted U-shaped relationship between arousal and performance that
varies by task demands. Performance is low when arousal is too low or too high but is optimal when
arousal is just right. The critical level of arousal to facilitate optimal performance is related to task
complexity. In order to facilitate concentration, a lower level of arousal is optimal for tasks with high
cognitive demands such as solving difficult problems.
Despite its popularity, the Yerkes-Dodson law is somewhat limited because it only addresses the
arousal dimension of the affective experience, while ignoring valence. However, discriminating
valenced from valenceless arousal is quite important because valenceless arousal is unlikely to impact
performance (Isen, Daubman, & Nowicki, 1987). For example, climbing a flight of stairs a few times
does get the heart rate up, but this type of increased arousal does not impact performance (Isen, 2003).
On the other hand, positive and negative affective experiences do play a role in problem solving. For
example, flexibility, creative thinking, and conceptually-driven top-down processing have been linked
to positive affect (Clore & Huntsinger, 2007; Fielder, 2001; Isen, 2008; Isen et al., 1987), while
negative affect has been associated with a localized, narrowly-focused, and stimulus-driven bottom-up
cognitive processing (Bless & Fielder, 1995; Hertel, Neuhof, Theuer, & Kerr, 2000; Schwarz, 1990,
2000).
A two dimensional valence-arousal affect model (Barrett, 2006; Russell, 2003) does serve as a
useful framework to understand the impact of affective states on problem solving activities. However,
there is reason to challenge the adequacy of basing an entire theory of affect on arousal and valence
alone (Fontaine, Scherer, Roesch, & Ellsworth, 2007; Schwarz & Skurnik, 2003), because the
relationship between positive and negative affective states and performance outcomes is more
complex, and, in some cases counterintuitive. For example, certain negative emotions, such as
confusion, could have a positive effect on problem solving because they force students to stop and
think in order to resolve troublesome impasses (Craig, Graesser, Sullins, & Gholson, 2004; D’Mello &
Graesser, in press-b; Graesser, Chipman, King, McDaniel, & D'Mello, 2007).
Focusing on general moods (i.e., positive or negative) during problem solving also runs the risk
of overlooking the ebb and flow of dynamically changing affective states. It is more informative to
monitor the fluctuations from one emotion to another and how these emotional transitions impact
problem solving performance than simply noting whether a learner is generally in a positive or
negative mood while solving a problem (D'Mello & Graesser, 2010a; McLeod, 1988). Therefore, a
model of emotion and problem solving should extend beyond a basic valence-arousal framework and
address how emotions such as confusion, frustration, and anxiety naturally arise during the problem
solving process.
According to goal-appraisal theories of emotion (Mandler, 1976, 1999; Stein et al., 2008; Stein &
Levine, 1991), when learners are deeply engaged, they attempt to assimilate new information into
existing knowledge schemas. When new or discrepant information is detected, attention shifts to the
discrepant information, the autonomic nervous system increases in arousal, and the learner
experiences a variety of possible affective states depending on the context, the amount of change, and
whether important goals are blocked. In the case of extreme novelty, the event evokes surprise. When
there is positive feedback on an action or achievement of a difficult goal, the emotion is positive, as in
the case of delight or eventually contentment. In contrast, confusion and frustration occur when the
discrepancy or novelty triggers an impasse that blocks an important superordinate learning goal (e.g.,
solving a difficult problem or understanding a complex topic) and possibly results in the students
getting stuck. The students then need to actively problem solve to resolve their confusion and
frustration. For example, getting stuck and not being able to move past an obstacle due to the absence
of an available plan would be interpreted as intensely negative because goal attainment is obstructed.
On the other hand getting stuck, being confused, but arriving at a solution via a rare eureka moment
might represent a negative to positive transition (D'Mello & Graesser, 2010a).
Research Questions
The contemporary theories of emotion (Barrett, 2009; Brosch et al., 2010; Frijda, 2009; Izard, 2007;
Lazarus, 2000; Moors, 2009; Russell, 2003; Scherer, 2009; Schwarz, in press; Schwarz & Skurnik,
2003; Stein et al., 2008) convey broad links between cognition and emotions, but they do not directly
explain and predict the sort of emotions that occur during learning and problem solving. There is also
insufficient empirical research to conceptualize a broad, yet sufficiently detailed, model of emotional
dynamics during problem solving. Therefore, the main goal of this paper is to collect and analyze data
at a sufficiently detailed level of resolution as a first step towards developing such a model.
Specifically, we were interested in answering the following questions: (1) What are the affective states
that students experience during complex problem solving activities?, (2) What are the temporal
dynamics of these affective experiences?, (3) To what extent do contextual factors (e.g., feedback,
response times) impact student affect?, and (4) How are the affective states linked to performance?
We addressed these questions with an exploratory study that tracked student emotions while
solving difficult analytical reasoning problems. The study is exploratory because although there has
been some research on monitoring emotions during one-on-one tutoring sessions with ITSs (Arroyo et
al., 2009; Baker, D'Mello, Rodrigo, & Graesser, 2010; Conati & Maclaren, 2009; D’Mello &
Graesser, in press-a; Forbes-Riley, Rotaru, & Litman, 2008; McQuiggan, Mott, & Lester, 2008), the
research on systematic investigation of emotion during non-scaffolded problem solving is sparse and
scattered.
Our focus on analytical reasoning as the problem solving domain was motivated by four primary
reasons. First, these problems are complex and require considerable cognitive skills such as being
attentive, efficiently representing knowledge, causal reasoning, and constraint satisfaction. Second, in
addition to imposing significant cognitive demands, the problems are rife with contradictions,
incongruities, anomalies, and uncertainty – all factors that are expected to trigger a diverse set of
emotional reactions. Third, there are important practical implications for studying analytical reasoning
problems. These problems are included in standardized tests such as the Law School Admissions Test
(LSAT), a test that is required for admission into law school in the United States (LSAT, 2008).
Fourth, as discussed above, several ITSs include problem solving modules due to the widely
acknowledged importance of problem solving activities for deep learning. Therefore, a model of
students’ emotions during effortful problem solving activities can be used to inform the development
of ITSs that aspire to be responsive to students’ affective states in addition to their cognitive states.
METHOD
The study involved 41 students solving analytical reasoning problems from the Law School
Admissions Test (LSAT). Students provided judgments of their emotions via a retrospective affect
judgment procedure (Graesser et al., 2006). We focused on a model of student emotions consisting of:
(a) basic emotions (anger, disgust, fear, happiness, sadness, surprise,) (Ekman, 1992), (b) learningcentered emotions (anxiety, boredom, confusion, contempt, curiosity, eureka, frustration) (D'Mello,
Olney, & Graesser, in press; Rodrigo & Baker, in press), and (c) the non-affective neutral state.
Participants
Participants were 41 traditional college-age undergraduate students from a southern college in the
United States. The participants were selected from a population of students that were enrolled in a
college program that offered practice testing for graduate school standardized tests. There were 26
females (63%) and 15 males (37%). Thirty-two of the 41 students (78%) were Caucasians and the
remaining nine were African-Americans (22%). All of the participants indicated that they were
interested in attending law school. Participants received monetary compensation of $30 for their
participation.
Procedure
The experiment was divided into two phases. Students solved analytical reasoning problems for 35
minutes in Phase 1. Phase 2 involved a retrospective affect judgment procedure.
Phase 1: Problem Solving. Students signed an informed consent before beginning the experiment.
They were instructed that they would be solving 28 analytical reasoning problems and that their
problem solving sessions would be videotaped. Although all of the students indicated that they were
interested in attending law school, we attempted to maximize each student’s incentive to do well on
the problems by offering a monetary compensation. Hence, students were informed that they would be
paid two dollars for each correct answer they provided. All students were paid $30 at the end of the
experiment, regardless of the amount of problems answered correctly. A sample problem is presented
in the Appendix. Each problem had a scenario and approximately 5-6 sub-questions pertaining to that
scenario.
Students interacted with two software programs that were displayed on a Tablet PC. The left half
of the screen displayed a customized program (see Figure 1) that: (A) displayed each problem
scenario, (B) asked a specific question pertaining to the scenario (i.e., a sub-problem), (C) displayed a
set of answer alternatives, one of which represented the correct answer, (D) allowed students to select
an answer, (E) let students finalize their choice by clicking “next”, and (F) provided feedback
(“Correct”, or “Incorrect”) (not shown in Figure 1). Students were not allowed to return to a problem
after it had already been completed. The program also displayed the proportion of elapsed time and the
proportion of completed problems via two progress bars (see bottom of Figure 1).
Effectively solving the analytical reasoning problems required a considerable amount of thought
and effort including knowledge representation, drawing diagrams, taking notes, making inferences,
organizing knowledge, and other related activities. Students used a software application, Windows
Journal™, to take notes and draw with a stylus. The Windows Journal program was displayed on the
right half of the screen (Figure 1).
In order to expand the range of emotions that students may experience while solving the LSAT
problems, we experimentally manipulated the feedback the system provided to students. Feedback was
manipulated so that incorrect feedback was randomly provided for 25% of the responses (i.e.,
providing negative feedback to correct responses and vice versa). Therefore, there were four feedback
levels, (a) correct response + positive feedback, (b) incorrect response + negative feedback, (c) correct
response + negative feedback, and (d) incorrect response + positive feedback. Levels (a) and (b)
represent situations where accurate feedback was provided (75% of the time), while contradictory
feedback was provided for levels (c) and (d) (25% of the time). We randomly selected between cases
(c) and (d) for situations in which contradictory feedback was provided.
It should be noted that the use of false feedback was not deemed to have any harmful long-term
effects because both positive and negative feedback were provided and students were not exposed to
any patently incorrect domain content. Furthermore, the experimental protocol was approved prior to
data collection and students were fully debriefed at the end of the experiment.
Fig. 1. Annotated screen shot of problem solving environment
The experimenter left the room after demonstrating the software interfaces to the students.
Students interacted with the system for approximately 35 minutes (the analytical reasoning section of
the LSAT is 35 minutes long). Four streams of information were collected during the problem solving
session. First, the students’ face was recorded with a commercial webcam. The recording also
included speech and other audio generated during the interaction. A video of the students’ screen was
recorded using a screen capture program (Camtasia Studio™). Students’ mouse movements (i.e., the
movements of the stylus of the Tablet PC) were recorded at a sampling rate of 10Hz. Finally, a time
stamped log file that included information on the problems the students were working on, their
responses, the feedback that was provided, what buttons were clicked, and other similar features was
recorded for offline analysis.
Phase 2: Retrospective Affect Judgment.
Judgment Students were given a five-minute
minute break after the problem
solving phase of the experiment. Students then completed a retrospective affect judgment protocol
(Afzal & Robinson,, 2009; Graesser et al., 2006; Kort, Reilly, & Picard, 2001).
2001) The procedure began
by synchronizing and displaying the videos of the students’ face and screen that were captured during
the interaction (see Figure 2). Videos of the screen were included to facilitate the affect judgment
procedure by allowing students
students to incorporate contextual factors of the problem solving process with
their facial expressions. Students
Students provided affect ratings over the course of viewing these videos.
Fig. 2. Retrospective affect judgment protocol. The left monitor displays a video of the student’s screen while
the right monitor displays a video of the student’s face.
Studentss were provided with a list of affective states (anger,
(
anxiety
ety, boredom, confusion,
contempt, curiosity, disgust, eureka, fear, frustration, happiness, sadness, surprise, and neutral) with
definitions (see Table 1). These definitions were specifically tailored to the problem solving task.
They were only
ly designed to serve as a guide because studentss are more likely to rely on their personal
experiences associated with experiencing, expressing, and labeling these states (i.e., folklore) (Russell,
2003). Hence, studentss based their judgments on: (a) videos of their faces, (b) screen captures of the
Tablet PC (i.e., the context), (c) recent memories of the interaction,
interaction (d) definitions of the affective
states, and (e) personal experiences associated with these emotions.
Students were required to make affect judgments at predetermined points in the session that we
deemed to be relevant. These included: (a) Problem Onset - seven seconds after a new problem was
displayed, (b) During Solution - halfway between the presentation of the problem and the submission
of the response, and (c) After Feedback - three seconds after the feedback was provided. These time
parameters were selected on the basis of pilot testing with the present system and previous research
involving the retrospective affect judgment protocol (Graesser et al., 2007).
The face and screen videos automatically paused at these affect judgment points where students
were required to make a judgment. The system also allowed students to rewind the videos and alter
their current judgment. In addition to the three pre-specified points where students were forced to
make an affect judgment (fixed judgments), students were able to manually pause the videos and
provide affect judgments at any time (spontaneous judgments).
It is worth noting three important points pertaining to the present affect judgment methodology.
First, this procedure was adopted because it affords monitoring students’ affective states at multiple
points, with minimal task interference, and without students knowing that these states were being
monitored. Second, it has been previously used with some success (Graesser et al., 2006; Rosenberg &
Ekman, 1994). Third, the offline affect annotations obtained via this retrospective protocol correlate
with online recordings of facial activity and gross body movements in expected directions (D'Mello &
Graesser, 2010b). Although all methods have limitations, the present method appears to be a viable
approach to track emotions at a relatively fine-grained temporal resolution.
Table 1
Definition of affective states
Affective State Definition
Anger
Anxiety
Boredom
Confusion
Contempt
Curiosity
Disgust
Eureka
Fear
Frustration
Happiness
Neutral
Sadness
Surprise
negative affect toward material or person to an extreme degree
nervousness, anxiety, negative self-efficacy, embarrassment
uninterested in the current problem
poor comprehension of material, attempts to resolve erroneous belief
annoyance and/or irritation with another person
desire to acquire more knowledge or learn the material more deeply
annoyance and/or irritation with the material and/or their abilities
sudden realization about the material, a ha! moment
feelings of panic and/or extreme feelings of worry
difficulty with the material and an inability to fully grasp the material
satisfaction with performance, feelings of pleasure about the material
displays no visible affect, at a state of homeostasis
feelings of melancholy, beyond negative self-efficacy
genuinely does not expect an outcome or feedback
RESULTS AND DISCUSSION
The analyses were designed to address the four research questions presented in the Introduction.
Analysis 1 investigated the incidence of the affective states that students experienced during problem
solving. Analysis 2 involved an investigation of the temporal dynamics of these states. Analysis 3
investigated associations between contextual factors and students’ emotions. The contextual factors
examined were problem phase, judgment type (fixed vs. spontaneous), feedback type, problem
difficulty, and response time. Finally, Analysis 4 investigated the extent to which affective states were
related to problem solving outcomes. Specifically, we investigated whether student affect while
solving a problem could predict how well the problem was solved.
Analysis 1. Incidence of Affective States
A frequency analysis on the self-reported
self reported affect judgments indicated that there were 772 (M
( = 19, SD
= 5) affect judgments after a new problem was presented, 768 (M
( = 19, SD = 5) judgments at the
midpoint of the problem solving process, and 766 (M
( = 19, SD = 5) judgments after feedback was
provided. In addition to these 2,306 judgments at fixed points (i.e.,
(i.e. when students were required to
provide an affect judgment), there were 486 spontaneous judgments (M
( = 12, SD = 11). All students
provided at least
st one spontaneous affect judgment, although there was considerable variation among
the students. Taken together, there were 2,792 (M
( = 68, SD = 19) affect judgments from 41 students.
Figure 3 depicts examples of some of the affective expressions of the students.
students.
Fig. 3. Examples of affective states
Table 2 provides statistics on the occurrence of the various affective states. A repeated measures
ANOVA indicated that there were significant differences in the proportion of emotions experienced by
the students, F(13,
(13, 520) = 35.31, MSe = .007, p < .001, η2 = .469. Bonferroni posthoc tests revealed
the following pattern in the data: (Boredom = Confusion = Curiosity = Frustration) > Anger, Anxiety,
Contempt, Disgust, Eureka, Fear, Sadness, Surprise (p < .05). One minor exception to this pattern was
that there were no significant differences between the occurrences of boredom and anxiety. Happiness
occurred more frequently than fear and sadness, less frequently than frustration, and was on par with
the other emotions.
Table 2
Distribution of affective states
Frequencies
Affective States 000
N
P
Routine
Boredom
Confusion
Curiosity
Frustration
39
36
33
39
Sporadic
Anxious
Happiness
Proportions
000
M
SD
.951
.878
.805
.951
.106
.092
.138
.105
25
35
.610
.854
Exceptional
Contempt
Eureka
Anger
Disgust
Fear
Sadness
Surprise
17
21
17
26
6
13
25
Neutral
41
One-sample t-test
000
t(40)
p
d
.108
.062
.142
.071
3.14
4.01
3.85
4.68
< .010
< .001
< .001
< .001
.49
.63
.60
.73
.042
.055
.045
.049
-.162
.300
.112
.763
-.24
.04
.415
.512
.415
.634
.146
.317
.610
.027
.025
.022
.030
.008
.012
.027
.047
.039
.039
.037
.028
.024
.031
-.351
-4.58
-5.12
-3.97
-10.3
-11.1
-5.52
< .010
< .001
< .001
< .001
< .001
< .001
< .001
-.55
-.72
-.79
-.62
-1.6
-1.7
-.84
1.00
.311
.208
Notes. N = number of students that experienced the state at least once. P = proportion of students that
experienced the state at least once.
A set of follow-up analyses attempted to isolate a subset of the affective states that occurred at
levels greater than chance, where ℎ = (1 − )/ = (1 − .311)/13 = .053.
One-sample t-tests comparing the proportion of each emotion to the chance level of .053 revealed the
following patterns in the data: (a) boredom, confusion, curiosity, and frustration occurred at levels
greater than chance, (b) anger, contempt, disgust, eureka, fear, sadness, and surprise occurred at levels
less than chance, and (c) anxiety and happiness occurred at chance levels (see Table 2).
These data support a tripartite classification of the emotions that accompany effortful problem
solving activities: (a) routine emotions that include boredom, confusion, curiosity, frustration, (b)
sporadic emotions such as anxiety and happiness, and (c) exceptional emotions such as anger,
contempt, disgust, eureka, fear, sadness, and surprise. The four routine emotions occurred at levels
greater than chance; they comprised 64% of the observations (after excluding neutral), and, on
average, these four emotions were experienced by 90% of the students. The two sporadic emotions
occurred at random levels, comprised 14% of the observations, and occurred in 73% of the sessions
(on average). Finally, the seven exceptional emotions occurred at levels less than chance. They
collectively comprised 22% of the observations and, on average, were observed in less than half (44%)
of the sessions.
Ekman’s (1992) basic emotions have been at the forefront of emotion research for the last several
decades. In order to determine if they are equally applicable to problem solving, we compared the
occurrence of the basic emotions (anger, disgust, fear, happiness, sadness, surprise) to the learningcentered emotions (anxiety, boredom, confusion, contempt, curiosity, eureka, frustration). A paired
sample t-test confirmed that the basic emotions (M = .154, SD = .104) occurred at significantly lower
rates than the learning-centered emotions (M = .535, SD = .180), t(40) = -11.761, p < .001, d = 2.59.
This finding is on par with previous studies (D'Mello, Craig, Sullins, & Graesser, 2006; Lehman,
Matthews, D'Mello, & Person, 2008) that indicate that learning activities involve more complex
cognitive-affective amalgamations rather than the basic emotions.
Analysis 2. Temporal Dynamics of Affective States
Our results so far indicate that there are graded differences in the affective experiences that
accompany effortful problem solving activities. The present analysis attempted to compare the
persistence of the states. Persistence refers to a property by which an affective state observed at time is also observed at + 1. In other words, a state () can be considered to be persistent if experiencing
it at one time interval increases the likelihood of experiencing the state at the subsequent time interval
i.e., ( → ). On the other hand, an affective state is ephemeral if experiencing it at decreases
the likelihood that it will be observed at + 1. Finally, an affective state is random if observing it at is not related to the probability of its occurrence at + 1.
We utilized the Likelihood metric (Equation 1) (D'Mello, Olney, & Person, in press; D’Mello,
Taylor, & Graesser, 2007) in an attempt to characterize the affective states along this tripartite
classification scheme. The metric quantifies the likelihood that the current affective state ()
influences the next affective state (!) after correcting for the base rate of !. According to this metric,
if "( → !) ≈ 1, we can conclude that state ! reliably follows state above and beyond the prior
probability of state !. If, on the other hand "( → !) ≈ 0, then ! follows at the chance level.
Furthermore, if "( → !) < 0, then the likelihood of state ! following state is much lower than
the base rate of !.
"( → !) =
%(!|) − %(!)
1 − %(!)
(Equation 1)
Our immediate goal is to assess the likelihood that affective state observed at time is also
observed at time + 1. This can be easily accomplished by modifying the metric such that the current
state () and the next state (!) are the same (as illustrated in Equation 2).
"( → ) =
%( | ) − %( )
1 − %( )
(Equation 2)
In order to detect significant affect state persistence, we compared the likelihood of each state
repeating itself to a hypothesized mean of 0 (normalized base rate) using a one-sample t-test. The
results of the tests are presented in Table 3 where it appears that the data supports a two-way
classification scheme (persistent and random) instead of a three-way classification scheme, as there
are no instances of ephemeral states.
Table 3
Persistence of affective states
Descriptives (Likelihood)
M
SD
Persistent
Anger → Anger
Anxious → Anxious
Boredom → Boredom
Confusion → Confusion
Curiosity → Curiosity
Disgust → Disgust
Frustration → Frustration
.102
.097
.134
.066
.049
.079
.067
Random
Contempt → Contempt
Eureka → Eureka
Fear → Fear
Happiness → Happiness
Sadness → Sadness
Surprise → Surprise
-.009
.036
.185
.002
.031
-.006
Affective State
One-sample t-test
00000
t
df
p
d
.214
.216
.222
.181
.143
.196
.151
1.96
2.24
3.76
2.18
1.95
2.05
2.76
16
24
38
35
32
25
38
.068
.035
.001
.036
.060
.051
.009
.48
.45
.60
.36
.34
.40
.44
.158
.219
.377
.147
.153
.072
-.23
.75
1.20
.08
.72
-.45
16
20
5
34
12
24
.824
.460
.283
.933
.484
.654
-.06
.16
.49
.01
.20
-.08
Analysis 3. Contextual Influences on Affective States
The importance of context in shaping affective experience cannot be ignored for at least two primary
reasons (Aviezer et al., 2008; Russell, Bachorowski, & Fernandez-Dols, 2003; Stemmler, Heldmann,
Pauls, & Scherer, 2001). First, examining the context surrounding an emotional expression can lead to
a deeper explanation of the emotional experience. For example, confusion while solving a problem can
be contrasted with confusion after receiving feedback for the solution. The first form of confusion can
be attributed to being perplexed with the problem itself, while confusion after feedback is more related
to the problem solving outcome. Though similar, these two forms of confusion might have distinct
manifestations and differentially impact performance.
Context also plays an important role in disambiguating between different manifestations of the
same emotion. As an example consider surprise, an emotion with significant arousal but with an
ambiguous valence dimension. Receiving negative feedback for a correct response would probably
evoke a degree of surprise. The valence of this exemplar of surprise is likely to be negative. But
surprise could also be expected when an incorrect answer yields positive feedback. However, this type
of surprise would have positive valence and might spark enthusiasm and curiosity.
In summary, context is critical because it helps disambiguating between various exemplars of a
prototypical emotion (Russell, 2003). For example, the two forms of confusion discussed above are
different exemplars of a prototypical “confused” state. Examining confusion (i.e., the prototype) out of
context (i.e., without the exemplar) is therefore quite meaningless.
We conducted five sets of analyses that investigated relationships between the underlying context
and the emotions that emerged out of that context. The contextual factors were (a) problem phase, (b)
judgment type, (c) feedback, (d) problem difficulty, (e) response time, and (f) session time.
Problem Solving Phase. Students were prompted to make affect judgments at particular points in the
problem solving process. These judgment points occurred after the onset of a new problem (P:
problem onset), midway between the presentation of the problem and the submission of the response
(S: during solution), and after receiving feedback (F: after feedback). A 3 × 14 (problem phase ×
affect) repeated measures ANOVA investigating the distribution of emotions during the three phases
of the problem solving process (see Table 4) revealed a significant problem phase × affect interaction,
F(26, 1040) = 15.21, MSe = .059, p < .001, partial ηP2P = .275. Bonferroni posthoc tests on the
problem phase × emotion interaction revealed several interesting patterns pertaining to the extent to
which students experienced certain emotions at different points in the problem solving process. For
simplicity, we restrict our discussion to the more prominent affective states observed. These include
boredom, confusion, frustration, curiosity, happiness, and anxiety.
Table 4
Distribution of affective states during the three problem solving phases
Problem
Onset (P)
During
Solution (S)
After
Feedback (F)
Affect
M
SD
M
SD
M
SD
Sig. Patterns
Anger
Anxious
Boredom
Confusion
Contempt
Curiosity
Disgust
Eureka
Fear
Frustration
Happiness
Neutral
Sadness
Surprise
.022
.042
.106
.092
.027
.138
.030
.025
.008
.105
.055
.311
.012
.027
.039
.045
.108
.062
.047
.142
.037
.039
.028
.071
.049
.208
.024
.031
.006
.043
.115
.135
.016
.186
.013
.028
.008
.071
.012
.363
.003
.002
.018
.057
.108
.126
.041
.202
.029
.049
.027
.081
.030
.286
.017
.009
.048
.022
.089
.029
.039
.038
.046
.027
.009
.146
.144
.276
.021
.066
.095
.049
.146
.054
.087
.067
.074
.057
.032
.142
.123
.256
.044
.087
F>P>S
P > Fa
Notes. a P = S and S = F for anxiety
S>P>F
S>P>F
P=F>S
F>P>S
F>P>S
S>P=F
F>P>S
The results indicated that boredom was evenly distributed across all three judgment points (i.e., P
= S = F). This suggests that when students disengage, potential affect-inducing events such as the
presentation of a new problem or feedback do little to alleviate their boredom. On the other hand,
confusion and curiosity displayed a strikingly different (compared to boredom) pattern of occurrence
(i.e., S > P > F). These emotions were most frequently observed in the midst of problem solving,
followed by the presentation of a new problem. Experiences of confusion and curiosity were rare after
feedback was provided.
Frustration and happiness were another pair of affective states with similar occurrence patterns
(i.e., F > P > S). These states were most frequent after feedback was provided followed by the onset of
a problem. Frustration and happiness rarely occurred during the process of deriving a solution to the
problem. These affective states seem to be complimentary to confusion and curiosity in that they occur
after a resolution has been reached (i.e., feedback), as opposed to confusion and curiosity that occur
while the student tries to arrive at a solution. Consequently, confusion and curiosity appear to be
related to the problem solving process, while frustration and happiness are related to the problem
solving outcome (or product). Furthermore, although their occurrences are rare, anger and surprise
also appear to be linked to the outcome of problem solving, which is what could be expected.
Judgment Type. In addition to being required to provide judgments at the predetermined fixed points,
students also had the option of providing judgments at any time during the retrospective judgment
procedure (spontaneous judgments). The fixed judgments consisted of our best guess of when an
emotion was most likely to occur. However, the spontaneous judgments represented instances where
our estimated judgment points missed the mark. Therefore, it is quite conceivable that a different set
of affective responses can be elicited from the fixed judgment points (i.e., problem onset, during
solution, and after feedback) when compared to spontaneous judgments points.
A 2 × 14 (judgment type × affect) repeated measures ANOVA on the distribution of emotions for
fixed versus spontaneous judgments revealed a significant judgment type × affect interaction, F(13,
520) = 2.29, MSe = .010, p < .01. However, the effect size for this interaction (partial η 2 = .054) was
substantially smaller than the problem phase × affect interaction (partial η 2 = .275), indicating that
most of the variance was explained by the differences in emotions observed at the different points in
the problem solving process.
Nevertheless, there were occasional differences in the distribution of emotions across the two
judgment types. Bonferroni posthoc tests revealed that confusion was reported at significantly higher
rates for the spontaneous (M = .164, SD = .167) than the fixed (M = .086, SD = .065) judgment points.
A different pattern emerged for curiosity and happiness. Both these emotions were observed at higher
rates for the fixed judgments (curiosity: M = .121, SD = .130; happiness: M = .071, SD = .059) when
compared to the spontaneous judgments (curiosity: M = .079, SD = .136; happiness: M = .026, SD =
.069).
P
P
P
P
Feedback. Feedback is critical in both human and computer tutoring because it is directive (i.e., tells
students what needs to be fixed), facilitative (i.e., helps students conceptualize information), and has
motivational functions (Black & William, 1998; Lepper & Woolverton, 2002; Shute, 2008). Feedback
strategies of tutors have received considerable attention from educational researchers, with a handful
of meta-analyses devoted exclusively to the effectiveness of feedback as a pedagogical and
motivational tool (Azevedo & Bernard, 1995; Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Shute,
2008). However, relatively little is known about the impact of feedback on students’ affective states,
although some data is beginning to emerge (D'Mello et al., 2006; D'Mello, Craig, Witherspoon,
McDaniel, & Graesser, 2008).
The study included an experimental manipulation pertaining to the type of feedback the system
provided to students. Feedback was provided after the students submitted an answer to a problem and
can be subdivided into four categories on the basis of the quality of the student responses (correct or
incorrect) and the type of feedback provided by the system (positive or negative). The four feedback
categories are: (a) PP = correct response + positive feedback, (b) NN = incorrect response + negative
feedback, (c) PN = correct response + negative feedback, and (d) NP = incorrect response + positive
feedback. Accurate feedback was provided for categories PP and NN while contradictory feedback
was provided for categories PN and NP.
We performed a 4 × 14 (feedback type × affect) repeated measures ANOVA to investigate
associations between feedback and student affect after feedback was provided. Since our major focus
is on emotions after feedback, we excluded spontaneous affect judgments as well as judgments after
problem presentation and during problem solution. The results revealed that there was a significant
feedback type × affect interaction, F(39, 1560) = 8.92, MSe = .023, p < .001, partial η 2 = .182.
Bonferroni posthoc tests comparing the occurrence of each emotion across the different feedback
categories support several important conclusions (see Table 5).
Posthoc tests showed that boredom and curiosity were not related to feedback, confusion was
somewhat related, and frustration and happiness were strongly related to the type of feedback
provided. The fact that the occurrence of boredom was independent of the type of feedback provided
supports a general characterization of boredom during learning activities. Bored students essentially
disengage from the learning session to a point where external stimulation (via a new problem or
feedback) does little to remove students out of a state of persistent ennui. Simply put, boredom begets
more boredom.
A rather different interpretation can be applied to curiosity not being related to the type of
feedback. Curiosity is more prominent during the problem solving process (discussed above).
Therefore, the fact that feedback was not related to levels of curiosity strengthens the characterization
of curiosity as a process- rather than a product-related emotion.
Similar to curiosity, earlier we characterized confusion as a process-related emotion. However, it
appears that confusion is more prominent after negative versus positive feedback. Taken together,
these results support a more refined categorization of confusion, because this emotion appears to have
complimentary process- as well as product-related manifestations. Students experience confusion
during the problem solving process and also when they are provided negative feedback to an answer
they think is correct (i.e., the product).
Consistent with their characterization as product-related emotions, frustration and happiness were
highly related to feedback. Frustration occurred when negative feedback was provided regardless of
whether the feedback was accurate (NN) or contradictory (PN). A reverse pattern was observed for
happiness. This state was more prominent when positive feedback was provided irrespective of
whether the feedback was accurate (PP) or contradictory (NP). Therefore, the two product-related
emotions appear to be frustration (negative) and happiness (positive).
Contradictory to our expectations, it appeared that providing contradictory feedback was not
noticeably associated with students’ emotions. Instead, it is the valence (PP, NN) rather than the
presence or absence of a contradiction (PN, NP) that influenced most of the feedback-related effects. It
P
P
might be the case that the students were simply unaware of the contradictory feedback. This is a
plausible speculation since contradictory feedback was only provided for 25% of the cases.
Table 5
Distribution of affective states for different feedback types
PP
Affect
Anger
Anxiety
Boredom
Confusion
Contempt
Curiosity
Disgust
Eureka
Fear
Frustration
Happiness
Neutral
Sadness
Surprise
NN
PN
NP
M
SD
M
SD
M
SD
M
SD
Sig. Patterns
.000
.034
.055
.005
.016
.061
.005
.062
.003
.009
.333
.381
.000
.036
.000
.103
.113
.031
.081
.128
.031
.129
.022
.040
.306
.317
.000
.088
.095
.011
.146
.046
.058
.027
.078
.006
.013
.249
.014
.183
.042
.032
.176
.044
.257
.092
.162
.065
.130
.039
.044
.250
.047
.270
.094
.073
.077
.041
.057
.017
.038
.008
.071
.000
.008
.266
.000
.131
.032
.107
.229
.170
.183
.083
.137
.052
.187
.000
.052
.359
.000
.278
.106
.196
.000
.037
.065
.024
.037
.025
.012
.033
.004
.060
.174
.280
.000
.126
.000
.165
.187
.109
.173
.097
.078
.130
.026
.223
.303
.406
.000
.260
NN > NP = PP
NN > PP
NN > NP = PP a
PP > NN = PN
NN=PN<NP=PP
PP>NP>NN=PN
PP > NN = PN
NN > NP = PP
Notes a PN > NP for disgust
Problem Difficulty. We investigated the extent to which problem difficulty was associated with
students’ emotions. We focused on the emotions that occurred while the students were in the process
of solving the problem (i.e., affect judgments after problem presentation, feedback, and spontaneous
judgments were excluded). Our analyses proceeded by computing a difficulty value for each of the 28
problems. The difficulty level of a problem was operationalized as the proportion of students that
correctly solved it. In this fashion, problem difficulty was computed for all 28 problems and was
normally distributed (M = .432, SD = .285). We dichotomized this variable into low and high
difficulty problems via a median split (median = .432).
A 2 × 14 (difficulty level × affect) repeated measures ANOVA indicated that there was a
significant but small interaction between problem difficulty and affect, F(13, 520) = 2.65, MSe = .006,
p < .01, partial η 2 = .062. Bonferroni posthoc tests revealed that the significant difference was for
boredom, but not for any of the other affective states. It was found that boredom occurred at a higher
rate for easy (M = .139, SD = .151) compared to difficult problems (M = .086, SD = .113). So as could
be expected, problems that do not sufficiently challenge and engage students are linked to boredom, a
finding which is consistent with flow theory (Csikszentmihalyi, 1990) and the control-value theory of
emotions (Pekrun et al., 2010).
P
P
Response Time. We investigated associations between response time (i.e., time between problem
presentation and answer submission) and student emotions. We focused on the emotions that occurred
while the students were in the middle of solving the problem and excluded judgments after problem
presentation, feedback, and spontaneous points. Our analyses proceeded by dichotomizing each
student’s response times into a low and high group via a median split procedure (median = 73.8
seconds).
A 2 × 14 (response time × affect) repeated measures ANOVA indicated that there was a
significant interaction between response time and student affect, F(13, 520) = 3.04, MSe = .006, p <
.001, partial η 2 = .071. The significant differences were for boredom and confusion but not for any of
the other emotions. It appears that students have shorter response times when they are bored (Mlow =
.146, SDlow = .152, Mhigh = .083, SDlow = .109), but a reverse pattern is observed when students are
confused (Mlow = .10, SDlow = .113, Mhigh = .172, SDlow = .187). One interpretation to this finding is
that bored students have faster response times because they tend to move ahead without investing any
substantial cognitive resources in the task. Conversely, confused students have longer response times
because they take the time to effortfully deliberate in order to alleviate their perplexity.
P
Session Time. We investigated the extent to which session time was associated with the affective
states of the student. It is quite conceivable that students begin the session with more enthusiasm and
curiosity, which gradually transitions into boredom as time progresses. For example, within the
context of learning with ITSs, there is some evidence that confusion occurs earlier in a tutoring
session, while boredom increases as the session progresses (D'Mello, Craig et al., 2008).
The analysis proceeded by dividing each 35-minute session into three 10-minute time intervals
(the last 5 minutes were ignored1). We then computed the distribution of emotions for each of the time
intervals. A 3 × 14 (time interval × affect) ANOVA indicated that there was no significant time
interval × affect interaction (p = .506).
Analysis 4. Affect and Problem Solving Outcomes
Students were required to solve 28 difficult problems in approximately 35 minutes. However, on
average they only solved 19 problems in the allotted time yielding a mean completion rate of 67.5%
(SD = 18.8). The mean precision score was .46 (SD = .15) and the mean recall was .308 (SD = .133)2.
The pertinent question of whether student affect was related to problem solving performance was
addressed by comparing the distribution of emotions for problems that students answered correctly to
problems that were answered incorrectly.
Fixed Judgment Points
After Feedback on Previous Problem. We were unable to investigate whether feedback on the
current problem (% ) was related to the outcome of its solution because feedback on the current
1
A warning indicating that time is running out was automatically issued during the last 5 minutes of the problem
solving phase. After receiving this warning, participants demonstrated heightened anxiety and a general state of
panic. The emotions experienced during the last 5 minutes were not included in these analyses as they are more
of an artifact of the warning message rather than the natural steps of problem solving.
#0123045678903369:5;<=8>636?
#0123045678903369:5;<=8>636?
2
Precision =
;Recall =
#0123045678<::672:6?
:0:<5#0123045678
problem is provided after its solution. However, we were able to investigate whether feedback on the
previous problem (%C ) could predict whether the current problem was answered correctly.
A 2 × 14 (response type [correct | incorrect] × affect) ANOVA revealed that student affect after
receiving feedback for problem %C was related to the outcome for problem % , F(13, 520) = 2.19,
MSe = .008, p < .01, partial η 2 = .052. Bonferroni posthoc tests revealed that the significant
differences were only for the product-related emotions of frustration and happiness. It appears that
increased levels of frustration after receiving feedback for the previous problem was linked to
diminished performance on the current problem, FrustrationCORRECT (M = .121, SD = .149) <
FrustrationINCORRECT (M = .173, SD = .172). On the other hand, being happy after receiving feedback
on the previous problem was linked to enhanced performance on the current problem,
HappinessCORRECT (M = .176, SD = .184) > HappinessINCORRECT (M = .111, SD = .103).
P
P
After Onset of Current Problem. An ANOVA indicated that student affect a few seconds after a
problem was presented was related to how well they solved the problem, F(13, 520) = 1.76, MSe =
.007, p < .05, partial η2 = .042. It appears that being curious after a problem statement was presented
was related to how well the problem was solved, CuriosityCORRECT (M = .243, SD = .238) >
CuriosityINCORRECT (M = .176, SD = .227).
During Solution of Current Problem. An ANOVA indicated that affective states in the midst of
solving a problem was related to problem solving outcomes, F(13, 520) = 2.37, MSe = .006, p < .01,
partial η2 = .056. Bonferroni posthoc tests revealed that enhanced curiosity while solving a problem
was linked to positive outcomes, CuriosityCORRECT (M = .216, SD = .237) > CuriosityINCORRECT (M =
.161, SD = .204). A reverse pattern emerged for frustration. It appears that increased frustration during
the problem solving process was negatively related to performance, FrustrationCORRECT (M = .05, SD =
.089) < FrustrationINCORRECT (M = .084, SD = .095). Therefore, being curious immediately after being
presented with the problem statement and maintaining this level of curiosity while solving the
problem, is associated with positive outcomes. In contrast being frustrated after receiving feedback on
the previous problem, and sustaining this negative emotion while solving the current problem, is
unfavorable for performance.
As could be expected we discovered that experiences of eureka were positively linked to positive
outcomes, EurekaCORRECT (M = .04, SD = .073) > EurekaINCORRECT (M = .014, SD = .043). Therefore,
although “a-ha” moments that resonate with the eureka experience are rare, they are valuable
indicators of successful outcomes.
Spontaneously Reported Affective States
An ANOVA confirmed that there was a significant relationship between spontaneously reported affect
and problem solving outcomes, F(13, 364) = 3.86, MSe = .017, p < .001, partial η2 = .121. Bonferroni
posthoc tests revealed that spontaneous reports of disgust during the problem solving process were
negatively linked to performance, DisgustCORRECT (M = .002, SD = .008) < DisgustINCORRECT (M = .077,
SD = .135). There were no differences for any of the other emotions. Therefore, although disgust
rarely occurs (3% of the emotions), students that are disgusted to a point that they feel the need to
voluntarily report their disgust rarely provide a correct answer.
Discussion on Relationships between Emotions on Problem Solving Outcomes
The links between students’ affective states at the various phases of the problem solving process in
conjunction with spontaneously reported emotions highlight important relationships between emotions
and problem solving outcomes. It appears that students anguish over past failure because frustration
after receiving negative feedback on the previous problem was negatively related to the quality of the
solution associated with the current problem. However, students also revel in past successes.
Happiness associated with a positive outcome on the previous problem was positively linked to
success on the current problem.
An example of the relationship between affect and problem solving outcomes is illustrated in
Figure 4. The figure depicts moving averages of performance (dotted lines) and relevant affective
states (solid lines) for a sample student.
Fig. 4. Affective states and problem solving performance for a sample student.
M. Avg = moving average
This student appears to oscillate between bouts of failure (problems 1-3 and 10-17) and success
(problems 4-9 and 17-22). As illustrated in Figure 4A, the initial drop in performance is mirrored by
an increase in frustration (problems 1-3), while frustration diminishes when problems 4-9 are correctly
answered. A new round of failure (problems 10-17) is accompanied by a small rise in frustration,
which stays constant as performance gradually improves (problems 17-22). It is intriguing to note that
happiness and frustration show an excitatory-inhibitory relationship as depicted in Figure 4B. Early
failure that increases frustration is accompanied by a drop in happiness. But happiness increases as
performance improves and frustration dissipates.
The results also indicated that being curious when a new problem is presented and maintaining
that level of curiosity during the problem solving process (i.e., after problem presentation and before
answer submission) was positively associated with performance. This alignment between curiosity and
problem solving performance is exemplified in Figure 4C. Failure is accompanied by a drop in
curiosity, but curiosity levels are rejuvenated when performance improves.
Finally, although confusion was not statistically linked to problem solving outcomes, this sample
student’s experiences of confusion were closely linked to performance (Figure 4D). For example,
initial poor performance is associated with a sharp increase in confusion while confusion dissipates as
performance improves.
GENERAL DISCUSSION
Our results support the hypothesis that emotions play a critical role in effortful problem solving
activities. Since emotion and motivation are inextricably bound to learning (Meyer & Turner, 2006;
Snow, Corno, & Jackson, 1996), the diverse and rich emotional tapestry observed in this study
provides important clues into students’ motivational levels. The students in this study were sampled
from a population of students that planned on taking the LSAT in the near future, ostensibly because
law school was a viable career choice. The alignment between the task (i.e., solving sample LSAT
problems) and goals (i.e., taking the actual LSAT for law school admission) may have positively
influenced their motivation levels. Students’ motivation levels could also have been enhanced with the
monetary incentive of two dollars for each question they answered correctly. Although monetary
incentives may not be common in real world learning situations, other extrinsic motivators are part
and parcel of most educational tasks. Whether it is parental approval, completion of a class,
acceptance into college, or obtaining a promotion, the learning environment cannot be divorced from
the realities of the real world, which is rife with sources of extrinsic motivation.
Our multi-faceted investigation on the occurrence of the emotions, their temporal dynamics,
contextual underpinnings of problem phase, feedback, response time, and problem difficulty, and on
relationships between affect and problem solving outcomes supports some important conclusions into
the affective dimension of problem solving. We proceed by taking stock of the major findings by
assimilating the various analysis threads, listing some of the limitations of the study and offering
potential solutions, and discussing applications of our findings for ITSs.
Assimilation of Major Findings
The results indicated that affective states differ in their probability of occurrence as regular, routine, or
sporadic emotions, their temporal behavior as persistent or random emotions, their manifestations as
product or process related emotions, and whether they facilitate or impede the problem solving
process. An examination of the more frequent emotions along these dimensions provides a broad
picture of the role of emotions during problem solving (see Table 6).
Table 6
Characterization of emotions of four primary dimensions
Persistence
(Persistent,
Ephemeral,
Random)
Manifestation
(Product,
Process)
Impact on Performance
(Beneficial, Harmful)
Affect
Incidence
(Routine,
Sporadic,
Exceptional)
Curiosity
Boredom
Routine
Routine
Persistent
Persistent
Process
-
Beneficial
-
Frustration
Happiness
Routine
Sporadic
Persistent
Random
Product
Product
Harmful
Beneficial
Confusion
Routine
Persistent
Both
-
Note. Blank cells indicate that there is a lack of empirical evidence to support a categorization.
Curiosity and Boredom. Curiosity was the most frequent emotion and was strongly associated with
problem solving performance. Curiosity can be characterized as a routine and persistent emotion that
is intimately related to the problem solving process and positively predicts performance. Berlyne
(1978) describes curiosity as a form of deliberate, exploratory behavior, which in motivated students is
a natural correlate to solving problems that are challenging, exciting, and require an active exploration
of the problem space to arrive at a solution. The fact that curiosity is related to interest (Berlyne, 1960,
1978; Deci, 1992; Izard & Ackerman, 2000; Tobias, 1994), is one possible explanation why curiosity
was positively linked to performance.
Boredom, on the other hand, is the antithesis of interest, engagement, and curiosity. Boredom
routinely occurs, is quite persistent, and is neither a product nor a process related emotion. Bored
students disengage to an extent where any external stimulation via feedback or a new problem is
ineffective in capturing their interest. This pattern of boredom is consistent with previous research that
tracked boredom during tutorial sessions with ITSs (Baker et al., 2010; D’Mello & Graesser, in pressb). Although boredom was not negatively associated with performance in the present study, we
suspect that this lack of a relationship might be attributed to the relatively short 35-minute problem
solving session. Boredom might have a more negative effect over longer time spans. For example,
there is considerable research that highlights the detrimental effects of boredom such as negligible
learning, lower self-efficacy, diminished interest in educational activities, and, most importantly,
increased attrition and dropout (Craig et al., 2004; Farrell, Peguero, Lindsey, & White, 1988; Larson
& Richards, 1991; Mann & Robinson, 2009; Pekrun et al., 2010; Perkins & Hill, 1985; Robinson,
1975). Taken together, these results emphasize the importance alleviating boredom while promoting
curiosity and interest so students might pursue more productive trajectories of thought.
Frustration and Happiness. Frustration and happiness were the two major product-related emotions.
Frustration accompanied negative feedback while happiness was related to positive feedback. Since
students provided correct responses to approximately 50% of the problems, one would expect the
incidence of these two affective states to be approximately equal. However, frustration occurred at
twice the rate as happiness, suggesting that the valence dimension alone does not provide an adequate
explanation on the occurrence of these two emotions. An examination of these emotions along a
temporal dimension provides some additional insights. It appears that frustrated students are more
likely to stay frustrated, while happy students routinely transition into other affective states. Simply
put, frustration is unrelenting while happiness is fleeting.
Confusion. Confusion appears to be a routine and persistent emotion with both product- and processrelated manifestations, a finding that substantiates existing research that highlights the importance of
this emotion to problem solving and deep thinking (Festinger, 1957; Graesser et al., 2005; Graesser &
Olde, 2003; Piaget, 1952; Rozin & Cohen, 2003; Silvia, 2009; VanLehn et al., 2003). The high levels
of confusion could be attributed to the problems being riddled with complications, salient contrasts,
and other obstacles – all factors that put students in a state of cognitive disequilibrium (Graesser et al.,
2005; Graesser & Olde, 2003). Confusion is often accompanied by effortful cognitive activities as
students try to resolve impasses, thus returning to a state of cognitive equilibrium. This can be
considered to be a form of productive confusion that forces students to stop and think. But confusion
also has a less productive form, namely hopeless confusion, where students are unable to achieve a
resolution and get stuck. Unresolved confusion transitions into frustration which over time triggers
boredom (D'Mello & Graesser, 2010a). We suspect that alternating between these two forms of
confusion might explain why this emotion was not explicitly linked to problem solving outcomes.
Limitations and Possible Resolutions
There are four primary limitations with this study. Perhaps the most significant limitation pertains to
the fact that the correlational design we adopted limits our ability to draw causal relationships between
the primary variables (e.g., emotions and problem solving outcomes). It is possible to experimentally
induce particular affective states so that causal links can be tested, however, most of the methods for
affect induction, such as exposure to affectively charged stimuli such as films and images (Coan &
Allen, 2007), are divorced from the context of the primary task (i.e., problem solving). For example,
the anxiety induced from exposing participants to images of spiders (Ohman & Soares, 1993) is
ostensibly quite different from anxiety associated with poor performance during problem solving.
Hence, the added experimenter control afforded by context-free affect induction techniques is
accompanied by a substantial drop in ecological validity. This is a compromise that is incompatible
with the present goal of monitoring the antecedents and consequents of naturally occurring emotions
that are inextricably bound to the problem solving process. Nevertheless, while the present paper
focuses on correlational links between a large set of variables, future work should systematically
induce affect in a context-sensitive fashion so that important causal links can be tested.
A second limitation pertains to exclusively relying on self-reports for affect measurement. Selfreports are limited by the participant’s ability and sensitivity to his or her emotions and the accuracy of
the reports depends on the honesty of the student. Self-reports also require the emotion to be
consciously accessible, so subtle unconscious affect experiences are likely to be missed. Hence, it is
important to confirm that the major findings of the study are replicated when alternate methods are
used to measure student emotions. Some alternate methods include live observations (Craig et al.,
2004; Rodrigo & Baker, in press), offline judgments by peers, trained judges, and teachers (D'Mello,
Taylor, Davidson, & Graesser, 2008; Graesser et al., 2006), and physiological and behavioral
instrumentation (Arroyo et al., 2009; Calvo & D’Mello, 2010). Perhaps the most defensible position is
to consider multiple measures in order to obtain more reliable and valid measurements.
The third and fourth limitation can be attributed to students being required to make forced-choice
affect judgments and that mixed emotions (e.g., confusion + anxiety) were not tracked. It is important
that these issues be addressed in replication studies where mixed emotions are monitored and students
have some flexibility in making their judgments. For example, students could be provided with the
option of selecting an "other" category, reporting combinations of emotions that they are
simultaneously experiencing, or using a dimensional measurement instrument like the Affect Grid
(Russell, Weiss, & Mendelsohn, 1989).
Applications of our Findings for ITSs
Tutoring in mathematics and science routinely involves periods of problem solving (Aleven &
Koedinger, 2002; Aleven et al., 2006; Anderson, 1990; Gertner & VanLehn, 2000; Lesgold et al.,
1992). These effortful problem solving activities often involve failure, and students experience a host
of negative emotions such as frustration, anxiety, and boredom. There is no sterile learning
environment that always promotes positive affective experiences because negative emotions inevitably
accompany the natural steps of making mistakes, being stuck, and resolving impasses. The challenge
for ITS developers is to leverage emerging basic research on affect and learning towards the
development of computer environments that promote learning at deeper levels of comprehension and
mastery in a manner that effectively coordinates cognition and emotions. The present study directly
contributes to this goal of developing affect-sensitive ITSs that detect and respond to students’
emotional states in addition to their cognitive states.
In particular, we have identified curiosity, boredom, confusion, and frustration as the major
affective states that students experienced during problem solving. Anxiety and happiness were
occasionally experienced, while the remaining six emotions were rare. Importantly, with the exception
of happiness, the basic emotions did not appear to be very relevant, at least within the 35-minute
problem solving sessions adopted in this study. Although this finding warrants replication with diverse
student populations, different problem solving contexts, and alternate methodologies to track
emotions, it is consistent with previous studies that have monitored emotions during tutoring sessions
with ITSs (D'Mello et al., 2006) and expert human tutors (Lehman et al., 2008). Taken together, these
results are suggestive of an important point of divergence between general affective-computing
research, which primarily focuses on the basic-emotions (Calvo & D’Mello, 2010; Zeng, Pantic,
Roisman, & Huang, 2009), and the specialized niche of affective learning environments (Calvo &
D'Mello, in preparation) where the learning-centered emotions are more prominent. It is these
learning-centered emotions that should be on the radar of affect-sensitive ITSs.
While the incidence data narrows the landscape of relevant emotions, the persistence data
provides some clues into the relative importance of responding to these emotions. It appears that in
addition to being very frequent, boredom is also one of the most persistent emotions. Once triggered,
boredom adopts a persistent temporal quality and students risk disengaging to a point that any further
instruction is essentially futile. There is also some evidence that tutorial interventions are not very
effective at alleviating boredom (D’Mello & Graesser, in press-b), so a proactive strategy of predicting
and preventing boredom might be more productive than a reactive strategy of detection and regulation.
On the other hand, curiosity, which was closely associated with productive problem solving leading to
successful performance, was comparatively less persistent. This suggests that it might be important for
ITSs to implement interventions that promote curiosity and engagement in order to ignite and nurture
sparks of interest that can be sustained over multiple sessions.
The results from our analysis of the contextual cues (e.g., problem phase, feedback) surrounding
emotional experiences can be used to scaffold the development of automatic affect detection systems.
Emotion detection is a major challenge for affect-sensitive ITSs because an ITS can never respond to
student emotions if it cannot detect those emotions. Although most systems rely on physiological and
bodily cues to detect emotions (Arroyo et al., 2009; Castellano, Kessous, & Caridakis, 2008; D'Mello
& Graesser, 2010b), the link between affective experience and expression is diffuse, fuzzy, murky,
and possibly indeterminate (Calvo & D’Mello, 2010; Russell et al., 2003). Since context plays a major
role in shaping emotional experiences, it is important to couple bottom-up diagnostic assessments of
affective reactions (i.e., face, speech, posture) with top-down contextually-driven predictive
assessments (Conati & Maclaren, 2009). For example, we know that some emotions are closely related
to the problem solving process, while others are tightly coupled to the feedback received (i.e., the
product). This information can be used to develop context-based affect prediction systems that guide
physiological and bodily-based affect detectors.
In summary, we provided a sketch of how some of the insights obtained from the present analysis
of affect during problem solving can be applied to advanced learning technologies that sense and
respond to student affect. What we have not addressed is explicit strategies to regulate negative
emotions and promote more productive states. Although this question is beyond the scope of the
present study, it is possible to speculate on some possible affect regulation strategies that can be
implemented in ITSs. For example, an ITS that senses negative emotions can respond by offering hints
to diffuse confusion, empathetic statements to alleviate frustration, motivational statements to avert
anxiety, and novel challenges to offset boredom (Burleson & Picard, 2007; D’Mello, Craig, Fike, &
Graesser, 2009; Forbes-Riley & Litman, 2010; Robison, McQuiggan, & Lester, 2009; Woolf et al.,
2010). Positive emotions such as curiosity, engagement, and interest could be fostered by encouraging
students to select their own problems since these emotions can be stimulated in environments that
foster students’ freedom of choice and when students perceive a degree of value in the learning
activity (Guthrie & Alvermann, 1999; Lepper & Woolverton, 2002; Pekrun, 2010; Pekrun et al.,
2010). It is our hope that these affect-sensitive interventions will fortify students with the necessary
scaffolds that encourage perseverance to conquer failure and its resulting negative emotions and
starting over with hope, determination, and even enthusiasm.
ACKNOWLEDGMENTS
This research was supported by grants awarded by the U. S. Office of Naval Research (N00014-05-10241), the National Science Foundation (ITR 0325428, HCC 0834847), and the Institute of Education
Sciences (Grant R305A080594). Any opinions, findings and conclusions, or recommendations
expressed in this paper are those of the authors and do not necessarily reflect the views of the funding
agencies.
REFERENCES
Afzal, S., & Robinson, P. (2009). Natural Affect Data - Collection & Annotation in a Learning Context. Paper
presented at the Proceedings of 2009 International Conference on Affective Computing & Intelligent
Interaction, Amsterdam.
Aleven, V., & Koedinger, K. (2002). An effective metacognitive strategy: Learning by doing and explaining
with a computer-based Cognitive Tutor. Cognitive Science, 26(2), 147-179.
Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of helpseeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101-128.
Anderson, J. (1990). Analysis of student performance with the LISP tutor. In J. Frederiksen, R. Glaser, A. M.
Lesgold & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 27-50).
Hillsdale, NJ: Erlbaum.
Anderson, J., Douglass, S., & Qin, Y. (2005). How should a theory of learning and cognition inform instruction?
In A. Healy (Ed.), Experimental cognitive psychology and it's applications (pp. 47-58). Washington, DC.:
American Psychological Association.
Arroyo, I., Woolf, B., Cooper, D., Burleson, W., Muldner, K., & Christopherson, R. (2009). Emotion sensors go
to school. In V. Dimitrova, R. Mizoguchi, B. Du Boulay & A. Graesser (Eds.), Proceedings of 14th
International Conference on Artificial Intelligence In Education (pp. 17-24). Amsterdam: IOS Press.
Aviezer, H., Hassin, R., Ryan, J., Grady, C., Susskind, J., Anderson, A., et al. (2008). Angry, disgusted, or
afraid? Studies on the malleability of emotion perception. Psychological Science, 19(7), 724-732.
Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction.
Journal of Educational Computing Research, 13(2), 111-127.
Baker, R., D'Mello, S., Rodrigo, M., & Graesser, A. (2010). Better to be frustrated than bored: The incidence
and persistence of affect during interactions with three different computer-based learning environments.
International Journal of Human-Computer Studies, 68 (4), 223-241.
Bangert-Drowns, R. L., Kulik, C. L. C., Kulik, J. A., & Morgan, M. (1991). The instructional-effect of feedback
in test-like events. Review of Educational Research, 61(2), 213-238.
Barrett, L. (2006). Are emotions natural kinds? Perspectives on Psychological Science 1, 28-58.
Barrett, L. F. (2009). Variety is the spice of life: A psychological construction approach to understanding
variability in emotion. Cognition & Emotion, 23(7), 1284-1306.
Berlyne, D. (1960). Conflict, arousal, and curiosity. New York: McGraw-Hill Inc.
Berlyne, D. (1978). Curiosity in learning. Motivation and Emotion, 2, 97-175.
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347-364.
Black, P., & William, D. (1998). Assessment and classroom learning. Assessment in Education: Principles,
Policy & Practice, 5(1), 7-74.
Bless, H., & Fielder, K. (1995). Affective states and the influence of activated general knowledge. Personality
and Social Psychology Bulletin, 21, 766-778.
Boekaerts, M., Pintrich, P. R., & Zeidner, M. (Eds.). (2000). Handbook of self-regulation. San Diego: Academic
Press.
Bransford, D., Brown, A., & Cocking, R. (Eds.). (2000). How people learn: Brain, mind, experience, and school
committee on developments in the science of learning Washington, DC: National Academy Press.
Brosch, T., Pourtois, G., & Sander, D. (2010). The perception and categorisation of emotional stimuli: A review.
Cognition & Emotion, 24(3), 377-400.
Brown, A. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In
F. E. Weinert & R. H. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65-116).
Hillsdale: Erlbaum.
Burleson, W., & Picard, R. (2007). Evidence for gender specific approaches to the development of emotionally
intelligent learning companions. IEEE Intelligent Systems, 22(4), 62-69.
Calvo, R., & D'Mello, S. (Eds.). (in preparation). New perspectives on affect and learning technologies. New
York: Springer.
Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and
their applications. IEEE Transactions on Affective Computing, 1(1), 18-37.
Castellano, G., Kessous, L., & Caridakis, G. (2008). Emotion recognition through multiple modalities: face,
body gesture, speech. In C. Peter & R. Beale (Eds.), Affect and Emotion in Human-Computer Interaction
(LNCS, vol. 4868) (pp. 92-103). Heidelberg: Springer.
Chi, M., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. (2001). Learning from human tutoring. Cognitive
Science, 25(4), 471-533.
Clore, G. L., & Huntsinger, J. R. (2007). How emotions inform judgment and regulate thought. Trends in
Cognitive Sciences, 11(9), 393-399.
Coan, J., & Allen, J. (Eds.). (2007). Handbook of emotion elicitation and assessment. New York: Oxford
University Press.
Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect.
User Modeling and User-Adapted Interaction, 19(3), 267-303.
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role
of affect in learning. Journal of Educational Media, 29, 241-250.
Cromley, J., Azevedo, R., & Olson, E. (2005). Self-regulation of learning with multiple representations in
hypermedia. In C.-K. Looi, G. McCalla, B. Bredeweg & J. Breuker (Eds.), Artificial intelligence in
education: Supporting learning through intelligent and socially informed technology (pp. 184-191).
Amsterdam: IOS Press.
Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper and Row.
D'Mello, S., Craig, S., Sullins, J., & Graesser, A. (2006). Predicting affective states expressed through an emotealoud procedure from AutoTutor's mixed-initiative dialogue. International Journal of Artificial Intelligence
in Education, 16(1), 3-28.
D'Mello, S., Craig, S., Witherspoon, A., McDaniel, B., & Graesser, A. (2008). Automatic detection of learner's
affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1-2), 45-80.
D'Mello, S., & Graesser, A. (2010a). Modeling cognitive-affective dynamics with Hidden Markov Models. In R.
Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Cognitive Science Society (pp. 27212726). Austin, TX: Cognitive Science Society.
D'Mello, S., & Graesser, A. (2010b). Multimodal semi-automated affect detection from conversational cues,
gross body language, and facial features. User Modeling and User-adapted Interaction, 20(2), 147-187.
D'Mello, S., Olney, A., & Graesser, A. (in press). Affect, meta-affect, and affect regulation during complex
learning. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning
Technologies. New York: Springer.
D'Mello, S., Olney, A., & Person, N. (in press). Mining collaborative patterns in tutorial dialogues. Journal of
Educational Data Mining, 2(1), 1-37.
D'Mello, S., Taylor, R., Davidson, K., & Graesser, A. (2008). Self versus teacher judgments of learner emotions
during a tutoring session with AutoTutor. In B. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.),
Proceedings of the 9th international conference on Intelligent Tutoring Systems. Berlin, Heidelberg:
Springer-Verlag.
D’Mello, S., Craig, S., Fike, K., & Graesser, A. (2009). Responding to learners’ cognitive-affective states with
supportive and shakeup dialogues. In J. Jacko (Ed.), Human-Computer Interaction. Ambient, Ubiquitous
and Intelligent Interaction (pp. 595-604). Berlin/Heidelberg: Springer.
D’Mello, S., & Graesser, A. (in press-a). Emotions during Learning with AutoTutor. In P. Durlach & A. Lesgold
(Eds.), Adaptive Technologies for Training and Education. Cambridge, U.K: Cambridge University Press.
D’Mello, S., & Graesser, A. (in press-b). The half-life of cognitive-affective states during complex learning.
Cognition & Emotion.
D’Mello, S., Taylor, R. S., & Graesser, A. (2007). Monitoring affective trajectories during complex learning. In
D. McNamara & J. Trafton (Eds.), Proceedings of the 29th Annual Cognitive Science Society (pp. 203208). Austin, TX: Cognitive Science Society.
Deci, E. L. (1992). On the nature and functions of motivation theories. Psychological Science, 3(3), 167-171.
Dweck, C. (2002). Messages that motivate: How praise molds students' beliefs, motivation, and performance (in
surprising ways). In J. Aronson (Ed.), Improving academic achievement: Impact of psychological factors
on education (pp. 61-87). Orlando, FL: Academic Press.
Efklides, A. (2006). Metacognition and affect: What can metacognitive experiences tell us about the learning
process? Educational Research Review, 1(1), 3-14.
Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3-4), 169-200.
Entwistle, N. (1988). Styles of learning and teaching. London: David Fulton.
Farrell, E., Peguero, G., Lindsey, R., & White, R. (1988). Giving voice to high-school students - Pressure and
boredom, ya know what Im sayin. American Educational Research Journal, 25(4), 489-502.
Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press.
Fielder, K. (2001). Affective states trigger processes of assimilation and accommodation. In K. Martin & G.
Clore (Eds.), Theories of Mood and Cognition: A User’s Guidebook (pp. 85-98). Mahwah, NJ: Erlbaum.
Fontaine, J., Scherer, K., Roesch, E., & Ellsworth, P. (2007). The world of emotions is not two-dimensional.
Psychological Science, 18(12).
Forbes-Riley, K., & Litman, D. (2010). Designing and evaluating a wizarded uncertainty-adaptive spoken
dialogue tutoring system. Computer Speech and Language, 25(1), 105-126.
Forbes-Riley, K., Rotaru, M., & Litman, D. (2008). The relative impact of student affect on performance models
in a spoken dialogue tutoring system. User Modeling and User-Adapted Interaction, 18(1-2), 11-43.
Frijda, N. H. (2009). Emotions, individual differences and time course: Reflections. Cognition & Emotion,
23(7), 1444-1461.
Gertner, A., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier,
C. Frasson & K. VanLehn (Eds.), Proceedings of the International Conference on Intelligent Tutoring
Systems (pp. 133-142). Berlin / Heidelberg: Springer.
Graesser, A., Chipman, P., King, B., McDaniel, B., & D'Mello, S. (2007). Emotions and learning with
AutoTutor. In R. Luckin, K. Koedinger & J. Greer (Eds.), 13th International Conference on Artificial
Intelligence in Education (pp. 569-571). Amsterdam: IOS Press.
Graesser, A., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge
construction. Discourse Processes, 45(4-5), 298-322.
Graesser, A., Lu, S., Olde, B., Cooper-Pye, E., & Whitten, S. (2005). Question asking and eye tracking during
cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down.
Memory and Cognition, 33, 1235-1247.
Graesser, A., McDaniel, B., Chipman, P., Witherspoon, A., D'Mello, S., & Gholson, B. (2006). Detection of
emotions during learning with AutoTutor. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual
Conference of the Cognitive Science Society (pp. 285-290). Austin, TX: Cognitive Science Society.
Graesser, A., & Olde, B. (2003). How does one know whether a person understands a device? The quality of the
questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524536.
Graesser, A. C., D'Mello, S. K., & Person, N. (2009). Meta-knowledge in tutoring. In D. Hacker, J. Dunlosky &
A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 361-382). Mahwah, NJ.:
Erlbaum.
Guthrie, J. T., & Alvermann, D. E. (1999). Engaged reading: Processes, practices, and policy implications. New
York: Teachers College Press New York.
Heider, F. (1958). The psychology of interpersonal relations. New York: John Wiley & Sons.
Hertel, G., Neuhof, J., Theuer, T., & Kerr, N. (2000). Mood effect on cooperation in small groups: Does positive
mood simply lead to more cooperation? Cognition and Emotion, 14, 441-472.
Isen, A. (2008). Some ways in which positive affect influences decision making and problem solving. In M.
Lewis, J. Haviland-Jones & L. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 548-573). New York,
NY: Guilford.
Isen, A., Daubman, K., & Nowicki, G. (1987). Positive affect facilitates creative problem solving. Journal of
Personality and Social Psychology, 52, 1122-1131.
Isen, A. M. (2003). Positive affect, systematic cognitive processing, and behavior: Toward integration of affect,
cognition, and motivation. In F. Dansereau & F. J. Yammarino (Eds.), Multi-level issues in organizational
behavior and strategy (pp. 55-62). Oxford: JAI/Elsevier Science.
Izard, C., & Ackerman, B. (2000). Motivational, organizational, and regulatory functions of discrete emotions.
In M. Lewis & J. Haviland-Jones (Eds.), Handbook of emotions (2nd ed., pp. 253-264). New York:
Guilford Press.
Izard, C. E. (2007). Basic emotions, natural kinds, emotion schemas, and a new paradigm. Perspectives on
Psychological Science, 2(3), 260-280.
Jonassen, D., Peck, K., & Wilson, B. (1999). Learning with technology: A constructivist perspective. Upper
Saddle River, NJ: Prentice Hall.
Klein, J., Moon, Y., & Picard, R. (2002). This computer responds to user frustration: Theory, design, and results.
Interacting with Computers, 14(2), 119-140.
Kort, B., Reilly, R., & Picard, R. (2001). An affective model of the interplay between emotions and learning.
Paper presented at the Proceedings of the International Conference of Advanced Learning Technologies,
Madison, WI.
Larson, R. W., & Richards, M. H. (1991). Boredom in the middle school years - Blaming schools versus
blaming students. American Journal of Education, 99(4), 418-443.
Lazarus, R. (2000). The cognition-emotion debate: A bit of history. In M. Lewis & J. Haviland-Jones (Eds.),
Handbook of Emotions (2nd ed., pp. 1-20). New York: Guilford Press.
Lehman, B., Matthews, M., D'Mello, S., & Person, N. (2008). What are you feeling? Investigating student
affective states during expert human tutoring sessions. In B. Woolf, E. Aimeur, R. Nkambou & S. Lajoie
(Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 50-59). Berlin,
Heidelberg: Springer.
Lepper, M., & Woolverton, M. (2002). The wisdom of practice: Lessons learned from the study of highly
effective tutors. In J. Aronson (Ed.), Improving academic achievement: Impact of psychological factors on
education (pp. 135-158). Orlando, FL: Academic Press.
Lesgold, A., Lajoie, S., Bunzo, M., & Eggan, G. (1992). SHERLOCK: A coached practice environment for an
electronics troubleshooting job. In J. H. Larkin & R. W. Chabay (Eds.), Computer-assisted instruction and
intelligent tutoring systems (pp. 201-238). Hillsdale, NJ: Erlbaum.
LSAT. (2008). Law School Admissions Test (Publication. Retrieved August 14, 2008: http://www.lsat.org
Mandler, G. (1976). Mind and emotion. New York: Wiley.
Mandler, G. (1999). Emotion. In B. M. Bly & D. E. Rumelhart (Eds.), Cognitive science. Handbook of
perception and cognition (2nd ed., pp. 367-382). San Diego, CA: Academic Press.
Mann, S., & Robinson, A. (2009). Boredom in the lecture theatre: an investigation into the contributors,
moderators and outcomes of boredom amongst university students. British Educational Research Journal,
35(2), 243-258.
McLeod, D. B. (1988). Affective issues in mathematical problem solving: Some theoretical considerations.
Journal for Research in Mathematics Education, 19(2), 134-141.
McQuiggan, S., Mott, B., & Lester, J. (2008). Modeling self-efficacy in intelligent tutoring systems: An
inductive approach. User Modeling and User-Adapted Interaction, 18(1-2), 81-123.
Meyer, D., & Turner, J. (2006). Re-conceptualizing emotion and motivation to learn in classroom contexts.
Educational Psychology Review, 18(4), 377-390.
Moors, A. (2009). Theories of emotion causation: A review. Cognition & Emotion, 23(4), 625-662.
Ohman, A., & Soares, J. (1993). On the automatic nature of phobic fear - Conditioned electrodermal responses
to masked fear-relevant stimuli. Journal of Abnormal Psychology, 102(1), 121-132.
Pekrun, R. (2010). Academic emotions. In T. Urdan (Ed.), APA educational psychology handbook (Vol. 2).
Washington, DC: American Psychological Association.
Pekrun, R., Goetz, T., Daniels, L., Stupnisky, R. H., & Raymond, P. (2010). Boredom in achievement settings:
Exploring control–value antecedents and performance outcomes of a neglected emotion. Journal of
Educational Psychology, 102(3), 531-549.
Perkins, R. E., & Hill, A. B. (1985). Cognitive and affective aspects of boredom. British Journal of Psychology,
76(MAY), 221-234.
Piaget, J. (1952). The origins of intelligence. New York: International University Press.
Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experience in higher education.
Buckingham: Open University Press.
Robinson, W. P. (1975). Boredom at school. British Journal of Educational Psychology, 45, 141-152.
Robison, J., McQuiggan, S., & Lester, J. (2009). Evaluating the consequences of affective feedback in intelligent
tutoring systems. Paper presented at the International Conference on Affective Computing & Intelligent
Interaction,, Amsterdam.
Rodrigo, M., & Baker, R. (in press). Comparing the incidence and persistence of learners’ affect during
interactions with different educational software packages. In R. Calvo & S. D'Mello (Eds.), New
Perspective on Affect and Learning Technologies. New York: Springer.
Rosenberg, E., & Ekman, P. (1994). Coherence between expressive and experiential systems in emotion.
Cognition & Emotion, 8(3), 201-229.
Rozin, P., & Cohen, A. (2003). High frequency of facial expressions corresponding to confusion, concentration,
and worry in an analysis of maturally occurring facial expressions of Americans. Emotion, 3, 68-75.
Russell, J. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145172.
Russell, J. A., Bachorowski, J. A., & Fernandez-Dols, J. M. (2003). Facial and vocal expressions of emotion.
Annual Review of Psychology, 54, 329-349.
Russell, J. A., Weiss, A., & Mendelsohn, G. A. (1989). Affect Grid - A single-item scale of pleasure and arousal.
Journal of Personality and Social Psychology, 57(3), 493-502.
Scherer, K. R. (2009). The dynamic architecture of emotion: Evidence for the component process model.
Cognition & Emotion, 23(7), 1307-1351.
Schwarz, N. (1990). Feelings as information: Informational and motivational functions of affective states. In E.
Higgins & R. Sorrentino (Eds.), Handbook of motivation and cognition (pp. 527-561). New York: Guilford
Press.
Schwarz, N. (2000). Emotion, cognition, and decision making. Cognition and Emotion, 14, 433-440.
Schwarz, N. (in press). Feelings-as-Information Theory. In P. Van Lange, A. Kruglanski & T. Higgins (Eds.),
Handbook of theories of social psychology: Sage.
Schwarz, N., & Skurnik, I. (2003). Feeling and thinking: Implications for problem solving. In J. Davidson & R.
Sternberg (Eds.), The Psychology of Problem Solving (pp. 263-290). New York: Cambridge University
Press.
Shute, V. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189.
Silvia, P. J. (2009). Looking past pleasure: Anger, confusion, disgust, pride, surprise, and other unusual aesthetic
emotions. Psychology of Aesthetics Creativity and the Arts, 3(1), 48-51.
Snow, R. E., Corno, L., & Jackson, D. (1996). Individual differences in affective and conative functions. In D.
Berliner & R. Caldee (Eds.), Handbook of educational psychology. New York: Macmillan.
Spering, M., Wagener, D., & Funke, J. (2005). The role of emotions in complex problem-solving. Cognition and
Emotion, 19(8), 1252-1261.
Stein, N., Hernandez, M., & Trabasso, T. (2008). Advances in modeling emotions and thought: The importance
of developmental, online, and multilevel analysis. In M. Lewis, J. M. Haviland-Jones & L. F. Barrett
(Eds.), Handbook of emotions (3rd ed., pp. 574-586). New York: Guilford Press.
Stein, N., & Levine, L. (1991). Making sense out of emotion. In A. O. W. Kessen, & F, Kraik (Eds.) (Ed.),
Memories, thoughts, and emotions: Essays in honor of George Mandler (pp. 295-322). Hillsdale, NJ:
Erlbaum.
Stemmler, G., Heldmann, M., Pauls, C., & Scherer, T. (2001). Constraints for emotion specificity in fear and
anger: The context counts. Psychophysiology, 38(2), 275-291.
Tobias, S. (1994). Interest, prior knowledge, and learning. Review of Educational Research, 64, 37-54.
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. (2003). Why do only some events cause
learning during human tutoring? Cognition and Instruction, 21(3), 209-249.
Weiner, B. (1986). An attributional theory of motivation and emotion. New York: Springer-Verlag.
Winnie, P. H. (2001). Self-regulated learning viewed from models of information processing. In B. Zimmerman
& D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd
ed., pp. 153-189). Mahwah: Lawrence Erlbaum Associates.
Woolf, B., Arroyo, I., Muldner, K., Burleson, W., Cooper, D., Dolan, R., et al. (2010). The effect of motivational
learning companions on low achieving students and students with disabilities In J. Kay & V. Aleven (Eds.),
Proceedings of 10th International Conference on Intelligent Tutoring Systems (pp. 327-337). Berlin /
Heidelberg: Springer.
Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual,
and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 3958.
APPENDIX: SAMPLE ANALYTICAL REASONING PROBLEM
[SCENARIO]
In a single day, exactly seven people—Anna, Brett, Claudia, Dean, Erica, Frank, & Georgia—come in to
interview for a job. Each interview is individual and lasts for one hour. There are two available positions,
secretary and messenger. Each person can interview for either position or both. The following conditions apply:
Anna and Dean do not interview for the same position.
Frank interviews some time before Claudia.
No two consecutive interviews are for both positions.
Three people interview for the secretary position only.
Georgia interviews fifth.
[SUB QUESTION]
Which of the following could be the order in which the interviews take place?
a) Claudia, Dean, Frank, Brett, Georgia, & Anna
b) Anna, Dean, Frank, Brett, Georgia, & Claudia
c) Brett, Anna, Dean, Frank, Georgia, & Claudia
d) Anna, Frank, Dean, Georgia, Brett, & Claudia
e) Dean, Brett, Frank, Claudia, Anna, & Georgia