The Effect of Cognitive Load on Mouse Behavior
Mind Over Mouse: The Effect of Cognitive
Load on Mouse Movement Behavior
Completed Research Paper
G. Mark Grimes
University of Houston
[email protected]
Joseph S. Valacich
University of Arizona
[email protected]
Abstract
Identifying when users are experiencing heightened cognitive load has a number of important applications for businesses, online education, and personal computer use. In this paper, we present a method for detecting heightened cognitive load using a technique known
as mouse dynamics (MD). We provide an overview of cognitive science literature linking
changes in cognitive states to interruptions of fine motor control to provide a theoretical
base for our work. We then describe a laboratory experiment in which low, medium, and
high cognitive load were elicited while MD features were captured. We found that participants exhibited longer task duration, longer mouse movements, more direction changes,
and slower speed when under high cognitive load.
Keywords: Human-Computer Interaction, Mouse Dynamics, Cognitive Load
Introduction
Reducing the mental effort required to use an information system is a laudable goal that is desirable for
many reasons. Prior research suggests that IT adoption is directly influenced by how difficult a system is
to use (Davis 1989), that educational outcomes are impacted by comprehension difficulty (Merriënboer and
Sweller 2005; Paas Renkl, A., Sweller, J. 2003; Sweller 1994; Sweller 1988), and that overall task performance is degraded when mental workload is heightened (Hart & Staveland, L. 1988; Paas and Merriënboer
1993). Scenarios such as these may be described as having heightened cognitive load – the mental effort
and working memory required to complete a task. While it is likely not possible, nor desirable, to eliminate cognitive load altogether, being able to recognize when users are experiencing heightened cognitive
load has a wide range of practical applications for business, education, and personal information systems
usage. For example, such a system would be useful for determining when service interventions are necessary in e-commerce, identifying when students are experiencing difficulty understanding concepts in online
education, or identifying when users are finding information systems difficult to use.
In order to reduce the negative effects of cognitive load, we must first be able to identify when individuals
are experiencing heightened cognitive load. Several methods may be used to assess cognitive load including
indirect subjective measures (i.e., self-reported mental effort), direct subjective measures (i.e., self-reported
stress levels or difficulty), indirect objective measures (i.e., learning outcomes, physiological measures), and
direct objective measures (i.e., eye tracking, brain activity/fMRI) (Brünken et al. 2003). While each of these
approaches provide some insight into the level of cognitive load an individual is experiencing, they each also
have shortcomings that researchers and practitioners alike must contend with. Indirect methods such as
surveys suffer from response bias (Furnham 1986) and interruption of one’s work to ask survey questions
may lead to confusion (Speier et al. 1999), reduction of performance (Hudson et al. 2002; Kirmeyer 1988),
and dissatisfaction as users may experience “response burden” (Asiu et al. 1998; Porter et al. 2004). Direct
methods such as observing learning outcomes may be influenced by the measurement method or by traits
of the individual (Brünken et al. 2003; Mayer 2001) and physiological measures such as eye tracking (Paas
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
1
The Effect of Cognitive Load on Mouse Behavior
et al. 2003), heart rate (Paas and Merriënboer 1993), pupil dilation (Beatty 1982) and fMRI (Smith and
Jonides 1997) may interfere with the natural use of the system (Scheirer et al. 2002; Zimmermann et al.
2003), thus inhibiting the user from engaging in their primary task. In this paper, we suggest that it is
possible to unobtrusively detect when users are experiencing heightened cognitive load by analyzing their
mouse movement behavior using a technique known as mouse dynamics (MD).
The study of MD is a relatively new area of research, having received most of its attention over the last ten
years. Much like the voice, handwriting, typing, and gait, MD is considered a behavioral or “soft” biometric.
As such, the primary focus of MD research to date has been identification and authentication (I&A). Recent research, however, suggests that mouse movements and other fine motor control may be influenced by
changes in cognitive states. Therefore, we aim to answer the following research question: Can heightened
cognitive load be identified by analyzing mouse movement behavior?
In the balance of this manuscript we present a discussion of prior work that in cognitive and neurological
science that helps to inform how changes in cognitive load may impact fine motor control, specifically the
interaction of cognitions and mouse movement behavior. We then describe and present the results of an experiment designed to elicit low, medium, and high cognitive load and conclude with a discussion of possible
applications and direction for future research.
Prior Work
It is widely accepted that cognitive processing consists of a relatively limited working memory and a considerably larger long-term memory (Paas et al. 2003). When engaging in an activity, it is necessary that
some level of working memory be devoted to the task at hand - this is referred to as the level of cognitive
load that is being exerted (Paas and Van Merriënboer 1994). Cognitive load may be broadly categorized into
three types: intrinsic – load that is related to a specific topic, extraneous – load that is related to the way in
which tasks are presented, and germane – load related to the work needed to create a permanent store of
knowledge (Paas et al. 2003).
Cognitive Load and Fine Motor Control
Prior literature provides a great deal of support for the linkage between cognitive processes and fine motor control. To fully appreciate this relationship, it is beneficial to take a broader look at how information
travels through a generic communication system. One helpful way to understand this is information theory
(Shannon 1948). Information theory describes communication systems as consisting of five elements: information source, transmitter, channel, receiver, and destination (Figure 1), having bandwidth (W), signal
)
(S), noise (N), and a finite channel capacity (C), defined as C = W Log2 (S+N
. As information is transferred
N
between the transmitter and receiver, there is some probability of the information being disrupted by a
source of noise. Noise that is introduced into the channel consumes some of the available bandwidth for the
message, thus reducing the overall information capacity of the channel to below the theoretical maximum.
This creates what Shannon refers to as “the fundamental problem of communication [which is] reproducing
either exactly or approximately a message selected at another point” (Shannon 1948, p. 379).
Figure 1. A General Communication System
While Shannon’s work was originally designed to explain physical communication systems, information
theory has been used to successfully describe many human perceptual, cognitive and motor processes. For
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
2
The Effect of Cognitive Load on Mouse Behavior
example, one might consider the human body a communication network in which the brain is the source,
transmitting messages through the motor cortex, using the nervous system as the channel to send the message to the hand, which receives the message with instructions on how to interact with the mouse – the
message’s destination. Prior work has shown cognitive changes such as heightened arousal and negative
emotions to lead to neuromotor ”noise” including motor evoked potentials (Coelho et al. 2010; Coombes et
al. 2008), spinal cord activity (Smith and Kornelsen 2011), and motor track excitability (Tanaka et al. 2012;
Van Loon et al. 2010), giving insight into cognitive processes with great sensitivity (Freeman et al. 2011).
Changes in motor control resulting from interference as described by information theory has been a well
researched topic for over fifty years. One of the foundational theories in this work is Fitts’s law (Fitts 1954).
Fitts’s law builds on Shannon’s work by developing a formula to describe the performance (or difficulty) of
executing rapid and aimed motor movements. Fitts suggests that motor movements consist of three continuous variables: force, direction and amplitude, each of which consume some bandwidth in the communication channel (Fitts 1954; Fitts and Peterson 1964). Due to the finite channel capacity, there is an inherent
trade-off between the distance of movement (amplitude), the speed of the movement (duration), and the accuracy of the movement (variability of fine motor control). As noise, neuromotor or other, is introduced into
the system, there will be additional variability of fine motor control, which will be reflected in the amplitude
and duration of the movement – that is, the duration and/or distance of the movement will increase. Fitts’s
law has been tested in the context of several fine motor control skills that are relevant to mouse behavior
such as wrist rotation, finger movements, pinching and other physical actions (Fitts 1954; Fitts and Peterson
1964; Meyer et al. 1988; Meyer et al. 1990).
The work of Shannon and Fitts is operationalized into mouse movements by the stochastic optimized submovement (SOS) model (Meyer et al. 1988; Meyer et al. 1990). The SOS model explains that fine motor
movements are composed of multiple applications of force and direction, and that mouse movements may
be described as having two parts - an initial high-velocity phase in which fast, but generally imprecise, movements are made toward a target, and a deceleration phase in which speed decreases and accuracy increases
(Graham and MacKenzie 1996) - an action that might be described as “corrective” in nature. This decrease
in speed is necessary to increase precision as the target is approached since, as Fitts and Shannon suggest,
there is a limited information capacity for motor control, thus requiring a trade off in speed and precision.
Initially the application of force is directed at the center of the intended target and is referred to as the primary submovement, which is followed by zero (if the primary movement was on target) or more corrective
submovements until the target is reached (Figure 2). The secondary submovements are automatic and subconscious movements that provide adjustments to force and direction designed to help reach the target. The
SOS model suggests the mind automatically tries to minimize average total movement time by optimizing
both the number of submovements and velocity, however, as neuromotor noise is introduced there is less
available capacity for the intended corrective movements, thus leading to slower and/or less precise movement (Van Beers et al. 2004) and a greater number of secondary submovements, ultimately resulting in a
path that is slower, longer, and consists of a greater number of direction changes (Meyer et al. 1988; Meyer
et al. 1990).
Finally, two theories are directly related to the motor control processes necessary for choosing between
multiple options - the Hick-Hyman law (also known as Hick’s law; Hick 1952) and the response activation
model (Welsh and Elliott 2004). Hick’s law suggests that as individuals choose between multiple options,
the cognitive effort that must be exerted to evaluate the available choices may result in disruptions to fine
motor control. As more choices are presented there is more information to process, and some amount of
the available bandwidth for the channel will be consumed, thus leading to slower response times. This may
be the result of a search process to find the correct option or it may be the result of uncertainty in selecting
an appropriate response. Similarly, the response activation model (Welsh and Elliott 2004), suggests that
motor movements are an aggregation of all potential movements, made up of all potentially actionable cognitions. Therefore, as new cognitions enter the mind – for example, debating between two or more potential
answers to a question – the potential motor movement to make the alternative choices will influence the
movements used to make the choice that is ultimately selected. When competing cognitions are introduced,
motor movements will be less precise and response times will slow as the necessary cognitive resources are
being otherwise consumed.
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
3
The Effect of Cognitive Load on Mouse Behavior
Figure 2. Mouse Movements as Described by the SOS Model
Mouse Dynamics
Researchers interested in MD have used theories such as those described by Shannon, Fitts, and Meyer to
better understand mouse behavior in human-computer interactions. To date, much of this research has focused on using MD as a soft biometric in security contexts such as I&A, as many companies, governments,
and other funding agencies have a great deal of interest in increasing information systems security (Pantic et
al. 2006). Like other I&A approaches, MD authentication schemes typically rely on having a known “signature” for the user against which challenge signatures are compared. There are a few basic attributes that may
be used to make up an MD signature – x/y coordinates, button states, and timestamp – that are captured either every few milliseconds, or every time there is a state change (i.e., movement or clicks). These attributes
are used to derive MD features which fall into five broad categories: speed, direction, action, distance and
time (Ahmed and Traore 2007). From these basic features, higher level features such as acceleration, angular velocity (Ahmed and Traore 2007; Gamboa and Fred 2004; Hocquet et al. 2004), scroll wheel activity,
clicks/double clicks (Pusara and Brodley 2004), drag-and-drop operations, point-and-click operations, and
silence (Ahmed and Traore 2007; Jorgensen and Yu 2011) can be generated. These features are often aggregated into distinct time periods for analysis. Once features are generated either statistical or machine
learning techniques are used to classify users based on similarities and differences in the MD signatures.
One shortcoming of using behavioral biometrics such as MD for I&A is that, unlike physical biometrics,
behavioral biometrics tend to vary based on cognitive or physical states. For example, an individual’s handwriting may become sloppy when they get tired, or their vocalic characteristics may change as they experience stress or other states of emotional discourse. Similarly, as individuals experience changes in cognitive
states, computer interaction behavior may change (Bergadano et al. 2003; Grimes et al. 2013; Maehr 2008;
Vizer 2009). While this is a problem for I&A, it provides many interesting opportunities for using behavioral
biometrics as physiological indicators of cognitive changes.
Emerging work in MD has moved beyond I&A applications and has begun investigating cognitive states that
may influence mouse movement behavior. For example, prior work has found that when users were exposed to stimuli that elicited high arousal or negative affect they exhibited more direction changes, longer
mouse movements, and in some cases slower speed (Grimes et al. 2013). Mouse movements have also been
analyzed in the context of assessing an individual’s pleasure when reading text and answering questions in
the context of improving educational systems, or “intelligent tutoring systems” (Lali et al. 2014). In this
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
4
The Effect of Cognitive Load on Mouse Behavior
study, participants read and answered questions about three English reading comprehension passages. After answering the questions, participants were asked how much they liked the questions, how difficult they
found the reading, and how confident they were in their answers. Mouse data were collected every 100 ms
during interactions with the system. The researchers created 104 mouse movement features – 26 related to
position, speed and acceleration and 78 related to pauses and other movement characteristics. Using machine learning techniques such as K-nearest neighbor, multilayer perceptron and support vector machines,
they created a four level classification model based on difficulty and confidence. Although their findings
were mixed, with the two higher desirability (low difficulty/high confidence) and lower desirability (high
difficulty/low confidence) levels often being confused, the underlying work is promising.
Hibbeln et al. (2014) used similar techniques to detect deception in a mock insurance fraud scenario. In
this study, participants were endowed with 2,000 “coins” and were given a scenario in which 400 - 2,000
“coins” worth of damage had been done to their car. Mouse movements were elicited on two screens in
which participants had the opportunity to overstate the damage to their car, thus claiming more money for
themselves. The authors suggest that fraud is a subset of deception and therefore build on two commonly
agreed upon components of deception: that deception leads to cognitive or moral conflict, i.e., individuals
may double check, hesitate, reconsider, or question their actions when behaving deceptively, and that deception requires more cognitive effort than truth telling, as individuals must engage in strategic behaviors
to mask their deception, keep their story straight, etc. Mouse movements were divided into segments with
start and end points determined by identifying points at which mouse movement paused for over 200 ms
or where the trajectory of the mouse changed by over 45 degrees. The authors found that fraudulent reporting resulted in increased normalized distance, decrease of speed, increase in response time, and a greater
number of clicks.
Based on our understanding of MD and the neurological processes associated with cognitions and motor
control, we suggest that distance, speed, duration, and accuracy of mouse movements should all be impacted
by changes in cognitive load. Thus, we propose the following hypothesis:
Hypothesis 1: When experiencing high cognitive load, individuals will exhibit changes in MD
including a) longer total duration b) longer Euclidean distance, c) slower movements, d) more
direction changes and e) more clicks than when under low cognitive load.
In the following section, we describe a laboratory experiment that was used to test this hypothesis.
Methodology
Sixty-eight students (32 male) from a junior level MIS class at a large university in the Southwest United
States voluntarily participated in this study in exchange for course credit. The average age of participants
was 20.7 years (min=20, max=25, SD=1.12) and the ethnic makeup was diverse, with 60% White/Caucasian,
18% Asian, 13% Hispanic, 4% Black/African American, and 4% Other. System malfunctions invalidated the
data from 15 participants, leaving a sample size of 53 for the analysis.
Upon arriving at the laboratory, participants were seated at computer terminals and informed of their participant rights. The computers were running a full screen Chrome web browser at 1920x1080 resolution and
were equipped with privacy screens and noise isolating headphones to minimize external distractions. Prior
to completing the study, participants completed an unrelated task that took approximately 30 minutes, then
proceeded to the experiment described here.
A purpose built web application was used to elicit mouse movements from participants in three similar but
increasingly difficult tasks. JavaScript code using the jquery library (http://www.jquery.com) was embedded in the web site to monitor the entire area of a web page for mousemove(), mousedown(), and mouseup()
events. When these events were detected, a JavaScript function was called to record the x and y coordinates
of the mouse cursor, the state of the mouse button, and a millisecond (ms) POSIX timestamp of when the
event occurred. The raw movement data were stored in a MSSQL database and were subsequently transformed into MD features including start and stop coordinates, Euclidean distance, speed, number of direction changes, and number of clicks in 500 ms time intervals.
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
5
The Effect of Cognitive Load on Mouse Behavior
The interface for the task is similar to that used in prior MD research (Freeman et al. 2011), with an anchor
button in the bottom center of the screen, response buttons located at opposite corners of the screen, and a
stimuli presentation area in the middle of the screen (Figure 3). Each button was 50 x 50 pixels (px), giving
participants precise targets on which to click. The stimuli presentation area was used to display a stream
of numbers that were approximately 2” in height. Each time the participant clicked the anchor button the
next number in the stream appeared for approximately one second, then disappeared. The participants were
required to click one of the response buttons, then click the anchor button again to advance. Participants
completed eighty iterations of each task, then self-reported the level of difficulty of the task. Pilot tests
identified 80 iterations as an appropriate number of trials to facilitate engagement, but not so many as to
induce fatigue. Further, for operational reasons it was desirable that the three tasks take a total of between
10 and 20 minutes. Analysis of completion times indicates the average time to complete all three tasks
was eleven minutes and fourteen seconds, with each individual task averaging approximately three to five
minutes, thus meeting our operational requirements.
Figure 3. Experiment Interface
The first task was designed to require very little cognitive effort. Participants were instructed to monitor
the stream of eighty numbers for the number 5. When they saw the number 5, they were to click the green
button in the top right corner of the screen. When they saw any number other than 5, they were to click the
red button in the top left of the screen. As expected, participants were very successful in this task, marking
the correct response 99.6% of the time (SD = 0.9, min = 95, max = 100).
The second task was designed to elicit slightly higher cognitive load by requiring lag-1 number recall and
simple comparison of two numbers. Participants were shown a stream of numbers similar to that used in
the first task. This time participants were instructed to click the green button in the top right corner of the
screen if the number on the screen was larger than the previous number, or the red button in the top left
corner of the screen if the number on the screen was smaller than the previous number. Overall participants
were slightly less successful in this task, marking the correct response 95.6% of the time (SD = 5.5, min =
74.4, max = 100).
The third and final task was considerably more cognitively demanding, requiring lag-2 recall, addition, and
comparison. As with the previous two tasks, participants were presented with a stream of eighty numbers.
This time they were instructed to add the previous two numbers together in their head and if the number
currently on the screen was larger than the sum of the previous two numbers click the green button in the
top right corner of the screen, or the red button in the top left corner of the screen if the number on the
screen was smaller than the sum of the two previous numbers. Participants were appreciably less successful
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
6
The Effect of Cognitive Load on Mouse Behavior
at this task, providing correct responses on average 81.3% of the time (SD = 13.2, min = 46.2, max = 98.7).
After each of the three tasks, participants completed the NASA Task Load Index (NASA-TLX; Hart & Staveland, L. 1988). The NASA-TLX is a multidimensional scale of perceived workload consisting of six items:
mental demand, physical demand, temporal demand, performance, effort, and frustration self-rated on a
seven point scale. While the full NASA-TLX instrument requires participants to provide a weighting of how
relevant they feel each of the six dimensions are to their personal workload, many studies use a “Raw” TLX
(RTLX) score which either averages, sums, or individually uses each of the six scores. The RTLX is faster
to administer, easier for participants to complete, and has demonstrated levels of sensitivity similar to the
complete instrument (Hart 2006), therefore, we opt to use the RTLX with averaged scores in this study.
After completing the three tasks and corresponding RTLX scales, participants read a debriefing screen and
were allowed to leave.
Analysis
To ensure the tasks elicited the expected levels of cognitive load, we performed two manipulation checks.
First, analysis of variance was conducted on the RTLX measures reported for each task. Participants reported RTLX scores that were in line with expectations. For task 1, participants reported an average RTLX
score of 3.03 (SD = 1.26). For task 2 participants reported a score of 3.70 (SD = 1.25) and for task 3
participants reported a score of 5.03 (SD = 1.17) (Figure 4). A repeated measures ANOVA confirmed there
was a significant difference between the three tasks, F (2, 104) = 83.2, p < .001 and a multilevel linear model
revealed significant differences between tasks 1 and 2, b = 0.66, t(104) = 4.20, p < .001 and between tasks 2
and 3, b = 1.34, t(104) = 8.46, p < .001. Effect sizes were calculated using the compute.es package in R (R
Core Development Team 2008). Cohen 1988 suggests d-values of less that 0.2 may be considered small, 0.5
considered medium, and over 0.8 considered large. Based on this guidance, the effect of the procedures in
tasks 2 and 3 on self-reported cognitive load, d=0.82 and d=1.62 respectively, may be considered large.
Figure 4. RTLX Scores for Each Task
Next we used an indirect objective measure of performance outcomes by examining the percentage of correct answers given for each of the tasks. As with the RTLX measures, there was a statistically significant
difference in performance for each of the three tasks. Participants scored on average 99.6% (SD = 0.9) on
task 1, 95.6% (SD = 5.5) on task 2, and 81.3% (SD = 13.2) on task 3. A multilevel linear model revealed a
significant difference with a medium effect (d=0.53)between the scores on tasks 1 and 2, b = 4.01, t(104) =
2.71, p < .01, as well as a significant difference with a large effect (d=1.87) between the scores for tasks 2 and
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
7
The Effect of Cognitive Load on Mouse Behavior
3, b = 14.28, t(104) = 9.64, p < .001. These results suggest there are indeed three distinct levels of cognitive
load being elicited, with task 2 eliciting a slight increase in cognitive load and task 3 eliciting a much larger
increase in cognitive load.
Proceeding to the hypothesis testing, we first examine the average total duration to complete each task.
Hypothesis 1a suggests that heightened cognitive load will lead to longer task duration. To test H1a, the
total time to complete each task was calculated. The data show that, on average, task 1 took 190.3 seconds
(SD = 22.5, min = 155, max = 251), task 2 took 208.9 seconds (SD = 35.4, min = 155, max = 344)
and task 3 took 276.7 seconds (SD = 89.3, min = 168, max = 621) (Figure 5). A multilevel linear model
was used to evaluate the differences between the groups. The analysis revealed a marginally significant
difference between tasks 1 and 2, b = 18.54, t(104) = 1.84, p = .068, with a small effect (d=0.36) and a
significant difference between tasks 2 and 3, b = 67.87, t(104) = 6.74, p < .001, with a large effect (d=1.31),
thus lending partial support to H1a.
Hypothesis 1b posits that when experiencing heightened cognitive load individuals will exhibit longer mouse
movements than when under lower levels of cognitive load. To test H1b, the Euclidean distance per task iteration was calculated. Participants traversed, on average, 1,683.8 px (SD = 1, 120.0, min = 1, 533.8, max =
1, 969.0) on task 1, 1,749.6 px (SD = 1, 214.4, min = 1, 523.9, max = 2029.6) for task 2, and 1,839.8 px
(SD = 1, 645.5, min = 1, 566.4, max = 2, 296.0) for task 3 for each iteration of the task (Figure 6). A multilevel linear model was used to analyze the data and revealed a significant difference between tasks 1 and
2, b = 65.78, t(104) = 2.98, p < .01, with a medium effect size (d=0.58) and a significant difference between
tasks 2 and 3, b = 90.24, t(104) = 4.10, p < .001, with a large effect size, (d=0.80). These results give support
to H1b.
Figure 5. H1a: Task Duration
Figure 6. H1b: Distance
Hypothesis 1c suggests that individuals will move the mouse at a slower speed when experiencing heightened
cognitive load. In task 1, the average speed of movement was 844.6 px/sec (SD = 101.6, min = 632.9, max =
1, 052.2), for task 2 the average speed was 793.8 px/sec (SD = 129.6, min = 458.1, max = 1, 026.3) and for
task 3 the average speed was 650.4 px/sec (SD = 158.3, min = 301.0, max = 909.2) (Figure 7). A multilevel
linear model revealed a statistically significant difference with a medium effect size (d=0.55) between tasks
1 and 2, b = 50.80, t(104) = 2.84, p < .01 as well as a significant difference with a large effect size (d=1.56)
between tasks 2 and 3, b = 143.50, t(104) = 8.04, p < .001. These results give support to H1c.
To test H1d, that users will exhibit more direction changes when experiencing heightened cognitive load, the
average number of direction changes for each task was calculated. In task 1, the average number of direction
changes per task iteration was 7.0 (SD = 1.2, min = 4.7, max = 10.1), for task 2 it was 7.4 (SD = 1.5, min =
5.2, max = 11.7) and for task 3 it was 8.3 (SD = 2.1, min = 4.7, max = 14.0) (Figure 8). A multilevel linear
model was used to evaluate the difference between the groups. A marginally significant difference with a
small effect size (d=0.32) was found between tasks 1 and 2, b = 0.35, t(104) = 1.67, p = .098 and a significant
difference with a large effect size (d=0.83) was found between tasks 2 and 3, b = 0.93, t(104) = 4.29, p < .001.
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
8
The Effect of Cognitive Load on Mouse Behavior
These results give partial support to H1d.
Figure 7. H1c: Speed
Figure 8. H1d: Direction Changes
Finally, to test H1e, that users will exhibit more mouse clicks when under high cognitive load, the average
number of clicks for each task was calculated. It should be noted that a minimum of two clicks were required
for each iteration – one click on the anchor button, and one click on the response. We find that in task 1, the
average number of clicks per task iteration was 2.18 (SD = 0.11, min = 2.03, max = 2.52), for task 2 it was
2.19 (SD = 0.12, min = 2.03, max = 2.49) and for task 3 it was 2.23 (SD = 0.12, min = 2.05, max = 2.59).
A multilevel linear model was used to evaluate the difference between the groups. There was no statistically
significant difference between tasks 1 and 2, b = 0.01, t(104) = 0.36, p > .1 and only a marginally significant
difference with a small effect size (d=0.38) between tasks 2 and 3, b = 0.03, t(104) = 1.98, p = .05, thus
giving partial support for H1e.
Results
Overall we found that participants spent more time on the tasks, moved the mouse further, moved the mouse
more slowly, and exhibited more direction changes when under high cognitive load, thus providing support
for hypotheses 1a-d for the high cognitive load condition (task 3), and partial support in the moderate cognitive load condition (task 2). In all cases the effects were stronger for the higher level of cognitive load than
lower level. These findings are concisely reported in Table 1.
Task 1 - Task 2
Duration
(seconds)
Distance
(pixels)
Speed
(pixels/sec)
Direction
changes
Clicks
Task 1
Task 2
Task 3
t-value
190.3
276.7
208.9
1.84†
(22.5)
(89.3)
(35.4)
1,749.6
1,839.8
1,683.8
2.98∗∗
(1,120.0) (1,214.4) (1,645.5)
793.8
650.4
844.6
2.84∗∗
(129.6)
(158.3)
(101.6)
7.4
7.0
8.3
1.67†
(1.5)
(1.2)
(2.1)
2.23
2.19
2.18
0.36
(0.1)
(0.1)
(0.1)
†
∗
∗∗
p < .1,
p < .05,
p < .01,
Task 2 - Task 3
Cohen’s d
t-value
Cohen’s d
0.36
6.74∗∗∗
1.31
0.58
4.10∗∗∗
0.80
0.55
8.04∗∗∗
1.56
0.32
4.29∗∗∗
0.83
0.07
1.98†
0.38
∗∗∗
p < .001
Table 1. Results (Standard Deviations in Parentheses)
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
9
The Effect of Cognitive Load on Mouse Behavior
Hypothesis
1: When experiencing increased cognitive load, individuals will exhibit:
a) longer total duration
b) longer Euclidean distance
c) slower movements
d) more direction changes
e) more clicks
Supported?
Partial
Yes
Yes
Partial
No
Table 2. Overview of Hypothesis Testing
While previous research has suggested clicking behavior may be a useful feature, we did not find compelling
results to support this as both the significance levels and effect sizes are low. It is possible that this is an
artifact of the context – for example, the targets may have been large enough (50 px2 ) that missed clicks
were less prevalent. Fewer clicks may also be the result of participants having only two choices and not
being able to change their answer. Prior work suggests the increase in clicks is not due to missing the target,
but rather due to the changing of one’s answer (Hibbeln et al. 2014). If additional clicks are the result of
participants being unsure about which answer to select, the reason for the non-significance in this study
could be because there was a definite answer to the questions in this study and participants felt confident
about their answer when they submitted their response. Future work should explore this in more depth.
A summary of the hypothesis testing results is presented in Table 2. The results of this study suggest that
there are several MD features that may be useful for identifying changes in cognitive load and are highly
encouraging for the future of this research.
Limitations
There are a number of limitations to this study. First, the elicitation of cognitive load was very specific
and deliberate. Participants were engaging in an artificial task designed to subject them to cognitive load.
Future work might consider more general use cases such as having participants engage in normal computer
use behavior while being subjected to extraneous cognitive load or engaging in normal computer tasks of
varying levels of complexity. Also, as pointed out by Brünken et al. (2003), physiological measures are
only indirectly linked to cognitive load and may, in fact, be the result of other factors such as attention or
motivation to succeed. In the current scenario, we are unsure of how motivated our participants were to
succeed. While the RTLX score suggest individuals found the later tasks more difficult, it is possible that
their reduced performance was not because of the difficulty, but rather due to apathy toward the task.
Second, technical malfunctions during the experiment invalidated data from approximately 22% of the participants. While this is unfortunate, ongoing enhancements to the experimental system will address this
issue. Finally, the scope of the findings are limited to the mouse. It has been suggested that there are inherent differences in devices such as the mouse and trackpad (Ahmed and Traore 2007; Gamboa and Fred
2004; Jorgensen and Yu 2011), thus future work should attempt to replicate these findings using alternative
pointing devices.
Discussion
There are many reasons a task may have high or low cognitive load. A task may have low cognitive load
because it is simple to perform and requires few cognitive resources, or because it has been performed many
times in the past, and thus the execution is routine and requires few cognitive resources to reproduce. On the
other hand, a task may have high cognitive load because it is difficult, or because there is some external factor
complicating the task. Regardless of the reason, understanding when users are experiencing heightened
levels of cognitive load has a number of useful applications for commerce, online learning, and beyond. For
example, when individuals are interacting with a web site or service, such as in an e-commerce setting, it
is desirable that they be able to find the product and options they are looking for with minimal effort. If
increased cognitive load were to be detected by a system such as the one described here, it might be possible
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
10
The Effect of Cognitive Load on Mouse Behavior
to launch a just-in-time service intervention in order to improve the shopping experience.
These findings also have interesting implications for online learning. For example, by observing when students exhibit changes in mouse behavior (i.e., longer, slower movements with a high number of direction
changes), it may be possible to identify when they are having trouble understanding a concept - similar to
seeing a confused look on a student’s face in a face-to-face class. This could also be useful in assessments,
where it might be possible to determine if a student was sure of the answer they provided, or if they were
just guessing. This technique could be used to create adaptive tests that are able to more accurately gauge a
student’s mastery of the material. This is particularly important as online learning is a rapidly growing area
for information systems researchers, practitioners, and educators.
Finally, this technique has applications for ensuring task diligence by identifying when people are multitasking or have otherwise diverted their attention. This would be useful in scenarios where it is desirable
to ensure task vigilance. It might also be useful for helping to identify different use cases – for example,
detecting when individuals are using their computer casually while watching television vs. when they are
concentrating at the task at hand and adapting the interface appropriately.
Conclusion
In this paper we have presented a technique for detecting heightened cognitive load by analyzing mouse
movement behavior. This work has theoretical foundation in decades of research in cognitive science, motor movements, and HCI behavior. Using a laboratory experiment, we found that individuals exhibited
longer task duration, longer mouse movements, slower speed, and a greater number of direction changes
when experiencing heightened cognitive load. By using unobtrusive techniques such as monitoring mouse
movement behavior, we propose that it is possible to greatly increase the efficacy of information systems
using inexpensive and widely deployed hardware such as the standard computer mouse.
References
Ahmed, A. and Traore, I. 2007. “A New Biometric Technology Based on Mouse Dynamics,” IEEE Transactions on Dependable and Secure Computing (4:3) 2007, pp. 165–179.
Asiu, B. W., Antons, C. M., and Fultz, M. L. 1998. “Undergraduate perceptions of survey participation: Improving response rates and validity,” in Annual meeting of the Association of Institutional Research,
Minneapolis, MN.
Beatty, J 1982. “Task-evoked pupillary responses, processing load, and the structure of processing resources.”
Psychological bulletin (91:1), pp. 276–292.
Bergadano, F., Gunetti, D., and Picardi, C. 2003. “Identity verification through dynamic keystroke analysis,”
Intelligent Data Analysis (7), pp. 469–496.
Brünken, R., Plass, J. L., and Leutner, D. 2003. “Direct Measurement of Cognitive Load in Multimedia
Learning,” Educational Psychologist (38:1), pp. 53–61.
Coelho, C. M., Lipp, O. V., Marinovic, W., Wallis, G., and Riek, S. 2010. “Increased corticospinal excitability
induced by unpleasant visual stimuli,” Neuroscience Letters (481:3) 2010, pp. 135–138.
Cohen, J 1988. “Statistical Power Analysis for the Behavioral Sciences,” ().
Coombes, S. a., Gamble, K. M., Cauraugh, J. H., and Janelle, C. M. 2008. “Emotional states alter force control
during a feedback occluded motor task.” Emotion (8:1) 2008, pp. 104–113.
Davis, F. D. 1989. “Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly
(13:3), pp. 319–340.
Fitts, P. M. 1954. “The information capacity of the human motor system in controlling the amplitude of
movement.” Journal of Experimental Psychology (121:6), pp. 262–269.
Fitts, P. and Peterson, J. 1964. “Information capacity of discrete motor responses.” Journal of experimental
psychology (67:2).
Freeman, J. B., Dale, R., and Farmer, T. A. 2011. “Hand in motion reveals mind in motion,” Frontiers in
Psychology (2).
Furnham, A. 1986. “Response bias, social desirability and dissimulation,” Personality and Individual Differences (7:3), pp. 385–400.
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
11
The Effect of Cognitive Load on Mouse Behavior
Gamboa, H. and Fred, A. 2004. “A behavioural biometric system based on human computer interaction,”
Proceedings of SPIE - The International Society for Optical Engineering (5404:i), pp. 381–392.
Graham, E. and MacKenzie, C. 1996. “Physical versus virtual pointing,” in Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, pp. 292–299.
Grimes, G. M., Jenkins, J. L., and Valacich, J. S. 2013. “Exploring the Effect of Arousal and Valence on Mouse
Interaction,” in International Conference on Information Systems, Milan, Italy, pp. 1–15.
Hart, S. G. 2006. “NASA-Task Load Index (NASA-TLX); 20 Years Later,” in Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908.
Hart & Staveland, L., S 1988. “Development of the NASA-TLX: Results of empirical and theoretical research,” in Human mental workload, pp. 139–183v.
Hibbeln, M., Jenkins, J, Schneider, C., Valacich, J., and Weinmann, M. 2014. “Investigating the Effect of
Insurance Fraud on Mouse Usage in Human-Computer Interactions,” in International Conference on
Information Systems, Dean 2004. Aukland, New Zealand, pp. 1–16.
Hick, W. E. 1952. “On the rate of gain of information,” Quarterly Journal of Experimental Psychology (4:1)
1952, pp. 11–26.
Hocquet, S, Ramel, J., and Cardot, H 2004. “Users authentication by a study of human computer interactions,” in Proceedings of the 8th Annual (Doctoral) Meeting on Health, Science and Technology,
Hudson, J. M., Christensen, J., Kellogg, W. a., and Erickson, T. 2002. “I’d be overwhelmed, but it’s just one
more thing to do,” in Proceedings of the SIGCHI conference on Human factors in computing systems
Changing our world, changing ourselves - CHI ’02, vol. Volume No. 4, p. 97.
Jorgensen, Z. and Yu, T. 2011. “On mouse dynamics as a behavioral biometric for authentication,” in Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security - ASIACCS ’11, New York, New York, USA: ACM Press, p. 476.
Kirmeyer, S. L. 1988. “Coping with competing demands: interruption and the type A pattern.” The Journal
of applied psychology (73:4), pp. 621–629.
Lali, P., Naghizadeh, M., Nasrollahi, H., Moradi, H., and Mirian, M. S. 2014. “Your mouse can tell about your
emotions,” in 2014 4th International Conference on Computer and Knowledge Engineering (ICCKE),
Ieee, 2014, pp. 47–51.
Maehr, W 2008. eMotion: Estimation of User’s Emotional State by Mouse Motions,
Mayer, R. E. 2001. Cognitive Theory of Multimedia Learning.
Merriënboer, J. J. G. van and Sweller, J. 2005. Cognitive Load Theory and Complex Learning: Recent
Developments and Future Directions, vol. 17. 2. 2005, pp. 147–177.
Meyer, D. E., Abrams, R. A., Kornblum, S, Wright, C. E., and Smith, J. E. 1988. “Optimality in human motor
performance: ideal control of rapid aimed movements.” Psychological review (95), pp. 340–370.
Meyer, D. E., Meyer, D. E., Abrams, R. A., Abrams, R. A., Wright, C. E., and Wright, C. E. 1990. “SpeedAccuracy Tradeoffs in Aimed Movements: Toward a Theory of Rapid Voluntary Action,” in Attention
and Performance XIII, pp. 173–226.
Paas, F., Tuovinen, J. E., Tabbers, H., and Van Gerven, P. W. M. 2003. Cognitive Load Measurement as a
Means to Advance Cognitive Load Theory.
Paas, F. G. W. C. and Merriënboer, J. J. G. van 1993. “The Efficiency of Instructional Conditions: An Approach to Combine Mental Effort and Performance Measures,” Human Factors: The Journal of the Human Factors and Ergonomics Society (35:4), pp. 737–743.
Paas, F. G. W. C. and Van Merriënboer, J. J. G. 1994. “Instructional control of cognitive load in the training
of complex cognitive tasks,” Educational Psychology Review (6), pp. 351–371.
Paas Renkl, A., Sweller, J., F 2003. “Cognitive Load Theory and Instructional Design: Recent Developments,” Educational Psychologist (38:January 2015), pp. 1–4.
Pantic, M., Pentland, A., Nijholt, A., and Huang, T. 2006. “Human computing and machine understanding
of human behavior,” Proceedings of the 8th international conference on Multimodal interfaces - ICMI
’06 (), p. 239.
Porter, S. R., Whitcomb, M. E., and Weitzer, W. H. 2004. “Multiple surveys of students and survey fatigue,”
New Directions for Institutional Research (2004:121), pp. 63–73.
Pusara, M. and Brodley, C. E. 2004. “User re-authentication via mouse movements,” Proceedings of the
2004 ACM workshop on Visualization and data mining for computer security VizSECDMSEC 04 (),
pp. 1–8.
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
12
The Effect of Cognitive Load on Mouse Behavior
R Core Development Team 2008. R: A language and environment for statistical computing.
Scheirer, J., Fernandez, R., Klein, J., and Picard, R. W. 2002. “Frustrating the user on purpose: a step toward
building an affective computer,” Interacting with Computers (14:2) 2002, pp. 93–118.
Shannon, C. E. 1948. “A mathematical theory of communication,” The Bell System Technical Journal (27:July
1928), pp. 379–423. arXiv: 9411012 [chao-dyn].
Smith, E. E. and Jonides, J 1997. “Working memory: a view from neuroimaging.” Cognitive psychology (42),
pp. 5–42.
Smith, S. D. and Kornelsen, J. 2011. “Emotion-dependent responses in spinal cord neurons: A spinal fMRI
study,” NeuroImage (58:1) 2011, pp. 269–274.
Speier, C., Valacich, J. S., and Vessey, I. 1999. “The Influence of Task Interruption on Individual Decision
Making: An Information Overload Perspective,” Decision Sciences (30:2), pp. 337–360.
Sweller, J 1994. “Cognitive load theory, learning difficulty, and instructional design,” Learning and instruction (4), pp. 295–312.
Sweller, J. 1988. “Cognitive load during problem solving: Effects on learning,” Cognitive Science (12), pp. 257–
285.
Tanaka, Y., Funase, K., Sekiya, H., and Murayama, T. 2012. “Modulation of Corticospinal Motor Tract Excitability during a Fine Finger Movement under Psychological Pressure: A TMS Study,” International
Journal of Sport and Health Science (10), pp. 39–49.
Van Beers, R. J., Haggard, P., and Wolpert, D. M. 2004. “The role of execution noise in movement variability.” Journal of neurophysiology (91:2) 2004, pp. 1050–1063.
Van Loon, A. M., Van den Wildenberg, W. P. M., Van Stegeren, A. H., Hajcak, G., and Ridderinkhof, K.
R. 2010. “Emotional stimuli modulate readiness for action: a transcranial magnetic stimulation study.”
Cognitive, affective & behavioral neuroscience (10:2) 2010, pp. 174–181.
Vizer, L. M. 2009. “Detecting cognitive and physical stress through typing behavior,” in Proceedings of the
27th international conference extended abstracts on Human factors in computing systems CHI EA 09,
New York, New York, USA: ACM Press, p. 3113.
Welsh, T. and Elliott, D. 2004. “Movement trajectories in the presence of a distracting stimulus: evidence for
a response activation model of selective reaching.” The Quarterly journal of experimental psychology.
(57:6) 2004, pp. 1031–1057.
Zimmermann, P., Guttormsen, S., Danuser, B., and Gomez, P. 2003. “Affective computing–a rationale for
measuring mood with mouse and keyboard.” International journal of occupational safety and ergonomics
: JOSE (9:4) 2003, pp. 539–551.
Thirty Sixth International Conference on Information Systems, Fort Worth 2015
13