Game experience of the Number Navigation Game:
Effects on arithmetic fluency and motivation
Gabriela Rodríguez Padillaa, Boglarka Brezovszkya, Nonmanut Pongsakdia, Tomi Jaakkolaab,
Minna Hannula-Sormunenbc, Jake McMullenab, Erno Lehtinenab
a
b
c
Centre for Learning Research, University of Turku
Department of Teacher Education, University of Turku
Turku Institute for Advanced Studies, University of Turku
Abstract
The Number Navigation Game (NNG) is a mathematical Serious Game designed to enhance
students’ flexible and adaptive arithmetic strategies and to increase motivation towards math.
Fourth to sixth grade classrooms were randomly sorted into either an experimental group
(students n=642) which played the game during a ten-week period or into a control group
(students n=526) which continued with a traditional mathematics curriculum. The aims of
this study were to: 1) describe the effect of an intervention on students’ arithmetic fluency
and expectancy-values; 2) describe the effect of game experiences on the experimental
group’s arithmetic fluency and expectancy-values; and 3) examine whether pre-test math
motivation and gaming proficiency influence game experiences. Results indicate that
regardless of the intervention, all participants showed an increase in arithmetic fluency and a
decrease in expectancy-values, although the intervention had a small effect in accentuating
these changes. Both gaming proficiency and pre-test expectancy-values influenced students’
game experience with the NNG, and students’ game experiences moderated the effectiveness
of the intervention in increasing motivation towards math. Students who were already highly
motivated towards mathematics were the ones who benefited the most from the intervention.
Keywords: Serious Game, GEQ, motivation, expectancy-value model, arithmetic
fluency
1 1. Introduction
The term “Serious Games” (Abt, 1970) originally referred to board and card games created as
educational tools. With time it has become the most common way to refer to digital games
specifically designed to produce cognitive, motivational, or physiological outcomes. The
Number Navigation Game (NNG) is a Serious Game which has been developed to enhance
students’ arithmetic flexibility and adaptivity. In the NNG a player must navigate across the
seas, retrieving items with which to build villages. To do this, a player takes control of a ship
and sails it by inputting mathematical equations which take the ship from one numericallocation to another. The NNG was conceived as an engaging platform in which students can
explore and reflect upon number combinations and the relationships between numbers. It
offers extensive and situated practice through which flexible and adaptive arithmetic problem
solving skills can be strengthened.
A survey carried out amongst Finnish school teachers by Klemmetti and colleagues (2009)
revealed that 99% of respondents (N=291) believed Serious Games would motivate students’
learning. Despite this assumption, there is a lack of empirical studies providing evidence that
Serious Games are motivating tools or are able to significantly increase motivation towards
learning (Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012). The main aim of the present
study was to find out whether playing the NNG is effective at improving students’ arithmetic
fluency and motivation towards mathematics, whilst also taking into consideration their game
experiences. As a side objective, the relationship between a) students’ pre-test motivation and
gaming proficiency with b) their game experiences was also studied. The present study is part
of a larger project trying to understand the effectiveness of the NNG.
1.1. Arithmetic fluency
Arithmetic fluency has been described as a by-product of children’s number sense. Being
able to memorize and quickly and accurately retrieve basic number combinations is both a
2 sign of deeper conceptual understanding (Baroody, Bajwa, & Eiland, 2009) and a requisite
for further conceptual and procedural development (Canobi, 2009) such as adaptivity and
flexibility. A main aim of the study was to explore the effects of an intervention with the
NNG on changes in students’ arithmetic fluency, as these changes would indicate that playing
the NNG has positive mathematical outcomes. While this study takes mostly a motivational
perspective, these learning outcomes must be also considered when understanding a game’s
effectiveness.
1.2. Motivation and the expectancy-value model
Motivation was looked at from the perspective of Eccles. and colleagues’ expectancy-values
model (Berger & Karabenick, 2011; Eccles & Wigfield, 2002; Wigfield & Cambria, 2010),
which is widely used in educational studies due to its usefulness in predicting students’ future
performance, persistence, and choices (Eccles and Wigfield, 2002). Psychological,
sociocultural, and contextual factors such as the feedback children receive from parents,
schools, and peers influence the development of their expectancy-values (Wigfield and
Cambria 2010). Expectancy itself refers to how well a person believes they will perform a
task, whereas values refer to a person’s reasons for engaging in a task. In other words, the
“expectancy” part refers to self-efficacy, or beliefs in one’s own ability. Task “value” is
composed of four aspects: interest, or how enjoyable a person finds the task; attainment
value, or how important it is for the person to perform well at the task; utility, or how useful
the task is for a person’s life; and cost, or what is the price a person believes they pay in order
to perform well in a task, both in terms of effort and time. Throughout this study those five
components (interest, utility, attainment value, cost, self-efficacy) are referred to as
expectancy-values and understood as indicators of achievement motivation. (Wigfield and
Cambria, 2010) A main aim of the study was to explore the changes in expectancy-values
3 because while motivation is bound to a specific context or situation, expectancy-values still
influence an individual’s engagement in an activity (Wigfield and Cambria, 2010).
1.3. Game Experience
For the purposes of this study, game experience was considered from a framework composed
of seven dimensions: competence, challenge, flow, sensory and imaginative immersion,
negative affect, positive affect, and tension. These dimensions are measured post-play
through the Game Experience Questionnaire (GEQ). Although originally developed to
measure users’ experiences with commercial games for entertainment, the GEQ has also been
used for Serious Games and has become an important tool as it offers the possibility to
develop a common understanding of game experience (Oksanen, 2013; Nacke, Stellmach, &
Lindley, 2011; De Grove, Van Looy, & Courtois, 2010; Poels, IJsselsteijn, de Kort, & Van
Iersel, 2010; Gajadhar, Nap, de Kort, & IJsselsteijn, 2008; Ijsselsteijn, de Kort, Poels,
Jurgelionis, & Bellotti, 2007). Ijsselsteijn and colleagues (2007) define flow as a central
dimension of game experience, and characterize it as a form of immersion which arises when
a player feels a balance between how competent they are and how challenging the game is.
They argue, however, that immersion is a broad concept in which sensory and imaginative
immersion is separate from flow or challenge-immersion. Sensory and imaginative
immersion refers to the absorption a player might feel towards game features such as story,
game world, graphics, or sound. The other dimensions of positive affect, negative affect, and
tension, focus more on post-play emotions which indicate how enjoyable the game
experience was. Complementing these widely used dimensions of game experience, our study
included an additional dimension of “positive value”, which measures students’ belief in the
positive value of the game. This dimension was added considering both the NNG’s learning
aims, and the argument that in order to benefit from game based learning, users must first
4 believe in the positive value of these games (Whitton, 2009). Exploring the effect of students’
game experiences on the NNG’s arithmetic and motivational outcomes was part of the
study’s main aim. It is necessary to understand the types of game experiences students have
when playing the NNG; as it is clear that positive game experiences foster engagement while
negative ones hinder it, an unengaged student might stop playing or only do so reluctantly
(Oksanen, 2013), which might have repercussions on the effectiveness of the NNG in
enhancing students’ arithmetic fluency or expectancy-values.
1.4. Gaming Proficiency
Research has shown that it is important to consider gaming proficiency when studying the
effectiveness of Serious Games. Proficiency playing digital games has been proven to predict
both future performance in Digital Game based environments as well as affective and
motivational learning outcomes (Orvis, Horn, & Belanich, 2007; Pavlas, Heyne, Bedwell,
Lazzara, & Salas, 2010). It has been argued that proficiency furthermore has an impact on
student’s acceptance of video games in the classroom (Bourgonjon, Valcke, Soetaert, &
Schellens, 2010). The acceptance of videogame use in the classroom is essential, as Whitton
(2009) has argued that users must believe in the positive value of games to benefit from them.
Three main aspects were studied as indicators of students’ gaming proficiency: their gaming
self-efficacy, the frequency with which they play games for fun, and the frequency with
which they play games for learning. A side aim of this study was to explore the relationship
between a) students’ pre-test math expectancy-values and their gaming proficiency with b)
their game experiences. Since math expectancy-values indicate a student’s willingness to
engage in a math activity (Wigfield and Cambria, 2010) and gaming proficiency influence the
acceptance and outcomes of a gaming activity, it is necessary to consider both when looking
at the effectiveness of the NNG.
5 2. Research Questions
The objective of this study is to answer the following research questions:
•
What is the effect of the intervention on students’ arithmetic fluency and expectancyvalues?
•
What is the effect of students’ game experiences with the NNG on their arithmetic
fluency and expectancy-values?
•
What are students’ pre-test expectancy-values and gaming proficiencies, and do these
predict students’ game experiences with the NNG?
3. Method
3.1. Participants
1168 participants from sixty-one 4th-6th grade classrooms spread across four cities in Finland
participated in this study. Participation was voluntary both at the class level and the
individual level, and informed consent was acquired in writing from the parents of all
participants. From the total n=546 were female, n=620 were male, and there was missing data
on the gender of n=2. As to grade level, n=135 were 4th graders, n=606 were 5th graders, and
n=427 were 6th graders. The mean ages for our 4th, 5th, and 6th grade participants are 10 years
and 2 ½ months, 11 years and 2 ½ months, and 12 years and 3 months old respectively.
Classes were randomly assigned into control and experimental groups, with n=642
participants belonging to the experimental group and n=526 to the control group.
3.2. Design
Table 1 Experimental Design
Control
Experimental
ü
ü
Pre-Test
First Phase
Math Test
Pre-Questionnaire
6 demographics
ü
ü
gaming proficiency items
ü
ü
math expectancy-values
ü
ü
Gaming
intervention
(10 weeks)
ü
NNG
Post-Test
Math Test
ü
ü
ü
ü
Post-Questionnaire
math expectancy-values
ü
Gaming Experience Questionnaire
Gaming
intervention
NNG
ü
During the spring term 2014, the experimental group played NNG for a ten week period
while the control group continued with a traditional mathematics curriculum (see Table 1).
Afterwards, conditions were reversed. While the present study only encompasses this first
phase of the experiment, it is relevant to mention the reversal of conditions not only because
it would have been ethically questionable to deny participants in the control group the chance
to play, but also because the control group’s knowledge about the upcoming play sessions
could have an impact on some post-test measures. This must be considered when looking at
the results. Amongst the experimental group playing the NNG, the actual implementation of
the game varied greatly from class to class. Teachers were invited for a training session in
which they were informed about the NNG’s learning aims and play mechanics. As part of
their training, teachers were told sessions needed to last at least 30 minutes in order to give
their students enough time to make significant progress in the game. Nevertheless teachers
were free to decide how long play sessions would extend, how to space these sessions
throughout the intervention, what kind of support they’d provide their students, and whether
7 students would play individually or in pairs. In case students played in pairs, teachers chose
the criteria under which pairs would be formed. At the end of the school year, all participants
received a copy of the NNG, but were not otherwise monetarily rewarded. As part of the
testing procedures, however, students were thanked for contributing towards research, and the
importance of their participation as research team members was emphasized.
3.3. Materials
See Number Navigation Game description, attached.
3.4. Measures
Data used for this study was collected from questionnaires and math tests before and after the
intervention. Pre- and post- math tests were rigorously timed and structured, and were
imparted by trained testers following standardized procedures. Both pre- and postquestionnaires were filled out by students during regular class time under the guidance of
their teachers. The pre-questionnaire was identical for all participants while the postquestionnaire had additional items for the experimental group.
3.4.1. Arithmetic Fluency
Students’ fluency in solving basic arithmetic tasks is used in this study as an indicator of
cognitive outcomes. The test was adapted from the Math Fluency test of the WoodcockJohnson Tests of Achievement (Woodcock, McGrew, & Mather, 2001), in which students
have three minutes to answer as many simple arithmetic problems out of 160 as they can.
3.4.2. Math Expectancy-Values
Fourteen Likert-scale items (Appendix A) measuring math expectancy-values were
completed before and after the intervention by all participants. These items were studied
through principal component analysis with varimax rotation. Five separate factors (interest,
utility, attainment value, self-efficacy, and cost) were found, upholding the 5-factor model
developed by Eccles and Wigfield (2002). The explained variance of the model was 75.90%.
8 Data was adequate for factor analysis with a 0.90 Kaiser–Meyer–Olkin Measure, and
Barlett’s test of sphericity showed a significance of < 0.001. The factors were shown to have
good internal consistency and to be reliable across the two tests. At pre-test: interest
Cronbach’s α= 0.91, utility: α= 0.80, attainment value: α= 0.82, cost: α= 0.51, and math selfefficacy α= 0.81. Post-test reliabilities were the same except for cost, α= 0.58 at post-test. The
poor reliability of cost compared to the other measures can be due to this dimension only
having two items.
3.4.3. Gaming Proficiency
Measures considered for students’ gaming proficiency were taken from their prequestionnaire responses and focused on their gaming self-efficacy, the frequency with which
they play games for fun, and the frequency with which they play games for learning. Access
to technology was measured as a control variable. Gaming self-efficacy was a reliable
measure (α= 0.87) and was computed from items taken from Orvis (2007): I am certain I will
be good at most videogames I try to play, I am confident in my ability to play videogames,
and I am confident in using new technologies and software.
3.4.4. Game Experience
Only participants in the experimental group were asked to fill out the core experience module
of the GEQ after the intervention. The Finnish translations used by Oksanen (2013) were
used, although the questionnaire was further modified by removing 14 of the 42 items and
changing some of the phrasings to better suit our game and the ages of our respondents. The
original items of the GEQ and all the modifications, including the addition of the dimension
“belief in the positive value of the game”, can be found in Annex B. Each item consisted of a
statement and a 1-5 scale to indicate level of agreement, with answers ranging from not at all,
very little, a bit, quite a lot, and extremely. The factor structure of the 31 items of GEQ was
studied through principal component analysis with varimax rotation. Data was adequate for
9 factor analysis with a 0.95 Kaiser-Meyer-Olkin Measure and Barlett’s test of sphericity
showed a significance of <.001. Seven separate factors were found and used as basis for the
subscales. The explained variance of the model was 69.60%. The reliability of subscales was
as follows: Challenge α= 0.63, Competence α= 0.81, Flow α= 0.78, Immersion α= 0.78,
Negative affect α= 0.81, Positive Affect α= 0.94, Positive value α= 0.82, and Tension α=
0.76. The reliability for challenge is low. This could be due to the removal of two items,
although Oksanen (2013) also reported similar results.
4. Results
Results are organized into three subsections. The first subsection (4.1) consists of descriptive
statistics detailing students’ pre-test expectancy-values, gaming proficiency, and game
experiences with the NNG. The second subsection (4.2) describes the correlations of gaming
proficiency and pre-test expectancy-values with game experiences. Finally, the third
subsection (4.3) presents results from repeated-measures ANOVA analyses carried out on
SPSS inn order to find the effects of the intervention and game experiences on post-test
arithmetic fluency and expectancy-values.
4.1. Descriptive statistics
Descriptive statistics are presented in Table 2. Results support previously reported gender
differences showing that boys have higher feelings of interest, attainment value, and selfefficacy than girls, while girls have higher feelings of cost than boys do (Wigfield and
Cambria, 2010). With the exception of the higher interest reported by 5th graders in
comparison to 4th graders, there is a decrease by grade-level in expectancy-values. Overall,
there are strong and significant correlations between interest, utility, attainment value, and
math self-efficacy (Table 3). Cost somewhat correlates with interest, utility, and attainment
value, but does not correlate with math self-efficacy, although a negative correlation between
10 how good students feel they are at math and how much time and effort they feel they must
exert was expected.
11 Table 2 Descriptive Statistics
All Participants
Condition
Control
Gender
Experimental
Girls
Grade Level
Boys
4
5
6
N
M
SD
N
M
SD
N
M
SD
N
M
SD
N
M
SD
N
M
SD
N
M
SD
N
M
SD
Interest 1
1070
3.13
1.00
414
3.11
1.01
516
3.16
0.97
496
2.98
1.00
573
3.26
0.98
128
3.18
1.07
559
3.25
1.01
383
2.93
0.92
Interest 2
1010
3.12
0.98
414
3.17
0.96
516
3.04
1.01
469
2.97
0.93
540
3.24
1.01
129
4.42
0.71
474
3.22
0.95
407
2.96
0.92
Utility 1
1084
4.19
0.76
418
4.24
0.79
534
4.19
0.72
507
4.19
0.71
576
4.19
0.80
130
4.24
0.86
564
4.23
0.71
390
4.11
0.79
Utility 2
1017
4.21
0.73
418
4.29
0.73
534
4.13
0.73
474
4.21
0.71
542
4.20
0.75
127
4.42
0.71
477
4.20
0.75
413
4.15
0.70
Attainment
1068
3.50
0.87
409
3.49
0.89
510
3.54
0.82
496
3.40
0.87
571
3.59
0.86
127
3.66
0.93
560
3.53
0.89
381
3.40
0.80
999
3.43
0.88
409
3.45
0.89
510
3.38
0.89
468
3.37
0.86
530
3.48
0.90
125
3.59
0.95
470
3.47
0.90
404
3.33
0.84
Cost 1
1076
2.41
0.84
414
2.39
0.84
519
2.41
0.82
499
2.44
0.84
576
2.39
0.83
126
2.50
0.88
565
2.39
0.82
385
2.43
0.85
Cost 2
1007
2.31
0.85
414
2.35
0.81
519
2.22
0.88
471
2.38
0.82
536
2.24
0.87
127
2.37
0.98
469
2.28
0.82
411
2.32
0.84
Math Self-
1067
3.70
0.83
409
3.74
0.84
512
3.70
0.79
494
3.56
0.83
572
3.82
0.82
128
3.90
0.79
556
3.71
0.86
383
3.62
0.78
1002
3.65
0.85
409
3.67
0.82
512
3.63
0.88
463
3.50
0.82
538
3.79
0.84
128
3.87
0.87
469
3.67
0.86
405
3.57
0.82
1057
70.24
17.94
467
69.92
17.20
590
70.49
18.51
491
69.58
17.37
565
70.78
18.41
125
60.65
18.18
556
68.70
17.71
376
75.71
16.39
ExpectancyValues
Value 1
Attainment
Value 2
Efficacy 1
Math SelfEfficacy 2
Arithmetic
Fluency
Fluency 1
12 Fluency 2
1087
78.49
19.68
490
77.17
19.41
597
79.57
19.84
502
78.96
18.65
585
78.08
20.53
121
68.56
16.94
567
76.71
19.18
399
84.02
19.54
1085
3.51
1.04
489
3.54
1.05
596
3.49
1.04
503
3.03
0.94
581
3.93
0.94
130
3.49
1.08
565
3.49
1.08
390
3.55
0.97
Challenge
505
2.34
0.78
231
2.40
0.71
274
2.29
0.82
37
2.39
0.90
216
2.37
0.77
252
2.31
0.76
Competence
512
3.09
0.94
235
2.89
0.90
277
3.26
0.94
37
3.32
0.82
219
3.20
0.95
256
2.96
0.92
Flow
506
2.03
0.83
239
2.05
0.77
267
2.02
0.88
37
2.31
1.00
214
2.06
0.86
255
1.97
0.77
Immersion
514
2.02
0.91
238
1.98
0.86
276
2.05
0.96
38
2.15
1.14
218
2.13
0.97
258
1.91
0.81
226
3.06
1.02
270
3.10
1.05
34
2.69
1.16
215
3.09
1.01
247
3.12
1.03
496
3.08
1.03
236
2.21
0.97
270
2.31
1.12
36
2.53
1.26
216
2.34
1.09
254
2.16
0.97
506
2.26
1.05
238
2.24
0.87
277
2.32
1.02
37
2.58
1.18
220
2.34
0.98
258
2.19
0.88
515
2.28
0.95
516
2.56
1.26
239
2.52
1.20
277
2.60
1.31
37
2.20
1.22
221
2.71
1.28
258
2.49
1.23
Gaming
Gaming
SelfEfficacy
GEQ
Negative
Affect
Positive
Affect
Positive
Value
Tension
13 Regarding students’ gaming proficiency, access to technology was not an issue for
participants, as most (n=1080, missing data n=88) reported having access to at least one of
the following platforms: a PC or laptop at home (92%), a smartphone (87%), a gaming
console (86%), and/or a Tablet (65%). The mean score in gaming self-efficacy of all
participants was 3.51 (SD=1.04) (see Table 2). There were no considerable differences in
gaming self-efficacy by grade level, but there were differences between the gaming selfefficacy of girls (M=3.03, SD=0.94) and boys (M=3.93, SD=0.94). While access to
technology correlated with gaming self-efficacy and the frequency with which students play
for fun, these correlations are much smaller and weaker than anticipated. Based on
questionnaire responses, participants frequently play games for fun. Very few respondents
claimed to play never (2.3%) or rarely (9.6%). Most participants claim to play sometimes
(28.5%), frequently (31.5%), or on a daily basis (19.3%). Playing games for learning is still
very uncommon, with an overwhelming majority claiming they never (33%) or rarely
(39.4%) play for learning purposes, and only few reporting to play these games sometimes
(12.8%), frequently (4%), or on a daily basis (0.7%).
Mean scores from the experimental group’s game experiences with the NNG indicate that
there is room for improvement (Table 2). Out of a maximum score of 5, the dimensions with
the highest mean scores were competence (M=3.09) and negative affect (M=3.08). With the
exception of tension (M=2.56), all other dimensions of the GEQ were rated below the scale’s
midpoint of 2.5. Students’ experiences playing the NNG differed by gender and grade level
(Table 2). The most interesting differences concern grade level and game experiences. The
scores for positive affect, immersion, competence, and flow steadily decrease each grade
level while the scores for negative affect and tension increase. The only exception is tension,
as 5th graders gave a higher score to tension than 6th graders. Playtime (less or more than 5
14 hours) and play mode (individual play or pair play) were also examined but as there were no
significant differences between the groups, these analyses are not included.
4.2. Correlations of pre-test expectancy-values and gaming proficiencies with game
experiences
Table 3 Correlations of Pre-test expectancy values, gaming proficiencies, and the GEQ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1
6
1.
Interest 1
-
2.
Utility 1
.46*
*
3.
Attainme
nt Value
-
.60*
.57*
*
*
.20*
.24*
.29*
*
*
*
.56*
.42*
.56*
*
*
*
.06
.07*
.14*
-
1
4.
5.
Cost
Math
Self-
-
-.00
-
.02
.18*
Efficay 1
6.
Gaming
*
Self-
-
*
Efficacy
7.
Freq.
.21*
.17*
.20*
.24*
.10*
Games
*
*
*
*
*
-.04
-.03
.01
-
.01
-.02
-
.52*
.00
-
.10*
.0
for
Learning
8.
Freq.
.08*
Games
*
for Fun
9.
Challeng
e
-.04
-.01
-
.16*
-
-
.11*
*
.21*
.10*
-
0
*
15 10. Compete
nce
11. Flow
.29*
.10*
*
.20*
n
13. Negative
Affect
14. Positive
Affect
15. Positive
Value
16. Tension
.20*
.01
.01
*
-
-.03
*
*
12. Immersio
.32*
.18*
.18*
*
*
.17*
.09*
*
-.01
-.07
.13*
*
*
.02
*
.19*
.18*
.08
*
-
.09*
*
.04
.18*
*
*
.07
-.01
.12*
.05
*
.22*
.26*
.41*
-.01
.06
-.07
-.08
.00
-
9
.19*
-
*
.0
.49*
.38*
2
*
*
-
.21*
-
*
.0
.45*
.47*
.73*
0
*
*
*
-
.0
-
-
-
-
*
.19*
6
.28*
.25*
.57*
.55*
*
*
*
*
*
.14*
-.00
-
.01
.01
*
-.07
.0
.16*
*
.18*
.10*
.08
.18*
.0
*
2
.20*
.0
*
3
-
.44*
.53*
.77*
.79*
.64*
*
*
*
*
*
-
-
-
.54*
.44*
.66*
.72*
.51*
.73*
*
*
*
*
*
*
-
-
-
-
-
-
.13*
.12*
.0
.18*
.21*
.36*
.39*
*
*
4
*
*
*
*
-
-
.69*
.52*
.39*
*
*
*
-
**p<.01, *p<.05
The correlations of students’ a) pre-test math expectancy-values (interest, utility, attainment
value, cost, and math self-efficacy) and gaming proficiency (gaming self-efficacy, frequency
of play for fun, frequency of play for learning) with b) the dimensions of the GEQ (challenge,
competence, flow, immersion, negative affect, positive affect, positive value, tension) are
presented in Table 3. Utility did not correlate with the dimensions of the GEQ, as there was
only a very negligible correlation between utility and competence. Otherwise, high pre-test
expectancy-values correlated with positive game experiences playing the NNG while low
pre-test expectancy-values correlated with negative game experiences. However these
correlations are not very strong- the strongest one was between pre-test math self-efficacy
16 and competence (r=.41 p = <.01). There was a small but interesting negative correlation
between negative affect and cost (r=-.13 p = <.01). Students who felt that succeeding in math
requires a lot of their time and effort reported lower negative affect, which could mean they
appreciated a game-based platform to practice. The dimension of positive value added to the
GEQ for the purposes of this study refers to the game’s usefulness for learning math, yet
students’ self-reported positive value did not correlate either with their pre-test utility or
attainment value. This suggests that even when students believe math is useful or important
in their lives, they do not necessarily believe that playing the NNG is helpful achieving this.
This could be for many reasons: the game is misleadingly simple looking, so students might
be unaware of the complex strategic thought required; teacher-led reflection and support
metacognitive activities might be insufficient; the game might simply not be challenging
enough; or it could be an issue of design, with students requiring to complete many easy
maps before reaching more challenging ones, which would mean the game is not properly
scaffolded and design must be improved; or it could simply be that students see videogames,
even educational ones, primarily as a recreational activity.
Comparing the roles of math self-efficacy and gaming self-efficacy were of particular
interest. Both had a negative correlation with the dimension of challenge and a positive
correlation with dimension of competence, but in both cases the role of math self-efficacy
proved to be stronger and more significant. Math self-efficacy furthermore correlated with
immersion while gaming self-efficacy did not. Finally, gaming self-efficacy alone correlated
with negative affect, while math self-efficacy alone correlated with positive affect. When
students feel they are good at math, they enjoy playing NNG, whereas when students feel
they are good at games, they do not.
17 4.00 3.50 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Challenge Competence Flow Immersion Nega;ve Affect Posi;ve Affect Posi;ve Value Tension Never 2.20 2.85 1.72 1.56 3.49 1.76 1.61 3.00 Rarely 2.38 2.88 2.16 1.88 3.06 2.23 2.28 2.74 Some;mes 2.39 3.07 2.08 2.17 2.90 2.35 2.39 2.54 Frequently 2.33 3.11 2.02 2.00 3.11 2.26 2.30 2.46 Daily 2.36 3.17 1.98 1.91 3.28 2.23 2.22 2.66 Figure 1 Mean scores to the GEQ by frequency of playing games for fun per week
Frequently playing games for learning led to more positive experiences playing the NNG,
correlating positively with the dimensions of challenge, competence, flow, immersion,
positive affect, and positive value, and negatively with negative affect and tension. There was
no linear correlation between the frequency with which students play games for fun and their
experiences playing the NNG. Looking at mean scores, however, it appears that students who
play games for fun moderately have more positive experiences playing NNG than those
students who play games for fun never or daily (Figure 1). It seems that up to a certain extent,
playing games for fun makes participants more open to positive experiences playing the
NNG. Students who play games extremely rarely or extremely often are more critical than
those with more moderate gaming habits. In short, while both pre-test expectancy-values and
gaming proficiencies correlate with the different dimensions of the GEQ, they do so in
different ways and to different degrees. Being motivated towards math is a better predictor of
students’ enjoyment of NNG. Students who already feel motivated towards math are the ones
who will most enjoy their experiences playing NNG.
18 4.3. Repeated measures: the effect of condition and game experiences on arithmetic fluency
and expectancy-values
Repeated-measures ANOVA analyses were done on SPSS with the expectancy-value
measures as within-subjects factors. The first analysis looked at all participants and took
condition (experimental or control) as a between-subjects factor. For the second analysis,
only participants in the experimental group were considered, and groups based on game
experiences were used as a between-subjects factor. To create the game experience-groups, a
K-means cluster analysis was first carried out on SPSS to create cluster groups based on the
experimental group’s responses to GEQ. Three distinct experience-groups were found based
on game experience: one for students whose experiences playing NNG were mostly negative
(n=151), a second group for students whose experiences were mostly positive (n=83), and a
third group for students with mixed feelings about their experiences (n=206).
Table 4 Effect sizes by condition (experimental or control)
Main Effect (All Participants)
Interaction Effect (Condition)
F
p
ηp2
F
p
ηp2
Arithmetic Fluency
(1,986) = 589.52
< 0.001
0.374
(1,986) = 5.99
.015
0.006
Interest
(1,928) = 1.40
.237
0.002
(1,928) = 12.51
<0.001
0.013
Utility
(1,950) = 0.08
.78
0.000
(1,950) = 5.76
.017
0.006
Attainment Value
(1,917) = 17.53
< 0.005
0.019
(1,917) = 5.27
.022
0.006
Cost
(1,931) = 18.91
< 0.005
0.020
(1,931) = 6.82
< 0.01
0.007
Self-Efficacy
(1,919) = 9.52
< 0.01
0.010
(1,919) = 0.00
.98
0.000
Table 4 presents the effects of the intervention on arithmetic fluency and expectancy-values.
While all of participants’ arithmetic fluency increased, participants in the experimental group
showed more of an improvement (Table 2). Overall there was no main effect of time on
19 interest or utility, but there was a small but significant interaction effect between condition
and interest and between condition and utility. The experimental group’s mean interest and
utility scores decreased while the control group’s mean interest and utility scores increased.
This could perhaps be explained by the fact that participants in the control group were aware
they would be participating in a large experiment and playing the NNG soon and were
excited about it, although this is just speculation. There was a significant effect of time on
attainment value and on cost, with attainment value and cost somewhat decreasing for all
participants, although the decrease was more pronounced for the experimental group. The
interaction effects were small. There was a significant effect of time on math self-efficacy,
but in this case condition did not have a significant effect. Results are in line with previous
research reporting a general decrease of expectancy-values throughout the school term
(Wigfield and Cambria, 2010; Berger & Karabenick, 2011). Students in the control group did
counter the trend by showing an increase in interest and utility from pre- to post-test,
although data is not sufficient to determine whether this results from students’ anticipation of
a condition reversal. For the most part, playing the NNG did not have a large impact on
students’ expectancy values. Compared to the control group, the experimental group showed
a larger decrease in all dimensions, but this effect was quite small.
Table 5 Effect sizes of Game Experience on Cognitive and Motivational Outcomes
Main Effect
Interaction Effect
(only experimental group)
(game experience-groups)
F
p
ηp2
Arithmetic Fluency
(1,383)=238.34
<.0001
Interest
(1,401)=7.53
Utility
(1,408)=0.01
F
p
ηp2
0.384 (2,383)=0.45
.636
0.002
.006
0.018 (2,401)=10.90
<.0001
0.052
.912
0.000 (2,408)=14.34
<.0001
0.066
20 Attainment Value
(1,393)=14.05
<.0001
0.035 (2,393)=6.52
.002
0.032
Cost
(1,400)=13.04
<.0001
0.032 (2,400)=3.06
.048
0.015
Self-Efficacy
(1,395)=0.49
.484
0.001 (2,395)=6.80
.001
0.033
Table 5 presents the effects sizes of arithmetic fluency and expectancy-values by experiencegroups. Amongst the experimental group, there was a significant main effect of time from
pre-test to post-test on arithmetic fluency, interest, attainment value, and cost (Table 5). No
significant main effect of time on either utility or math self-efficacy was found. There were
no significant interaction effects between the groups with different gaming experiences
(negative, mixed, or positive) and arithmetic fluency, which means that while there was a
significant difference over time in the experimental group’s arithmetic fluency, this was not
related to their subjective experiences playing the NNG. However there was an interaction
effect between the groups with different gaming experiences and their expectancy-values.
The interest of students with positive experiences increased (Table 6). Although interest
decreased amongst students with either mixed or negative experiences, the decrease is more
clear-cut amongst the latter. While mean scores of utility and math self-efficacy did not
substantially change, there was nevertheless an interaction effect. Students who felt playing
the NNG was a positive experience became more convinced that math would be useful to
their lives, while the opposite was true for students with negative or mixed experiences.
Students’ beliefs of their self-efficacy in math remained stable amongst students with mixed
experiences, while they went down for students with negative experiences and up for students
with positive experiences. The attainment values of students with positive and mixed gaming
experiences remained for the most part stable, while the attainment values of students with
negative experiences went down. The cost scores of the group with positive experiences
remained stable, while students with mixed and negative experiences saw a decrease in cost.
21 Table 6 Effectiveness of the Intervention by Experience Groups: Descriptive Statistics
Arithmetic
Interest
Utility
Cost
Fluency
Pre-test
M (SD)
Mixed
Positive
Negative
Attainment
Math Self-
Value
Efficacy
Post-
Pre-
Post-
Pre-
Post-
Pre-
Post-
Pre-
Post-
Pre-
Post-
Test
test
Test
test
Test
test
Test
test
Test
test
Test
M (SD)
M
M
M
M
M
M
M
M
M
M
(SD)
(SD)
(SD)
(SD)
(SD)
(SD)
(SD)
(SD)
(SD)
(SD)
70.65
79.86
3.13
3.08
4.20
4.13
3.48
3.32
2.46
2.26
3.63
3.57
(17.60)
(19.58)
(0.84)
(0.86)
(0.62)
(0.68)
(0.70)
(0.75)
(0.77)
(0.77)
(0.70)
(0.74)
68.55
78.96
3.50
3.56
4.13
4.44
3.80
3.87
2.58
2.66
3.85
4.01
(18.77)
(21.39)
(0.86)
(0.81)
(0.87)
(0.60)
(0.69)
(0.68)
(0.77)
(0.85)
(0.83)
(0.77)
74.90
83.37
2.85
2.54
4.12
3.92
3.45
3.12
2.26
2.03
3.64
3.48
(19.04)
(21.26)
(1.03)
(1.07)
(0.73)
(0.78)
(0.88)
(0.96)
(0.84)
(0.90)
(0.85)
(0.98)
5. Conclusions
5.1. Discussion
Arithmetic fluency increased from pre-test to post-test, regardless of students’ game
experiences, and playing the NNG had a small effect on this fluency increase. This indicates
that the game was successful as a platform for extensive situated practice which enhances
mathematical skills. Further analyses need to be carried out in order to understand the game’s
effectiveness at enhancing also deeper mathematical skills such as arithmetic adaptivity and
flexibility. As for expectancy-values, attainment value, cost, and math self-efficacy decreased
for all participants despite the intervention. Unexpectedly, the control group’s interest and
utility increased from pre-test to post-test, which may or may not have been sparked by the
anticipation of having their turn playing the NNG. While the intervention had an effect on a
decrease of expectancy-values, effect sizes were quite small. Furthermore, when focusing
exclusively on the experimental group, results show that the quality of students’ game
22 experiences moderate the effect of the intervention on their expectancy-values, with mixed to
negative experiences resulting in a decrease of interest, utility, attainment value, cost, and
math self-efficacy while positive gaming experiences result in significant improvements in all
these dimensions.
The high mean scores of the game experience dimensions of negative affect and competence
suggest that the game might have been too simple, although it is unclear whether this was due
to the game’s form (game-play) or content (arithmetic strategies needed) or both. A
relationship between game experiences and students’ backgrounds was found. Both gaming
proficiency and pre-test expectancy-values had implications on the way participants rated
their experiences playing the NNG, although math expectancy-values correlated more
strongly with almost all dimensions of the GEQ, leading to the conclusion that having high
math expectancy-values is a stronger predictor of enjoying the NNG than gaming proficiency
is. Salient amongst results is that gaming self-efficacy correlated with negative affect while
math self-efficacy correlated with positive affect, which suggests that students who feel they
are good at playing digital games are prone to dislike the NNG while students who feel they
are good at math are prone to like it. Results support Whitton’s (2011) claim that it cannot be
assumed that a game dynamic will automatically make something interesting to learners who
have no interest in the subject itself (2011). Results further suggest that it is students who are
proficient at digital games who least enjoy playing the NNG. More studies are needed to
determine whether this could be remedied by improving the design of the game.
5.2. Implications
As it was established that high expectancy values result in better gaming experiences and that
experiences play a role on the effectiveness of the game in improving expectancy-values, it
seems that at its current form NNG helps students who already from the start are highly
motivated towards mathematics. The game version used in this experiment did not have
23 particularly motivating items. It seems that the gaming mechanism as such was not
motivating for the majority of students even though it resulted in improvement in
mathematical skills. These findings are valuable for the further development of the game. The
basic mechanism seems to work and result in improved mathematical skills but there should
be better ways to engage students and to give them enjoyable gaming experiences. Based on
the results it is important to further develop the game and analyze whether new game features
lead to meaningful improvements in gaming experiences. It is encouraging that regardless of
the quality of gaming experience, NNG is still effective in improving arithmetic fluency.
However it is essential that this increase in arithmetic fluency does not come hand-in-hand
with a decrease in expectancy-values.
5.3. Limitations and Future Directions
Conditions varied greatly between classrooms, so we cannot be sure what was the role of the
teacher for example in debriefing, feedback, support activities, or reflection. However, as the
goal was for the game to be used in the most natural school settings possible, it was decided
this would be achieved by giving teachers the freedom to use the game as they saw fit. A
major limitation of this study is its dependence on subjective and self-reported data. Informal
feedback from teachers paints a different picture on students’ experiences, as many teachers
claimed their students were very engaged while playing and enjoyed the experience. This will
be remedied in the future with the addition of a feedback feature within the game itself,
which will make it possible for players to give feedback on their affective states upon
completing a map, in a situated way that does not disrupt flow.
When looking at Serious Games, it’s important to acknowledge their oxymoronic nature
(Abt, 1987). Jenkins (2011) brings up the contradictory relationship of playing- a “freely
chosen irresponsibility”- and learning- an “assigned responsibility”. Along similar lines, it
has been argued that having teachers decide what games will be played, for how long, and
24 under which circumstances, will have repercussions on the levels of control felt by students
and consequentially on their motivation (Wouters, van Nimwegen, van Oostendrop & van der
Spek, 2013). Different results may be achieved when play is free and voluntary, as opposed
to a formal and prescribed school activity (Islas Sedano, 2013), as it has been reported that
playing in a different context, such as home, increases players’ enjoyment, identification, and
learning experiences (De Grove, Van Looy, Neys, & Jansz, 2012). An important next step
could be to study the effect of having students play voluntarily at their homes on motivation
and core gaming experiences.
Annex A
Interest
I like math.
I enjoy doing math.
Math is exciting to me.
Utility
Math is valuable because it will help me in the future.
Math is useful for me in everyday life.
Being good at math will be useful when I go to college or get a job.
Attainment
It is important to me to be a student who is good at math.
Value
I believe that being good at math is an important part of who I am.
It is important to me to be a student who understands operations.
Cost
I have to use a lot of time to do well in math.
I believe that success in math requires that I give up other activities that I
25 enjoy.
Self-Efficacy
I am sure that I can learn math.
I am certain I can do difficult math tasks.
I believe I will get a good grade in math.
Annex B
Original items
Modifications
Competenc
I felt skillful.
I felt skillful when I played.
e
I felt strong.
-
I was good at it.
I was good at playing.
I felt successful.
I felt successful when I played.
I was fast at reaching the
-
game's targets.
Challenge
I felt competent.
I felt I was good at the game.
I felt that I was learning.
I felt that I was learning when I played.
I thought it was hard.
I thought playing this game was hard.
I felt stimulated.
-
I felt challenged.
I thought the game was difficult enough.
I had to put a lot of effort into
I had to put a lot of effort when I played.
it.
Flow
I felt time pressure.
-
I felt completely absorbed.
I felt completely absorbed by the game.
I forgot everything around me.
I forgot everything around me when I
played.
I lost track of time.
I lost track of time when I played.
26 I was deeply concentrated in
I was deeply concentrated in the game.
the game.
I lost connection with the
-
outside world.
I was fully occupied with the
I was fully occupied with the game.
game.
Immersion
I was interested in the game's
I was interested in the game's story.
story.
It was aesthetically pleasing.
-
I felt imaginative.
I felt imaginative when I played.
I felt that I could explore
I felt I could explore how to navigate around
things.
numbers when I played.
I found it impressive.
-
It felt like a rich experience.
Negative
I thought about other things.
Affect
I was thinking about other things while
playing.
I found it tiresome.
I thought playing was boring.
I felt bored.
I was bored while playing.
I was distracted.
-
I was bored by the story.
The game's idea felt boring.
It gave me a bad mood.
Playing gave me a bad mood.
Positive
I felt content.
I felt content when I played.
Affect
I could laugh about it.
-
I felt happy.
I felt happy when I played.
I felt good.
I felt good when I played.
27 Tension
I enjoyed it.
I enjoyed playing.
I thought it was fun.
I thought playing was fun.
I felt tense.
-
I felt restless.
-
I felt annoyed.
Playing annoyed me.
I felt irritable.
I felt irritable when I played.
I felt frustrated.
-
I felt pressured.
-
Positive
+This game helped me learn math.
value of the
+I like math more after playing this game.
game
+I have gotten better at math after playing.
References
Abt, C. (1987). Serious Games. Boston: University Press of America.
Baroody, A.J., Bajwa, N.P., and Eiland, M. (2009). Why can't Johnny remember the basic
facts? Developmental Disabilities. 15(1), 69-79.
Berger, J., & Karabenick, SA (2011). Motivation and Students' Use of Learning Strategies:
Evidence of Unidirectional Effects in Mathematics Classrooms. Learning and Instruction, 21
(3), 416-428
Bourgonjon, J., Valcke, M., Soetaert, R., and Schellens, T. ( 2010) Students’ perceptions
about the use of video games in the classroom. Computers & Education, 54, 1145–1156
Canobi, K. (2009). Concept–procedure interactions in children’s addition and subtraction.
Journal of Experimental Child Psychology, 102, 131–149
28 De Grove F., Van Looy, J., Neys, J., and Jansz, J. (2012) Playing in School or at Home? An
Exploration of the Effects of Context on Educational Game Experience. Electronic Journal of
e-Learning. 10(2), 83 - 208
De Grove, F., Van Looy, J., & Courtois, C. (2010). Towards a Serious Game Experience
Model: Validation, Extension and Adaptation of the GEQ for Use in an Educational Context.
In L. Calvi, K. Nuijten, & H. Bouwknegt (Eds.), Playability and Player Experience (Vol. 10,
pp. 47-61). Breda University of Applied Sciences.
Eccles, J.S., and Wigfield, A. (2002).Motivational Beliefs, Values, and Goals. Annual
Review of Psychology. 53, 109-132.
Connolly, T.M., Boyle, E. A., MacArthur, E., Hainey, T. and Boyle, J. M. (2012). A
systematic literature review of the empirical evidence on computer games and serious games.
Computers & Education, 59, 661 – 686
Gajadhar, B., Nap, H., De Kort, Y., & IJsselsteijn, W. (2008). Shared fun is doubled fun:
Player enjoyment as a function of social setting. In P. Markopoulos, B. Ruyter, W.
IJsselsteijn, & D. Rowland (Eds.), Proceedings of the 2nd International Conference on Fun
and Games (pp. 106-117). Berlin, Germany: Springer-Verlag.
Ijsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., and Bellotti, F. (2007). Characterising
and Measuring User Experiences in Digital Games. 2007 International Conference on
Advances in Computer Entertainment Technology, Workshop 'Methods for Evaluating
Games - How to measure Usability and User Experience in Games', ACM Digital Library,
Salzburg, Austria.
29 Islas Sedano, C., Leendertz, V., Vinni, M., Sutinen, E., and Ellis, S. (2013)
Hypercontextualized Learning Games: Fantasy, Motivation, and Engagement in Reality.
Simulation & Gaming, 44(6), 821-845.
Jenkins, H. (2011) Foreword. In deFreitas, S. & Maharg, P. (Eds.). (2011) Digital Games and
Learning.
(pp. xxiii-xxvi) New York: Continuum
Klemetti, M., Taimisto, O., & Karppinen, P. (2009). The Attitudes of Finnish School
Teachers Towards Commercial Educational Games. In Kankaanranta, M. & Neittaanmäki, P.
(Eds.) Design and Use of Serious Games (pp. 97-106). Jyväskylä: Springer.
Nacke, L., Stellmach, S., and Lindley, C.A. (2011) Electroencephalographic Assessment of
Player Experience: A Pilot Study in Affective Ludology. Simulation & Gaming. 42(5), 632655
Oksanen, K. (2013). Subjective experience and sociability in a collaborative serious game.
Simulation & Gaming, 44(6), 767-793.
Orvis, K., Horn, B., & Belanich, J. (2007). The roles of task difficulty and prior videogame
experience on performance and motivation in instructional videogame. Computers in Human
Behavior, 24, 2415-2433.
Pavlas, D., Heyne, K., Bedwell, W., Lazzara, El., and Salas, E. (2010) Game-based Learning:
The Impact of Flow State and Videogame Self-efficacy. Proceedings of the Human Factors
and Ergonomics Society Annual Meeting. 54:2398
Poels, K., De Kort, Y. A. W., & IJsselsteijn, W. A. (2007). It is always a lot of fun!—
Exploring dimensions of digital game experience using focus group methodology. In
30 Proceedings of the 2007 Conference on Future Play (pp. 83-89). New York, NY: ACM
Press.
Poels, K., IJsselsteijn, W., de Kort, Y., and Van Iersel, B. (2010). Digital Games, the
Aftermath. Qualitative insights into Post Game Experiences. In R. Bernhaupt, R. (Ed.).
Evaluating User Experiences in Games. Berlin: Springer.
Whitton, N. (2011) Game Engagement Theory and Adult Learning. Simulation & Gaming.
42(5), 597-610
Whitton, N. (2009). Learning with Digital Games: A Practical Guide to Engaging Students in
Higher Education. New York: Routledge.
Wigfield, A. and Cambria, J. (2010). Expectancy-value theory: retrospective and prospective.
In T. Urdan, & S.A. Karabenich (Eds.), Advances in motivation and achievement. The next
decade of research in motivation and achievement, Vol. 16A (pp.35-70). London: Emerald.
Woodcock, R., McGrew, K., & Mather, N. (2001). Woodcock-Johnson tests of achievement.
Itasca, IL: Riverside Publishing.
Wouter, P., van Nimwegen, C., van Oostendorp, H., and van der Spek, E.D. (2013) A MetaAnalysis of the Cognitive and Motivational Effects of Serious Games. Journal of Educational
Psychology. 105(2), 249-265.
31