ARTICLE IN PRESS
Int. J. Human-Computer Studies 65 (2007) 709–723
www.elsevier.com/locate/ijhcs
Key factors of heuristic evaluation for game design: Towards massively
multi-player online role-playing game
Seungkeun Song, Joohyeon Lee
Cognitive Science Program, The Graduate School, Yonsei University, #134, Shinchon-dong, Seodaemoon-ku, Seoul 120-749, Korea
Received 14 December 2005; received in revised form 25 December 2006; accepted 3 January 2007
Communicated by S. Wiedenbeck
Available online 2 February 2007
Abstract
The computer game industry has become the fastest growing field of the entertainment industry. However, only a very small number of
computer game products overcome the costs of production and generate earnings. According to traditional marketing wisdom,
customers’ preferences must be analyzed correctly to create successful products, and in the gaming industry, such information must be
considered during the design process. This research aims to explore key factors of heuristic evaluation for game design. A review of
literature pertaining to computer games and HCI was conducted along with an empirical research of a Massively Multi-player Online
Role-playing Game (MMORPG). We identified 18 usability issues in MMORPG and presented a recommendation relevant to the issues.
Empirical data were applied to a new heuristic evaluation framework. We determined the relationship between key factors and four game
categories, such as game interface, game play, game narrative, and game mechanics. Moreover, the results presented 54 key factors for a
new heuristic evaluation framework for game design. The conclusion presents key implications of our research in a game design context,
particularly related to early design processes.
r 2007 Elsevier Ltd. All rights reserved.
Keywords: MMORPG game design; Usability; Heuristics evaluation; Design process
1. Introduction
The computer game industry has become the fastest
growing field of the entertainment industry. However, only
a very small number of computer game products overcome
the costs of production and generate earnings (Armstrong
and Hamm, 1997). According to traditional marketing
wisdom, customer preference is a core issue affecting the
creation of successful products (Kotler, 1994), and the
game design process is crucial to ensuring gamer satisfaction (Fabricatore et al., 2002).
Recent studies have shown that the business model for
Massively Multi-player Online Role-playing Game
(MMORPG) relies mainly on gamer subscription fees,
and that most gamers subscribe to only one game (Yee,
Corresponding author. Tel.: 82 2 2123 3108; fax: 82 2 364 2440.
E-mail addresses: [email protected] (S. Song),
[email protected] (J. Lee).
1071-5819/$ - see front matter r 2007 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ijhcs.2007.01.001
2003). The number of new subscribers tends to grow slowly
relative to other types of games (Woodcock, 2003). Since
the success of any future game title depends on how much
interest it can garner in the traditional game market
(Wagner, 2002), game novices should be able to enjoy
games easily by lowering the game entry threshold.
Consequently, researching game novices is as important
as researching game experts (Cornett, 2004).
When current game design processes are examined, they
proceed in the following order: synopsis, background and
context research, script writing, visualization and concept
art, interaction design, level design, materials, models and
animation, media editing, programming, integration, beta
test, and publishing (modified from Manninen, 2004). One
major problem of this process is that user satisfaction, or
whether the game is fun or not, can only be determined
after all the procedures have been completed. The time and
cost of game development could be reduced if user
satisfaction were known at the early stages of the design
ARTICLE IN PRESS
710
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
process, or even during game development. One way to
solve this problem is to provide an iterative evaluation
during specific design phases (Dumas, 2003).
The goals of this study, therefore, are to (1) explore the
key factors of heuristic evaluation for MMORPG design
and (2) to present key implications of evaluative support
during the early stages of the game design process.
2. Literature review
2.1. Current evaluation methods for computer games
Gamers can provide important feedback regarding game
play that will significantly enhance the fun they have while
playing it. Evaluation methods for acquiring gamer feedback have been applied to computer games with varying
success. Some evaluation methods, such as surveys, have
been used frequently to collect information from gamers.
A game designer would conduct a survey of gamers to find
out which game features are liked or disliked. Correctly
conducted surveys yield critical data that designers can
employ in the generation of game features. However,
surveys are difficult to conduct well. Survey questions are
difficult to write in a manner that yields high-quality data.
The sample for the survey must be large enough to conduct
statistically significant analyses and comparisons, and it
must be representative of the population of interest
(Bradburn and Sudman, 1988; Sudman et al., 1996).
Furthermore, surveys have retrospective attribute. Gamers
are asked to think about their past experiences with a game
when responding to a survey regarding a new game. It is
difficult to ensure that survey responses are unbiased as
respondents base their perceptions on long-time game
interactions that serve to distort their long-term memory
and allow for re-interpretation (Ericsson and Simon, 1993;
Van Someren et al., 1994). Therefore, it is difficult to draw
general conclusions from the data of such surveys.
The focus group interview has also been used in the game
design process for various purposes, including helping
game designers determine how gamers perceive a game.
Eight–twelve gamers meet to discuss topics that the game
designers are interested in. Focus group interviews are
typically useful for concept generation during the early
stages of the design process (Krueger, 2000). However,
focus group interviews do not adequately aid game
improvement because participants respond to abstract
gaming concepts rather than concept development (Fulton
and Medlock, 2002). Additionally, interviewees may be
sensitive to group dynamics and pressures that could
impact the quality of survey results. Because group
pressures can influence individual participants, results
obtained from group interviews may provide misleading
and biased data although they had not intended to (Fulton
and Medlock, 2002).
The most popular source of gamer opinion is beta testing
conducted early during the development process. In general,
beta testers are volunteers who are recruited from various
fields to play an early version of a computer game. They are
asked to provide information about technical bugs in the
software, but they can also provide feedback about issues
relating to game play. Although the results of beta tests are
crucial for identifying important bugs in games, they are less
effective in helping to identify and fix game play issues or
issues relating to fun. Beta testers are generally highly
advanced players rather than members of the general gamer
population. Moreover, feedback from beta tests is not
collected in a systematical manner. For example, beta testers
are asked to play the game as much as they can and provide
feedback about any problems they encounter. Consequently,
Pagulayan et al. (2003) mention that game designers have no
direct control over their game play because beta testers play
at home or work. These limitations prohibit obtaining
specific feedback from beta testers.
So far there have been some discussions about the
strengths and weakness of user-based evaluation versus
heuristic evaluation in Human–Computer Interaction field
(Kantner and Rosenbaum, 1997). User-based usability
evaluation can be an excellent source of behavioral
information about gamers. Usability research can help
identify issues that obstruct gamers from experiencing the
fun or enjoyment of a computer game. However, the
typically small sample sizes used in usability evaluations
prohibit generalization to a wider population. Moreover, as
usability feedback often comes late after development
decisions have been made, it is difficult to provide feedback
relevant to game development. That is, the delay between
when a product feature is developed and when usability
feedback is conveyed to the developers precludes the use of
those recommendations (Medlock et al., 2002). In order to
overcome this limitation, the heuristic evaluation method
has been developed to provide game designers with
immediate feedback to identify the general issues at the
beginning of the computer game design process. Desurvire
et al. (2004) conducted a comparison between user testing
and ‘Heuristic Evaluation for Playability’ (HEP) based on
the literature related to video games, computer games, and
board games. The efficiency of the heuristics method was
assessed with a new prototype game at the beginning of the
design process. The HEP was reviewed by several game
designers and experts. The playability evaluator conducted
HEP while focusing on how each heuristic was supported or
violated, and then identified the usability issue. At the same
time, four subjects were given instructions to begin the game
and think aloud while being asked several probing questions
during 2-h game play sessions. The results of the experiment
revealed the total number of issues identified from HEP to
be greater than the number of issues found from the user
study while user study issues, such as terminology,
characters, and verbiage, were found to be more gamespecific. Most notably, the study revealed a key implication
for usability evaluation methods in computer game.
Specifically, HEP was found to maintain qualitative
similarities and differences with user testing, but HEP was
found to be best suited for evaluating general issues at the
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
beginning of a game development cycle, during the
prototyping process. Although there is the strength of
HEP, it holds many issues that are cosmetic positives.
Consequently, HEP generates most issues that are only part
of the truth. In addition, the drawback of HEP is that it uses
surrogate users, not typical users of online game. The results
of heuristic evaluation are not primary user data and thus
are slightly suspected (Nielsen, 1993; Dumas and Redish,
1999). Moreover, HEP is highly dependent on the skills and
experience of the evaluators. Usability specialists may lack
domain expertise, and domain specialists are rarely trained
or experienced in usability methodology. In spite of the
weakness of heuristic evaluation, it is more proper to detect
general issues than user-based evaluation at the beginning of
a game development (Desurvire et al., 2004). Thus, it might
be a better idea for the evaluators to select the most
appropriate method between heuristic evaluation and usedbased evaluation according to the features of each stage in
game design process.
Prevalent game evaluation methods, such as surveys,
focus group interviews, beta-testing, and usability evaluations, have been reviewed briefly, with the positive and
negative aspects of each method criticized. The primary
purpose of this study is to extend the work of Desurvire
et al. (2004), to critically review all aspects of their
methods, and to develop a new heuristic evaluation method
that minimizes the negative aspects and employs the
positive aspects of each method. To this end, the first step
to be carried out is to extract key factors in evaluation
methods based on the characteristics of games and to use
them as evaluation methods.
2.2. New aspects of usability evaluation for computer games
The study of software usability has expanded rapidly in
the last 10 years, but there have been comparatively few
studies of usability in games (Federoff, 2002). This
disparity may result from the differences that exist between
computer games and traditional software productivity.
Usability of traditional software is primarily focused on
productivity, which comprises ease to learn and ease to use
as its goal. Quite differently, the goal of usability in games
is to reduce obstacles against fun, or to enhance the
enjoyment of participation. Games are fundamentally
different than general software. Games are mostly used
to enjoy leisure time or have fun. Eventually, games are
enjoyed in a manner similar to movies or novels. As a
result, the goal of usability evaluation in games must be
considered differently. Moreover, software productivity
requires an effort for consistency. If users use different
methods or tools each time, then they will suffer from
having to learn something new each time. Games, however,
have to provide a variety of experiences. Each time gamers
play a game, they should be led to learn new rules and
attempt new strategies to achieve goals. Otherwise, gamers
may easily become bored. Since they include the feature of
learning and exploration, games should provide a variety
711
Game Task
Game Narrative
Game Interface
Game
Play
Game
Mechanics
Fig. 1. The aspects of usability evaluation of computer games.
of experiences each time they are played. Furthermore,
some constraints are deliberately imposed on a game, while
they must be eliminated from traditional software. Such
constraints may cause unexpected design problems in
software productivity. A game, however, requires some
constraints to increase enjoyment. Consequently, we must
consider the manners in which usability differs between
computer games and traditional software, and thus how
usability should be evaluated (Pagulayan et al., 2003).
In summary, although traditional usability evaluation is
said to be superior to other evaluation methods, such as
surveys, focus groups, and beta-testing, it is necessary to
review the applicability of usability evaluation based on the
nature of games. As previously mentioned, traditional
usability evaluation, such as that used for software
productivity, may not be directly applied to computer
games. The primary focus of usability evaluation for
computer games should be on reducing obstructive factors
and promoting enjoyment instead of making the game easy
to use with minimal mistakes.
Given the above, what aspects of games can we evaluate
for usability? According to Clanton (1998), the different
usability issues in games include the game interface, game
play, and the game mechanics. Game interface comprises
all devices that gamers come into contact with as they
interact with a game. Game play signifies all the processes
that gamers go through to achieve the goal of a game.
Game mechanics comprises the physics of a game, which is
fabricated through the aggregation of artwork and
program. Besides these issues, usability is also influenced
by the tasks that gamers must or desire to accomplish, and
the narratives that yield fun, enjoyment, and curiosity
(Jenkins, 2002; Desurvire et al., 2004). Fig. 1 represents
aspects of usability evaluation as determined from a review
of literature pertinent to usability evaluation of games.
3. Empirical research
3.1. Overview of research
This research aims to explore key factors affecting
usability evaluation on the basis of empirical data. To this
end, we conducted an experimental study including the
ARTICLE IN PRESS
712
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
analysis of gamer tasks and a post-questionnaire. Usability
issues were revealed, and a solution for the issues was
explored. Empirical data were applied to an initial heuristic
evaluation framework. Through correspondence analysis,
we discovered a relationship between 54 key factors in four
game categories, being the game interface, game play, the
game narrative, and game mechanics. The 54 key factors
were derived from both empirical results and literature
review. This section presents the specific setting for our
experiment. Section 4 describes the experimental results. In
particular, Section 4.4 details the relationships between
known key factors, the game categories, and the 54 key
factors derived from a new framework of heuristic
evaluation for MMORPG.
3.2. Experiment material
Among the various MMORPG titles available, we
selected World of Warcrafts (WOW) for its commercial
success (ESA, 2005) and its record of excellent playability
as established by various game critics.1 According to
Kasavin (2006), a famous reviewer for Gamespot.com,
‘‘WOW is superficially similar to numerous other games.
The game’s interface is so slick and easy to learn and
understand, and the game play itself is so quickly intuitive,
that there is not even a tutorial to wade through; there are
just some helpful, optional pop-up tool tips, as well as an
excellent printed reference manual that goes into specific
detail about most of the various aspects of play.’’
WOW provides immediate feedbacks of the current
status using audio and visual interface. This function
serves as the criterion to measure correctly the performance
of tasks such as success, error, and failure. WOW provides
gamers with log data related to the extent of their
damage during combat, which was different from other
MMORPGs. This log data can be used as the objective
index for the empirical research.
In WOW, game progresses through quests. Quests
signify another game comprised of small units (i.e., a game
within a game). In case of novel or a play, they are often
subdivided through the concept of chapter. In the game
design, stage and map are utilized to reduce tedium and to
induce gamers’ immersion. In addition, the total content is
divided into subcontents by mission or quest. What gamers
should do for each quest is apparent because the total game
is divided into smaller units using quests in WOW.
Traditional MMORPGs, before the appearance of
WOW, had the problem that gamers often forgot their
goal due to the endless story and a wide game space. This
problem was solved through the concept of quests by
WOW, which is the first time in the game industry.
Moreover, gamers may perceive the problems and challenges easily that they must face to win the game because
/http://www.gamespot.com/pc/rpg/worldofwarcraft/review.html,
http://www.mmorpg.com/gamelist.cfm/gameId/15/setView/review, http://
pc.gamespy.com/pc/world-of-warcraft/571585p1.htmlS.
1
WOW has an excellent design that allows for proper
constraints per game content. These design factors in
WOW are regarded as the important nature of game play.
WOW is MMORPG based on the myth of North
Europe that is the motif of the ‘‘Lord of ring’’ or ‘‘Harry
Potter’’. WOW has a perfect narrative structure with its
sophisticated design. These aspects play an important role
in assessing game narrative in terms of usability.
WOW is a spectacular game due to its various extensions
in terms of content and design although it has a concise,
definite structure. In addition, WOW is unique because of
the introduction of new game features, such as auction
system, mail system, and diverse transportations. Such
things were not considered in other MMORPGs. These
aspects are relevant to game mechanics, which is achieved
through a combination of animation and programming.
WOW, therefore, is worth studying for game interface,
game play, game narrative, and game mechanics in
usability evaluation for MMORPG. For this reason, we
selected WOW for our experiment.
3.3. Subjects
Eighteen undergraduate students participated in the
experiment at Kyonggi University, Seoul, Korea. All
subjects were students attending a course, titled ‘‘Understanding of Contemporary Multimedia’’. The experiment
was conducted at the end of the term, and all the subjects
were given tasks related to computer games several times. In
this experiment, the subjects were classified into two groups
according to their level of expertise with MMORPGs. The
first level represented novices who were new to MMORPGs,
and the second level represented experts that were experienced with MMORPGs but new to WOW games. Subjects
were divided into two levels because playing patterns could
differ by experience level, and because game design and
evaluation would have to differ accordingly.
3.4. Experimental task
We developed experimental tasks based on the analysis
of the help function, or tutorial, for WOW, which is
represented by an exclamation mark during play of the
Korean version of WOW. For instance, if a gamer tries to
hunt for a monster, clicking on the monster results in two
exclamation marks in the middle of the bottom of the
computer screen. The first exclamation mark refers to a
battle mode, meaning, ‘‘Right clicking on the enemy and
approaching it initiates fighting. General fighting occurs
automatically.’’ The second exclamation mark is associated
with magic and power, meaning, ‘‘magic and power can be
applied by clicking on the various icons at the bottom on
the left-hand side.’’ After the battle ends, another
exclamation mark appears. This mark signifies the obtainment of trophies, and in the exclamation mark it states,
‘‘If you want to search for items on the dead body, then
you can click the right button of the mouse on the corpse.
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
713
Table 1
Fundamental tasks of the MMORPG game WOW
No.
Task
Detailed description
Right click a dead body, and then right click items on loot panel
Use a relevant skill after generating a rage
Right click anNPC with an orange exclamation point (i.e., !)
Right click an item on the backpack panel
Click a merchant NPC, and then right click an item on the backpack
Click a Holy Grail shaped-icon on the right button part (Menu Bar) in the interface
Hold down the left mouse button to look around the world. Hold down the right mouse button to turn
your character using the mouse
Click a backpack-shaped con on the right bottom part (storage) of the interface
Drag the ability to your action bar (i.e., quick slots)
Right click the enemy and approach it
Movement driven by mouse right click, Movement driven by keyboard (press ’mk-, AWSD)
Left click the target
After going looking for Lein Bishare NPC, gamers learn the ability suited for his or her own level
Chat with other PCs or use the community function
Join party, and conduct the role of warrior for party play
Task
Task
Task
Task
Task
Task
Task
1
2
3
4
5
6
7
Acquire items
Skilled-attack
Chat with a NPC with !
Equip an item
Trade items
Identify quest
Adjust viewpoint
Task
Task
Task
Task
Task
Task
Task
Task
8
9
10
11
12
13
14
15
Find inventory
Register the ability
First fight
Movement
Select a target
Learn abilities
Find party fellow
Party play
Right clicking on the item again leads to the gamer’s
inventory.’’ As explained above, instructions for playing a
game are typically provided at the beginning of a game, but
in WOW, game instructions are provided peculiarly via the
exclamation mark, or help icon in a unique way. This study
involved observation of all exclamation marks, which in
sum acted as a tutorial, and tasks were subsequently based
on tutorial observation. The experimental tasks are
provided in Table 1. Tasks frequently used in real game
play are developed for the early stages of the game so that
they may be learned quickly and naturally, because it
would often be uncertain how the first-time WOW gamers
would begin the game. In addition, we developed several
tasks that might be deeply associated with factors affecting
a usability evaluation of the game.
3.5. Experiment procedure
The experimental sessions were divided into four sections.
First, subjects were given instructions about the general
nature of the experiment. They were asked to fill out a prequestionnaire as well as a written consent to experiment
participation. The pre-questionnaire included the subjects’
demographic backgrounds, gaming experience and level of
expertise. Second, all subjects were asked to select only the
warrior class from the human race of the Alliance,2 one of
the player characters in the game, in order to guarantee a
2
Two character types, either alliance or horde members, exist in the
game of WOW. Alliance characters represent good, and horde characters
represent evil. The two character types face off in conflict. Our
participants were instructed to choose to be members of the alliance,
and then to be human from among several races, and then to be a warrior
from among some available classes. With these choices made, the
characters enter the game in the North-shire Valley, Elwynn Forest, a
region of the virtual world in which the game is played. If a different race
and type of character were chosen, then the gamers would begin the game
in a different location, thus leading to a different gaming experience. As
such, subjects were asked to choose the same character type, race, and
class to ensure a consistent experimental environment.
homogeneous experimental environment for the participants. The actual playing session ranged from 1 to 2 h. The
researcher recorded subjects’ utterance and actions during
game play using audio and video equipment. Third, a postquestionnaire was provided to determine subjects’ subjective
attitude towards game play after playing session. The postquestionnaire was divided into three sections: game interface, game play, and game narrative. The post-questionnaire
was based on 7-point Likert Scale. Finally, the participants
were debriefed by the researcher about the primary purpose
of this experiment, the need for confidentiality, the
participant’s responses and feelings, and so on.
The experiment station carried two PCs with Windows
XP, Intel Pentium IV 2.8 GHz processors with 512 MB
RAM, GeForce graphics card with 128 MB RAM, Highspeed LAN connections, dual 200 GB HDD, two video
camcorders, amplified goose-neck microphones, a movie
clip capture program called Fraps 2.5, and Premier 6.5
which is a digital movie clip editor.
3.6. Data analysis
The purpose of the study is to investigate the behavioral
differences between the expert and novice gamer groups,
both of whose members were encountering the game for the
first time. To this end, for the 15 tasks in the experiment, we
conducted a frequency analysis, computed the arithmetic
mean, and performed a t-test to compare the difference
between the performance of the novice and expert groups
(i.e., the elapsed time required to complete the 15 tasks).
3.7. The criterion for task experiment
We measured subject performance in completing their
tasks. Tasks completed with errors were those requiring
minimal prompting, and subjects were considered to
have failed a task when its completion required explicit
direction.
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
714
4. Empirical result
4.1. Pre-questionnaire results
The pre-questionnaire was administered to ascertain subject
profiles, including information such as age, sex, online gaming
experience, and general knowledge of online games. Table 2
summarizes these profiles. The subjects chosen for the study
represent the general demographic distribution of computer
gamers (ESA, 2005). Mean age of the subjects is 20.36 with
maximum 24, minimum 18, and standard deviation 1.934.
4.2. The behavioral differences between both groups
Table 3 summarizes the subjects’ performance of tasks.
The analysis of tasks shows significant differences between
Table 2
Subject’s profiles
Profile
Expert
Novice
Total
No. of subjects
9
9
18
Age
Mean: 19.22
Max: 20
Min: 18
SD: 0.667
Mean: 21.15
Max: 24
Min: 19
SD: 2.154
Mean: 20.36
Max: 24
Min: 18
SD: 1.934
Sex
Male: 8
Female: 1
Male: 4
Female: 5
Male: 12
Female: 6
Experience with
online games
Never: 0
Never: 3
Never: 3
1–2 years: 4
Over 2 years: 5
1–2 years: 6
Over 2 years: 0
1–2 years: 10
Over 2 years: 5
Beginner: 3
Beginner: 9
Beginner: 12
Intermediate: 3
Advanced: 3
Intermediate: 0
Advanced: 0
Intermediate: 3
Advanced: 3
General knowledge
of online games
novice and expert groups. However, a detailed presentation
of experimental task performance is not presented for
brevity. Instead, we present an analysis of some tasks with
frequent errors and failures.
Task 1, which involved the acquisition of items, was
performed with two procedures. First, the subject was
required to click the right button of the mouse on a dead
monster, and then click on the applicable item when the
trophy panel appeared. The average elapsed time required
to complete the task was 1 min 11 s for novices 24 s for
experts. The novices took about three times longer than the
experts, representing a significant difference in task
performance time (t(16) ¼ 1.917, po0.05). The task
showed high error (7/18) and failure (4/18) rates relative
to other tasks. Many subjects (11/18) thought that right
clicking the dead monster, which caused the appearance of
the trophy panel, was enough to obtain answers such as
‘‘the item has been acquired,’’ or ‘‘it’s showing the item I
acquired’’ (see Fig. 2).
There was also another case related to subject error in
acquiring items. When subjects (6/18) first clicked the right
button of the mouse on the dead monster, causing the
trophy panel to appear, they clicked the description part of
the item rather than the icon-shaped item on the trophy
panel. This behavior is attributed to the size of the item
explanation part being three times larger than that of the
icon-shaped item part in terms of the visual interface (see
Fig. 3).
Task 5 involved trading items acquired during hunting
for sale of purchase of goods from a merchant Non-Player
Character (NPC). For the novice group, task 5 took an
average of 1 min and 35 s, while the expert group took only
58 s. The novices spent twice longer than the expert, again
representing a significant difference in completion time
(t(14) ¼ 0.708, po0.05). As with case 3, the error (4/16)
and failure (3/16) rates were higher relative to those of
other tasks.
Table 3
Behavioral differences between novice and expert groups
No.
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Task
Tasks
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Acquire items
Skilled-attack
Chat with a NPC with !
Equip items
Trade items
Identify quest
Adjust viewpoint
Find inventory
Register abilities
First fight
Movement
Select a target
Learn abilities
Find party fellow
Party play
Error
7/18
3/18
3/18
2/18
4/16
3/18
1/12
1/17
0/7
0/18
0/18
0/18
1/9
1/8
0/5
Failure
4/18
8/18
5/18
4/18
3/16
2/18
2/12
2/17
3/7
1/18
0/18
0/18
0/9
0/8
1/5
Average elapsed time (min:s)
Novice
Expert
1:11
1:02
0:41
0:30
1:35
0:20
0:16
0:21
1:08
0:37
0:27
0:00
0:45
0:14
19:56
0:24
0:10
0:07
0:35
0:58
0:15
0:08
0:10
0:35
0:35
0:13
0:00
0:27
4:20
11:18
T-value
T (16) ¼ 1.917, po0.05
T (16) ¼ 0.032, po0.05
T (16) ¼ 0.705, p40.05
T (16) ¼ 0.316, p40.05
T (14) ¼ 0.708, po0.05
T (16) ¼ 0.553, p40.05
T (10) ¼ 1.201, po0.05
T (15) ¼ 1.195, po0.05
T (5) ¼ 0.839, p40.05
T (16) ¼ 0.212, p40.05
T (16) ¼ 2.022, p40.05
Null
T (7) ¼ 0.644, p40.05
T (6) ¼ 1.108, p40.05
T (3) ¼ 1.092, p40.05
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
715
Fig. 2. Case 1, error in acquiring items.
Fig. 4. Case 3, error in trading an item.
Fig. 3. Case 2, error in acquiring items.
Fig. 5. Case 4, error in trading an item.
Figs. 4 and 5 reveal many common errors. Fig. 4 shows
subjects dragging the item they wanted to sell from their
inventory to the trading panel. Some subjects (5/16) would
then click on that item repeatedly, reporting that they
continually clicked the item because there was no indication that it had already been sold. Fig. 5 shows some
subjects (5/16) dragging the item to the merchant NPC,
rather than dragging it to the trading panel, or right
clicking on the items in inventory. Such errors suggest that
the interface for trading items is not intuitive or natural.
The other task experiment results are presented in Table 3.
Scope was split into local usability issues and global
usability issues. Local usability issues represent problems
that occur only within a limited range of a product. Global
issues in usability represent overall design flaws. Severity is
divided into the following three levels. If the severity causes
such difficulty that the gamer can no longer play the game,
it is critical level. If it causes the gamer to become confused
and slows the game progress, it is major level. If it causes
problems but does not impede the way a gamer plays the
game, it is minor level (Dumas and Redish, 1999; Barnum,
2002). In the way seriousness was ranged in the postquestionnaire, points below 43 were recorded on 7-point
Likert scale, and a descending order ranking was adopted
in terms of the number of subjects who had been affected
by the seriousness. Usability issues were summarized in
Table 4, based on the task-based experiment and the postquestionnaire.
For brevity, we present only selected recommendations
for solving the most critical usability issues. A description
of all usability issues and recommendations is beyond the
4.3. Eighteen usability issues
Table 4 shows 18 usability issues derived through the
analysis of the post-questionnaire (see Appendix A) and
the experiment based on the 15 tasks during the game play
session.
We identified usability issues, measured their scope and
severity, and appraised their ranking as shown in Table 4.
Scope defines how widely a usability problem is distributed
throughout a product, and severity signifies the degree of
seriousness of the problem (Dumas and Redish, 1999).
3
Values below 4 signify a negative subjective attitude.
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
716
Table 4
Eighteen usability issues
No.
Usability issues
Severity
Scope
Users affected
Issue 1
Issue 2
Acquiring items caused confusion
There were many incongruent actions initiated by right-clicking the mouse
To chat with the NPC
To equip items
Trading items
Movement driven by mouse
Critical level
Critical level
Local
Global
17/18
15/18
13/18
13/18
9/18
Issue 3
Issue 4
There was no clear feedback when trading items
For warriors, there was no detailed instruction or tutorial explaining how to use skills
such as ‘Heroic Strike’ at the beginning of the game
Use of the Holy Grail-shaped icon to access the quest log was not intuitive
It was difficult to build story content
It was difficult to understand one control (Number 1 key) mapping onto two functions
(on/off)
It was difficult to have empathy for his or her character image
There were too many quests that required only hunting the Monster at the game’s
beginning
After identifying the help exclamation point, when the gamers wanted to see the
relevant content again, it was difficult to find that information
It was difficult to learn at the beginning of the game
There were too many hints in the game
It was difficult to read and understand the words presented within the game
It was difficult to change the automatic view
Game setting and key-binding setting were complicated
It was not easy to remember the game’s names of places and people
The item size on the loot panel was smaller than its description area
In the party play, the method of acquiring items was not fair
Critical level
Critical level
Local
Global
13/16
12/18
Critical level
—
Critical level
Global
—
Global
12/18
11/18
10/18
—
—
—
—
9/18
9/18
—
—
8/18
—
—
—
Critical level
—
—
Major level
—
—
—
—
Global
—
—
Local
—
8/18
8/18
8/18
7/12
6/18
6/18
6/18
3/8
Issue 5
Issue 6
Issue 7
Issue 8
Issue 9
Issue 10
Issue
Issue
Issue
Issue
Issue
Issue
Issue
Issue
11
12
13
14
15
16
17
18
Note: Total size o18, and number 16 of ‘13/16’ means that two subjects were missing among total sample size of 18.
scope of this research; we refer the reader to Song and Lee
(2006) for a more detailed account.
Issue 1 in Table 4 was the most critical issue. Most of
the subjects were asked to indicate their level of confusion
related to the execution of item acquisition task. Subjects
indicated that their confusion stemmed from a lack
of confirmation of item acquisition. In order to solve
this problem, the game should provide clear visual
representation about how to obtain items. Otherwise,
novices might give up game play at the beginning of
game. The acquisition of items is critical to game play.
All games including WOW should offer some visual
indication of item acquisition, or for any other task
completion.
Issue 2 highlights the difference between games and
general software products (i.e., MS Office, Web Browsers,
Graphic editing tools, etc.), which often assign special
actions to right clicking. In games, actions such as
movement, conversing with NPC, trade decisions, and
equipping items are not special occurrences, but rather
basic actions that are used frequently. To support such
actions, it would be more appropriate to provide left
clicking as the basic value, rather than right clicking.
Issue 3 showed many instances of the subjects dragging
an item from the inventory to the trading panel, and
clicking on it again only to have it returned to the
inventory. This problem occurred because the subjects did
not have an assurance as to whether the item was sold or
not. In WOW, the sale of an item yields a sound effect
representing the accumulation of money, and the visual
interface shows a tiny number that increases incrementally
with each sale. However, the sound and incremental
number change had no effect on gamers’ confirmation
with respect to trading items, and thus gamers were left to
wonder whether their trade was successful. To fix this issue,
a clear instruction or expression should be provided upon
the completion of a trade. Clear feedback is a critical issue
in the design of any game interface.
4.4. Key factors derived from a new framework of heuristic
evaluation in MMORPG
A process to derive key factors affecting game design
from literature studies and experimental results is presented
in this section. First, the game category was divided
into four subcategories by reviewing literatures related to
HCI and computer gaming by adopting a top-down
methodology. The key factors of each category were then
derived from empirical data by adopting a bottom-up
methodology.
The top-down method to obtain four game subcategories
and the details pertaining to them are first discussed.
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
717
Table 5
Key factors derived from empirical data
No.
Empirical data
Key factors in usability evaluation
Task experiment
Post-questionnaire (See Appendix)
Detailed key factors
Category
Issue 1
Task 1 (Acquire items)
—
Game interface
Issue 2
—
Clear feedback
Minimal control
Affordance of objects
Clear feedback
Adequate expression in help
Usage of metaphors similar to the real world
Building story content as emergent method
Natural mapping
The gamer empathizes through game play
The gamer empathizes with the game character
Building story content as emergent method
Retrievable within the game
Easy to learn at the beginning of the game
Hints presented within the game too numerous
Suitable names
Proper grouping, depth and breadth of menu items
Flexible usage according to Expertise
Suitable names
Proper grouping
Fair play between gamers.
Game
Game
Game
Game
Game
Game
interface
interface
interface
narrative
interface
Play
Game
Game
Game
Game
Game
Game
Game
Game
Game
Game
narrative
interface
play
interface
interface
interface
interface
interface
interface
play
Issue
Issue
Issue
Issue
Issue
Issue
3
4
5
6
7
8
Task
Task
Task
Task
Task
Task
Task
—
Task
—
3 (Chat)
4 (Equip items)
5 (Trade items)
11(Movement)
5 (Trade items)
2 (Skilled-Attack)
6 (Identify quest)
Issue
Issue
Issue
Issue
Issue
Issue
Issue
Issue
Issue
Issue
9
10
11
12
13
14
15
16
17
18
—
Task 6 (Identify quest)
—
—
—
Task7(Adjust Viewpoint)
—
—
Task 1 (Acquire items)
—
10 (First fight)
—
—
—
C3
—
B10, B11
C3
A10
B3, B4
B9
A6
—
A8,A9
A6
—
B12
The categories of the game interface, game play, game
narrative, and game mechanics were included in the
MMORPG usability framework, as shown in Fig. 1.
This kind of structure combines the three categories
asserted by Clanton (1998) and Federoff (2002) with
the game narrative suggested by Jenkins (2002) and
Desurvire et al. (2004). For the game interface, eight
of the 10 heuristic principles suggested by Nielsen (1994)
were adopted, with the exclusion of ‘Error prevention’
(Heuristic 5) and ‘Recovering from errors’ (Heuristic 9).
The two excluded heuristics do not seem to apply to
computer games, but admittedly could aid traditional
software productivity. Niesen’s Heuristics 5 and 9 do
not related to game design because gamer errors are closely
tied to the final consequences of a game in game play.
Game play and game mechanics have been devised based
on the empirical studies of Malone (1980, 1982), who
studied factors triggering fun in computer games, a subject
also contributed to by the game developer Crawford
(1982). Game narrative comprises the four criteria4 of
Jenkins (2002) and some assertions of Desurvire et al.
(2004).
To obtain heuristic evaluation factors through empirical
study, we identified 18 usability issues through the analysis
of task completion and post-questionnaires in Section 4.3.
4
In his article, Jenkins (2002) noted four criteria, being embedded
narratives, evocative space, emergent narrative, and enacting stories; game
design as a narrative architecture.
Game interface
The usability issues were identified to be factors that
preclude a gamer from continuing play or cause a delay in
game play associated with confusion. For example, gamers
may want to quit a game if completion of a task, such as
acquiring items, is continually confusing. Or, if gamers
suffer from frustration in trying to buy and sell items due
to the lack of clear feedback, they may also wish to
discontinue game play. It is essential that factors contributing to game cessation be accounted for during game
design. As such, they must be adopted as usability
evaluation factors (Table 5).
Based on the 18 usability issues, we examined the
relationships among usability evaluation factors as
determined by top-down methods. This examination
showed similarities between the 18 usability issues determined from the bottom-up approach and the usability
evaluation factors set using the top-down approach.
For example, issue 1 (confusion relating to item acquisition) was related to a clear feedback and the minimal
controls during the completion of task 1 (item acquisition).
Issue 2, which related to incongruent right-click results,
was connected to affordance of objects.5 This was derived
from Task 3 (chat with NPC), Task 4 (equip items),
Task 5 (trade items) and Task 11 (movement). Issue 6,
which showed story content to be difficult to build, was
5
Affordance of objects refers to the perceived and actual properties of
things, primarily those fundamental properties that determine just how the
thing could possibly be used (Norman, 1990).
ARTICLE IN PRESS
718
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
correlated to emergent narrative.6 This result was obtained
through the post-questionnaire (C3).7 What’s noteworthy
is that issue 2 (affordance) and issue 7 (natural mapping)8
are newly discovered usability evaluation factors which
could not have been established using a top-down
approach.
The final version of the usability evaluation framework
for MMORPGs was derived using the following approach.
In the initial framework, correspondence analysis was
conducted to determine the relationships between game
subcategories including the game interface, game play,
the game narrative, game mechanics and the 31 usability
factors determined from empirical data. From a statistical
viewpoint, the interface, game play, game narrative,
game mechanics, and 31 usability factors were nominal,
categorical data that could serve as the basis for study
results. That is, it is possible to identify the relationship
between two variables with correspondence analysis of
nominal, categorical data. In all, 30 game designers and
experts with more than 3 years experience in the
MMORPG field were asked to choose appropriate cells9
in matrix comprised of factors and subcategories in
Table 6, which means the correlation between four
subcategories and 31 usability factors. Each cell in Table
6 has the repeated, accumulated frequency. For example,
when we examined the relationship between vividness
factor and four subcategories, the frequency of interface
for vividness factor is 10, the frequency of play
for vividness factor is 3, the frequency of narrative for
vividness factor is 2, and the frequency of mechanics for
vividness factor is 18. This statistic result implies that there
is the highest relevancy between vividness and mechanics,
and the second highest relevancy between vividness and
interface. Moreover, we can find that play and narrative
have low relevancy between them. Table 6 summarizes the
results of the expert survey.
The correspondence analysis was conducted to identify
the relationships among 31 factors and the four game
6
The Sims as a sandbox or dolhouse game, suggesting that it should be
understood as a kind of authoring environment within which players can
define their own goals and write their own stories. Thus, emergent
narrative is not pre-structured or preprogrammed, taking shape through
the game play, yet they are not as unstructured, chaotic, and frustrating as
life itself (Jenkins, 2002).
7
Readers are advised to refer to Appendix A for the post-questionnaire.
8
Mapping is a technical term meaning the relationship between two
things, in this case between the controls and their movements and the
results in the world. Natural mapping, by which it was meant taking
advantage of physical analogies and cultural standards, leads to
immediate understanding (Norman, 1990).
9
Appropriate cells in matrix comprised of factors and subcategories in
Table 6 could be chosen by experts more than twice when they were
confident in the existence of duplicate relationships between the factors
and subcategories. For example, when experts thought that there are two
relationships between a factor (vividness) and two subcategories (Interface, Mechanics), they can respond to chose two times onto 2 cells (i.e.,
first cell means vividness in first column and interface in first row, second
cell means vividness in first column and Mechanics in fourth row)
simultaneously.
Table 6
Frequency between subcategories and factors
Factors
Subcategories
Interface Play Narrative Mechanics Active
Margin
Vividness
10
Reward
11
Replay
7
Recog.
20
Physic
10
Percept.-motor skill
14
Pace
3
Natural mapping
19
Modeless
7
Match metaphor
17
Learn
10
Interaction with narr.
3
Immed. display
12
Help
18
Goal
1
Flexibility
19
Feedback
21
Fair
1
Evocative sp.
1
Enacting st.
2
Empathy
1
Emergemt narr.
2
Embedded narr.
1
Difficulty
2
Curiosity
1
Control
19
Consistency
17
Challenge
1
Balance
3
Affordance
21
Aesthetics
19
Active margin
293
3
14
18
11
9
12
21
11
9
9
19
9
4
9
18
9
11
16
4
8
22
9
11
24
6
6
10
16
23
1
9
361
2
6
12
1
1
2
9
1
17
2
2
22
1
7
10
8
3
13
24
20
8
19
18
7
25
3
1
14
11
1
1
271
18
1
3
3
16
9
4
8
2
1
1
1
21
1
1
2
3
1
1
2
4
1
1
1
2
11
1
1
1
8
5
135
33
32
40
35
36
37
37
39
35
29
32
35
38
35
30
38
38
31
30
32
35
31
31
34
34
39
29
32
38
31
34
1060
Note: Appropriate cells in matrix comprised of factors and subcategories
in Table 6 could be chosen by experts more than twice when they were
confident in the existence of a relationship between the additional choices.
For example, when experts thought that there are two relationships
between a factor (vividness) and 2 subcategories (Interface, Mechanics),
they can respond by selecting two 2 (i.e., while the first cell means
vividness in the first column and interface in the first row, the second cell
means vividness in the first column and Mechanics in the fourth row)
simultaneously.
Therefore, we can examine the relationship between vividness factor and
four subcategories. According to frequency between the first column
(vividness factor) and four subcategories in Table 6, the frequency of
interface for vividness factor is 10, the frequency of play for vividness
factor is 3, the frequency of narrative for vividness factor is 2, and the
frequency of mechanics for vividness factor is 18. This statistic result
implies that there is the highest relevancy between vividness and
mechanics, and the second highest relevancy between vividness and
interface. Moreover, we can find that play and narrative have low
relevancy between them. Table 6 summarizes the results of the expert
survey.
subcategories. Fig. 6 shows the 31 factors for the four
subcategories using the coordinate system method. Fig. 6 is
a location map of each variable category computed by the
score of the row variable and the column variable.
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
719
review of literature relevant to computer game design and
HCI. We identified 54 key factors for game design
and broadly divided them into four subcategories, being
‘Game Interface’, ‘Game Play’, ‘Game Narrative’, and
‘Game Mechanics’. ‘Game Interface’ includes such
factors as feedback, metaphor, control, consistency,
recognition, flexibility, esthetics & minimalist design, help,
affordance, and natural mapping. Game Play involves
factors such as goals, learning, rewards, challenges, pace &
pressure, re-playability, empathy, fairness, difficulty, balance, perceptual-motor skill. ‘Game Narrative’ consists of
embedded narratives, evocative space, emergent narratives,
enacting stories, interaction between gamers and the
narrative, curiosity, and modeless. ‘Game Mechanics’
comprises factors including immediate display, physics,
and vividness. This study offers a new method for the field
of game design that may aid in the creation of more usable,
fun games.
5.2. Discussions and implications
Fig. 6. Correspondence analysis results.
According to this location map, game interface is closely
related to feedback, metaphor matching, control, consistency, recognition, flexibility, esthetics, help, affordance,
and natural mapping. On the other hand, game play is
strongly related to game goals, learning, reward, challenge,
pace, re-playability, empathy, fairness, difficulty, balance,
and perceptual-motor skill. Game narrative is strongly
related to embedded narratives, evocative space, emergent
narratives, enacting stories, interaction between the gamer
and the narrative, curiosity, and modeless.10 Game
mechanics were closely related to immediate display,
physics, and vividness. From the correspondence analysis,
it was possible to obtain 31 factors in four subcategories. In
addition, feedback, control, consistency, recognition, flexibility, esthetics, and help were subclassified based on the
heuristics studies of Nielsen (1994). In the end, 54 usability
evaluation factors were devised from the 31 usability
factors under the four subcategories (see Table 7).
Consequently, the final version of the heuristic evaluation
framework for MMORPGs was developed by effectively
using both top-down and bottom-up approaches.
5. Conclusion
5.1. Summary of the results
This study presents key factors for game design. We
derived key factors through a usability evaluation and a
10
If an application is modeless, it allows the user to do whatever they
want when they want without being restricted to certain features in certain
modes.
The limitations of this study stem from cultural factors.
This study was conducted in a distinctly Korean
setting, which is relevant to the results. For example, the
subjects did not naturally maintain empathy for their
character images (Issue 8) because Korean gamers
tend to prefer characters of a traditionally beautiful
appearance, such as those in Lineage 2 published in
NC soft, Korea. According to McGregor (2006),
Lineage 2 received 10 of 10 points for graphic quality
in a MMORPG.COM review. It was also difficult
for subjects to remember place and person names
(Issue 16) because Korean novices were not familiar
with WOW game terminology (see Table 4). Therefore,
our future works will include a more elaborate
experimental design that is capable of handling such
cultural complexities.
Despite the limitation, this research offers both strong
theoretical and practical contributions. The theoretical
contributions of this study relate to the new framework for
heuristic evaluation in MMORPG games, which is an
objective method for reducing obstacles to fun, rather
than obstacles to accomplishment (as in productivity
applications). This research confirms and extends the
current line of computer game usability research.
Previous usability researches have focused on efficiency,
ease to learn, and ease to use because they have been
concerned with product performance in terms of traditional usability. This research extends the current usability
framework by examining the fun of a game, rather than
just productivity.
In terms of practical application, this research offers 54
key factors derived from the study for heuristic evaluation
in MMORPG games, as first examined by Desurvire et al.
(2004). As discussed in Section 2.1, current computer
game evaluation methods are limited to surveys, focus
groups, beta-testing, and other traditional user-centered
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
720
Table 7
Fifty-four key factors derived from a new framework of heuristic evaluation in MMORPGs
Category
Key factors
I. Game interface
A. Feedback of the current status
Source
1. Clear feedback
2. Adequate feedback
Nielsen’s Heuristic 1 (1994),
Malone (1980, 1982):
Shneiderman (1992), Norman
(1990)
Issue 1
B. Metaphor matching
3. Usage of metaphors similar to the real world
Nielsen’s Heuristic 2 (1994),
Issue 5
C. Control and freedom
4. Immediate control
5. The default of control
6. The degree of freedom
Nielsen’s Heuristic 3 (1994),
Bickford (1997)
D .Consistency and standards
7. Consistent name, information, structure, and
expressions
8. The trends set by the game community
9. Consistent in control, color, typography, and
dialog design
Nielsen’s Heuristic 4 (1994),
Sanchez-Crespo Dalmau (1999)
E. Recognition rather than recall
10.
11.
12.
13.
14.
15.
Retrievable within the game
Suitable names
Proper grouping
A visual hierarchical structure
Clear visual distinctions
Proper depth and breath of menu
Nielsen’s Heuristic 6 (1994),
Norman (1990),
Issue 10;
Issue 13;
Issue 14;
Issue 16;
Issue 17
F. Flexibility and efficiency of use
16.
17.
18.
19.
20.
Flexible usage according to expertise
Options
Personalization
Quick performance methods
Automatic performances
Nielsen’s Heuristic 7 (1994),
Issue 15
G. Aesthetic and minimalist design
21. Aesthetic appreciation
22. Minimal expression
23. Minimal control
H. Help
24. The adequate help content
25. Adequate expression in help
26. Hints presented within the game not too
numerous
Nielsen’s Heuristic 8(1994),
Issue 1;
Issue 7;
Issue 8;
Issue 15
Nielsen’s Heuristic 10 (1994),
Issue 4;
Issue 10;
Issue 12
I. Affordance of objects
27. Primarily fundamental properties that
determine just how the thing could possibly be
used
Norman (1990),
Issue 2
J. Natural mapping
28. The relationship between two things, in this
case between the controls and their movements
and the results in the world
Norman (1990),
Federoff (2002),
Issue 7
II. Game play
K. Goal
29. A clear over-riding goal of the game at the
beginning of the game
Clanton (1998),
Malone (1982)
L. Learning
30. Multiple goals on each level
31. Easy to learn at the beginning of the game.
32. Hard to master the game
Crawford (1982)
Issue 11
M. Reward
33. Rewards that are more deeply immersed in the
game
Bickford (1997)
N. Challenge
34. Keep playing more in spite of high difficulty in
the game
Myers (1990)
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
721
Table 7 (continued )
Category
Key factors
O. Pace and
Pressure
P. Re-playability
Source
35. Pace to apply adequate pressure to the gamer
without discouraging them
Clanton (1998),
36. It should play the game again
Shelley (2001),
Shelley (2001)
Q. Empathy
37. The gamer empathizes through game play
38. The gamer empathizes through game character
Desurvire et al. (2004)
Issue 8
R. Fairness
39. The fair play between gamers
Clanton (1998),
Issue 18
S. Difficulty
40. Various difficulty levelsn according to each
gamer’s level
Malone (1980, 1982):
T. Balance
41. Game play should be balanced with Multiple
ways to win
Crawford (1982),
Malone (1980, 1982):
III. Game
narrative
U. Perceptual-motor skill
42. Behaviors are quick and well-timed responses
to changes and threats in the environment
Pagulayan et al. (2003)
V. Embedded
43. The story should be adequately laid on the
space
Jenkins (2002)
44. Evoking the materials of other famous works,
such as the Lord of ring, warcraft, or star war etc.
Jenkins (2002)
X. Emergent
narrative
45. Building story content as emergent method
Jenkins (2002),
Shelley (2001),
Issue 6;
Issue 9
Y. Enacting stories
46. The short stories reaffirm gamer interest after
they have become bored with general game play
Jenkins (2002)
Z. Interaction between gamer and
narrative
47. Discovering the story as part of game play
Desurvire et al. (2004)
AA. Curiosity
48. Maintaining curiosity regarding story outcome
Desurvire et al. (2004)
BB. Modeless
49. Perceived as modeless even if the game cannot
be modeless
Federoff (2002)
CC. Immediate display
50. Feedback given immediately to display
gamer’s control
Malone (1980, 1982):
DD. Physics
51. Feeling natural mechanics
Federoff (2002)
EE. Vividness
52. Correcting weight and momentum.
53. Unpredictable behaviors of Monster or NPCs
54. Unexpected game outcomes
narratives
W. Evocative space
IV. Game
mechanic
usability evaluation methods that are employed after
product development. However, framework developed
herein has the potential to identify critical problems
with game design, even when applied at early stages
of the design process. Therefore, this research extends
current computer game evaluation methods. For
example, the possibility of immediate feedback will
enable game designers to improve the quality of the
game content quickly, without resorting to the complex
procedures and longstanding computer game evaluation methods. The results of this study have clear
Crawford (1982),
Bickford (1997),
Malone (1980, 1982):
implications for overall game development, and should
serve to improve game quality at any stage of the design
process.
This study examined subjects playing WOW at a
specified game entry point with a specified character class
for approximately 2 h. Consequently, the study results
cannot be extended to the explanation or evaluation of
‘Deep Play’ or game play with other races or classes.
Future researches will include an assessment of key factors
for late-game design based on a usability evaluation game
play in the latter stages of the game.
ARTICLE IN PRESS
722
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
Appendix A. Post-questionnaire for game interface, game play, and game narrative
See Table A1 for further details.
Table A1
Post-questionnaire for game interface, game play, and game narrative
List
Post-questionnaire
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
Is the visual feedback, such as their status/score, and visual cue clearly presented?
Is the audio feedback, such as their NPC’s voice, and audio cue adequately provided?
Does it help that interactions with other characters or targets are analogous or similar to real-world situations?
Do you feel in control of the actions occurring on the screen?
Are consistent Names/Information/Structure/ Representation being provided?
Are instructions for the system retrievable within the game?
Is the flexibility provided with variable difficulty level?
Are game control tools and the on-screen interface unobtrusive and conducive to easy access to the game environment?
Are game controls and on-screen interface simple to provide easy access to the game environment?
Is help needed to engage in game play displayed through a tutorial?
B1
B2
B3
B4
B5
B6
B7
B8
B9
B10
B11
B12
B13
B14
B15
B16
B17
B18
Does it provide a clear overriding goal of the game at the beginning of the game?
Are there multiple goals on each level?
Is it easy to learn at the beginning of the game?
Is it hard to master the game?
Are there rewards for becoming more deeply immersed in the game?
Do you want to keep playing more in spite great game difficulty?
Does the game pace apply adequate pressure without discouragement?
Does it make the gamer want to play the game again?
Are the hints presented within the game not too numerous?
Does it make the gamer empathize with the character during game play?
Does it make the gamer empathize through game character?
Is the play between gamers fair?
Are the behaviors of Mob or NPCs predictable?
Are the outcomes of the game not predictable?
Does it provide variable difficulty levels that are suited to all levels of gamers?
Is game play balanced with multiple ways to win?
Is it perceived as modeless even if the game cannot be modeless?
Are you taught skills early that you expect to use later?
C1
C2
C3
C4
C5
C6
Is the story evenly distributed throughout the game?
Can you evoke the materials of other famous works, such as the Lord of Rings, Warcraft, Star Wars and so on?
Can you build story content via the emergent method?
Do the short stories cause you to become refocused on the game after becoming bored with game play?
Can you discover the story as part of game play?
Is the storyline designed so that it may create curiosity?
Note: A1–A10—game interface part; B1–B18—game play part; and C1–C6—game narrative part.
References
Armstrong, L., Hamm, S., 1997. CD-ROMs: The Giants Rule. Business
Week. The McGraw-Hill Companies Inc., New York.
Barnum, C.M., 2002. Usability Testing and Research. Pearson Education Inc.
Bickford, P., 1997. Interface Design: The Art of Developing Easy-to-Use
Software. Academic Press, Chestnut Hill, MA.
Bradburn, N.M., Sudman, S., 1988. Polls and Surveys: Understanding
What they Tell Us. Jossey-Bass, San Francisco.
Clanton, C., 1998. An interpreted demonstration of computer game
design. In: Proceedings of the Conference on CHI 98 Summary:
Human Factors in Computing Systems, pp. 1–2.
Cornett, S., 2004. The usability of massively multigamer online roleplaying games: designing for new users. In: Proceedings of the Conference
on CHI 2004, vol. 6(1), pp. 703–710.
Crawford, C., 1982. The Art of Computer Game Design. Retrieved from
/http://www.vancouver.wsu.edu/fac/peabody/game-book/Coverpage.
htmlS.
Desurvire, H., Caplan, M., Toth, J.A., 2004. Using heuristics to evaluate
the playability of games. In: Proceedings of the Conference on CHI
2004, pp. 1509–1512.
Dumas, J.S., 2003. User-based evaluations. In: Jacko, J., Sears, A. (Eds.),
Handbook for Human–Computer Interaction in Interactive Systems.
Erlbaum, London, pp. 1093–1117.
Dumas, J.S., Redish, J.C., 1999. A Practical Guide to Usability Testing,
revised ed. Intellect Ltd., Great Britain.
Entertainment Software Association, 2005. Essential Facts About the
Computer and Video Game at E3.
Ericsson, K.A., Simon, H.A., 1993. Protocol Analysis. MIT Press,
Cambridge, MA.
ARTICLE IN PRESS
S. Song, J. Lee / Int. J. Human-Computer Studies 65 (2007) 709–723
Fabricatore, C., Nussbaum, M., Rosas, R., 2002. Playability in action
videogames: a qualitative design model. Human–Computer Interaction 17 (4), 311–368.
Federoff, M.A., 2002. Heuristics and usability guidelines for the creation
and evaluation of fun in video games. Master Thesis, Indiana
University, Bloomington.
Fulton, B., Medlock, M., 2002. Beyond psychological theory: getting data
that improve games. In: Proceedings of Game Developers’ Conference,
2002.
Jenkins, H., 2002. Game design as a narrative architecture. In: WardipFruin, N., Harrigan, P. (Eds.), First Person: New Media as Story,
Performance, and Game. The MIT Press, Cambridge, MA.
Kantner, L., Rosenbaum, S., 1997. Usability studies of WWW Sites:
heuristic evaluation vs. laboratory testing. In: Proceedings of
the 15th Annual International Conference on Computer
Documentation Table of Contents, Salt Lake City, Utah, USA,
pp. 153–160.
Kasavin, G., 2006. Here is the Online Role-Playing Game You Should
Play, No Matter Who You Are. Gamespot.com.
Kotler, P., 1994. Marketing Management, Analysis, Planning, Implementation, & Control, eighth ed. Prentice-Hall, Englewood Cliffs, NJ.
Krueger, R.A., 2000. Focus Groups: A Practical Guide for Applied
Research. Sage, Thousand Oaks, CA.
Malone, T.W., 1980. What makes things fun to learn? A study of
intrinsically motivating computer games. Ph.D. dissertation, Department of Psychology, Stanford University.
Malone, T.W., 1982. Heuristics for designing enjoyable user interfaces:
lessons from computer games. In: Thomas, J.C., Schneider, M.L.
(Eds.), Human Factors in Computing Systems. Ablex Publishing
Corporation, Norwood, NJ.
Manninen, T., 2004. Rich interaction model for game and virtual
environment design. Ph.D. Thesis, Oulu University Press, Oulu,
Finland.
McGregor, C., 2006. MMORPG.COM. Retrieved from /http://www.
mmorpg.com/gamelist.cfm/setView/ratings/setRView/none/gameID/46S.
Medlock, M.C., Wixon, D., Terrano, M., Romero, R., Fulton, B., 2002.
Using the RITE method to improve products: a definition and a case
study. In: Proceedings of Usability Professional’s Association.
723
Myers, D., 1990. A Q-study of game player aesthetics. Simulation &
Gaming 21 (4), 375–396.
Nielsen, J., 1993. Usability Engineering. Academic Press Inc., New York, NY.
Nielsen, J., 1994. Heuristic evaluation. In: Nielsen, J., Mark, R.L. (Eds.),
Usability Inspection Methods. Wiley, New York.
Norman, D., 1990. The Design of Everyday Things. Doubleday,
New York.
Pagulayan, R.J., Keeker, K., Wixon, D., Romero, R., Fuller, T., 2003.
User-centered design in games. In: Jacko, J., Sears, A. (Eds.),
Handbook for Human–Computer Interaction in Interactive Systems.
Lawrence Erlbaum Associates, Inc., Mahwah, NJ, pp. 883–906.
Sanchez-Crespo Dalmau, D., 1999. Learn faster to play better: how to
shorten the learning cycle. Retrieved from /http://www.gamasutra.
com/features/20010815/shelley_01.htmS.
Shelley, B., 2001. Guidelines for developing successful games. Retrieved
from /http://www.gamasutra.com/features/20010815/shelley_01.htmS.
Shneiderman, B., 1992. Designing the Human Interface: Strategies for
Effective Human—Computer Interaction. Addison-Wesley Publishing
Company, Inc, Reading, MA.
Song, S., Lee, J., 2006. An exploratory study of the critical design
factors for MMO game design based on usability evaluation. In:
Proceedings of the Conference on HCI2006 Metamorphosis, vol. 2,
pp. 131–139.
Sudman, S., Bradburn, N.M., Schwarz, N., 1996. Thinking About
Answers: The Application of Cognitive Processes to Survey Methodology. Jossey-Bass, San Francisco.
Van Someren, M.W., Barnard, Y.F., Sandberg, J.A.C., 1994. The Think
Aloud Method: A Practical Guide to Modeling Cognitive Processes.
Academic Press Prof., Inc., San Diego, CA.
Wagner, J.A., 2002. Showdown in cyberspace: star wars vs. the sims.
Salon.com. Retrieved from /http://www.salon.com/tech/feature/2002/
07/09/mmorpg/index.htmlS.
Woodcock, B.S., 2003. An analysis of MMOG subscription growth.
Retrieved from /http://pw1.netcom.com/sirbruce/Subscriptions.
htmlS.
Yee, N., 2003. Gender and age distribution. The Daedalus Project 1–2.
Retrieved from /http://www.nickyee.com/daedalus/archives/000343.
phpS.