Utah State University
DigitalCommons@USU
ITLS Faculty Publications
Instructional Technology & Learning Sciences
10-2008
Problem-Based Educational Games: Connections,
Prescriptions, and Assessment
Andrew Walker
Utah State University
Brett E. Shelton
Utah State University
Follow this and additional works at: http://digitalcommons.usu.edu/itls_facpub
Part of the Educational Assessment, Evaluation, and Research Commons, and the Instructional
Media Design Commons
Recommended Citation
Walker, A., & Shelton, B. E. (2008). Problem-based educational games: Connections, prescriptions, and assessment. Journal of
Interactive Learning Research, 19(4), 663-684.
This Article is brought to you for free and open access by the Instructional
Technology & Learning Sciences at DigitalCommons@USU. It has been
accepted for inclusion in ITLS Faculty Publications by an authorized
administrator of DigitalCommons@USU. For more information, please
contact [email protected].
Problem-Based Educational Games: Connections,
Prescriptions, and Assessment
Andrew Walker
[email protected]
Brett E. Shelton
[email protected]
Utah State University
Manuscript Accepted for Publication, Journal of Interactive Learning Research
Abstract
The overwhelming success of the commercial game market has brought increased attention to
emerging work in educational game design. Much of the existing work in educational games a
strong similarity to the field of Problem-Based Learning (PBL), which has a rich history of
conceptual literature as well as empirical investigations. Despite apparent similarities between
the two fields, there has been no formal effort to explore the connections between them. This
conceptual paper examines the basic tenants of PBL with an eye toward making prescriptive
recommendations for the design and use of problem-based educational games. Examples within
existing educational games are discussed in the context of PBL features and outcomes.
Introduction
PBL originated at McMaster University in the late 1960s as a response to low
enrollments and general dissatisfaction with medical education (Barrows, 1996). This modern
Walker & Shelton 2
inception of PBL came soon after a North American revival of Montessori’s work and has a
strong relationship to the progressive education movement of the early 20th century espoused by
Montessori, Dewey, Neill, Steiner, and Kilpatrick (Rohrs & Lenhart, 1995) among others. That
said, it arose primarily out of a perceived need for medical education reform.1 Specifically,
students felt highly motivated as residents overcoming patient problems, but struggled in the two
preceding years of largely rote information with few apparent ties to their future practice
(Spaulding, 1969). Briefly, PBL is characterized as an approach to learning in which students
are given more control, asked to work in small groups, and most importantly acquire new
knowledge only as a necessary step in solving authentic problems representative of eventual
professional practice (Barrows, 1986). Since its modern inception in medical education a robust
interest in PBL has resulted in use across several disciplines (Savery & Duffy, 1995) and
educational levels (Savery, 2006) including: Business (Arts, Gijselaers, & Segers, 2002; Breton,
1999), economics (Garland, 1995; Son & Van Sickle, 2000), educational psychology
(Steinkuehler, Derry, Hmelo-Silver, & Delmarcelle, 2002), engineering (Van Treuren &
Havener, 1997), physics (Mestre, Dufrense, Gerace, & Hardiman, 1993), and social work and
sociology (Heycox & Bolzan, 1991; Rangachari, 1996; Ross & McNeil Hurlbert, 2004) among
others. More recently, there is a rising interest in educational computer games. This is due in
part to the popularity and commercial success of games with those under the age of eighteen.
Indeed, the video game industry is responsible for over 30 billion dollars in income, annually
outpacing the domestic and foreign proceeds for motion pictures (Macklin, 2006). Educators
have renewed their attempts to capture the attention of students by leveraging what games have
to offer in learning situations. The emerging philosophy and research in educational game
design has a parallel relationship to the underlying structure of PBL, even while no explicit
Walker & Shelton 3
derivations are cited. For example, Squire (2002), Paras & Bizzocchi (2005) and Shelton
(2007a) make a call for externally representing the connections between learning goals and game
play in educational game design. This parallels PBL, which calls for a strong relationship
between learning goals and the problems presented to learners (Barrows, 1985). Yet no work to
our knowledge explicitly addresses the use of PBL in the context of educational games or
attempts to map the relationship between educational game design and the structures of PBL. To
that end, the purpose of this work is to explore the features of PBL as well as the intended
outcomes of the approach in the context of educational game design. Examples within existing
games will be discussed and a rubric for the assessment of PBL alignment in educational games
will be presented.
Problems with Educational Game Design
Certainly, there exist numerous suggested approaches to educational game design,
including those that originate from computer game theory (Crawford, 1984; Prensky, 2001),
educational technology (Rieber, 1992, 1996; VanEck, 2006, June), instructional design
(Appelman, 2005; Kirkley, Tomblin, & Kirkley, 2005), motivation theory (Csíkszentmihályi,
1997; Malone & Lepper, 1987), learning sciences (Barab, Thomas, Dodge, Carteaux, & Tuzan,
2005; Dede, Salzman, Loftin, & Sprague, 1999; Squire, 2002), and business and industry
(Aldrich, 2005). The following discussion of the challenges that educational game designers face
does not adequately represent the field or all of the existing paradigms, and it is not meant to.
Rather, these issues are one node of interest to theorists and designers of a nascent field of
instruction.
The broad categorizations of what constitutes “computer games” and more generally
computer simulations for learning are not debated in this article. Instead, a variety of computer-
Walker & Shelton 4
based applications that include game-like elements and those that simulate real-world activity
and environments are considered. Although these encompass a wide spectrum of work they are
consistently referred to as educational games here. Differences between educational games and
simulations exist including genre, intended audience, overall design approach and desired
learning outcome. However, it is through examining both educational games and simulations that
contributions can be made to the design and evaluation of educational activities. Given this
broad definition, the following paragraphs discuss the three approaches that designers have taken
when considering educational computer games.
One approach mirrors closely that of commercially successful computer games which
motivate players by offering the challenge, interest, and gratification common to traditional
video games. These games are often linked to conquer, chase, and save-the-city kinds of
objectives that one experiences in Galaga, Pac-Man, or Missle Command. The modifications for
instructional purposes, then, come in the form of rewards or reinforcement removal. Such would
be the case of an electronic flash-card type of game that offers or takes away coins, points, or
some other extrinsic reward to reinforce correct behaviors or answers and weaken incorrect ones.
Generally, these games can be successful in their objectives, but are linked to less complex,
memorization based tasks or motor-based skills, such as learning the multiplication tables or
typing practice (Kirriemuir, 2003).
A closely associated approach involves taking commercially successful games and
building instructional lessons around the traditional game play, without modifying the game
itself. An example of this would be creating tasks for students within Sim City that help them
learn city planning, money management, or leadership skills. This approach has proven less
useful for learning purposes in part because it places a heavy burden on the instructor to guide
Walker & Shelton 5
students through relevant tasks and assess their progress (Kirriemuir & McFarlane, 2003). This
technique is also difficult to monitor, given the inherent nature of sim-games being largely
random and unpredictable, creating less control for the instructor and more opportunity for offtask behavior (Squire, Barnett, Grant, & Higginbotham, 2004).
A third approach has the opposite problem. Some educational computer games are
created with specific instructional objectives in mind. They have a progression, scaffolding, and
built-in assessments that focus students on more complex learning issues aligned with
meaningful goals. However, these games often have difficulty maintaining engagement with the
learner. Students have found games of this nature to be boring or unfulfilling (Kirriemuir, 2002;
Kirriemuir & McFarlane, 2003). Many corporate and training simulations are often created in
this way, with the understanding that the audience has other motivations in which to engage with
the simulation (e.g., career advancement, machinery upgrades, software updates, or skill
certifications). These simulations therefore need not incorporate game-like elements, and are
often aimed at audiences that are not motivated by computer games anyway.
A suggested approach to address these design issues has been to design educational
computer games with the constraint that all game play activity embedded within the game be
aligned with the instructional objectives (DeCastell & Jenson, 2003; Shelton, 2005a; Shelton,
2007b). The scenario around which the game is built can be engineered to balance authenticity
with alignment to the educational task and objectives. The game uses simulated scenarios that
are representative of the real-world, and the activity itself is useful for progression toward and
designed around a learning goal. It is this approach that allows for alignment between PBL and
educational game design. By articulating the relationship between designing educational
computer games and PBL a rubric emerges that can 1) aid in the design of problem-based
Walker & Shelton 6
educational games and 2) help establish a protocol for evaluating existing games that may or
may not be useful for the intended learning objectives.
PBL Characteristics
PBL involves 1) authentic problems that are the vehicle for acquisition of procedural and
content knowledge, 2) learner-centered activities, and 3) small group interactions that are 4)
facilitated by teachers (Barrows, 1986). Each of these four important characteristics will be
described in more depth below alongside examples of these characteristics from educational or
commercial games.
Authentic Problems
Intuitively enough, authentic problems are the central focus of instructional activities in
PBL. Jonassen (2000) defined a problem as the discrepancy between an initial state and a
resolved state. This aligns fairly well with the work of Barrows (2002) who advocated for
problems to be presented to the learner in unresolved form. It is the learner’s attempts to bring
these problems to meaningful resolution that result in both conceptual as well as procedural
understanding. Although Jonassen presents a useful continuum of problem types along
dimensions such as structuredness, complexity, and level of abstraction (2000) Barrows focused
in on problems that are somewhat ill-structured, involve multiple solution paths, and reflect the
kinds of problems encountered in professional life (Barrows, 1986, 1996; 2002). Ideally, the
initial state is far enough removed from the resolved state that the situation demands an
examination of multiple solution paths using free inquiry, an investigation of the problem as well
as the acquisition of and application of knowledge to resolve it. Problems are designed, selected,
and sequenced in a way that guides student learning (Barrows, 1986). They become a vehicle for
Walker & Shelton 7
attaining content knowledge (Savery & Duffy, 1995) that crosses several different disciplines
(Savery, 2006) as well the process knowledge needed to employ it (Barrows, 1996).
An example of presenting authentic problems can be found in the educational game
Quest Atlantis (Center for Research on Learning & Technology, 2000a). Players “go questing"
by traveling through virtual environments to identify and finish tasks designed to create
opportunities for learning in multiple ways.
Figure 1 - Quest Atlantis
Quest Atlantis includes interaction within 11 different worlds that each encompasses a
different theme of citizenship. Each world has 3 villages that address variations of the world's
theme by offering a large number of individual quests. Quests may embody activity in the virtual
world or the real world, but each exercise is tied to the citizenship theme of the visited village.
Villages include quests that address ecology, culture, power and health. The design of Quest
Atlantis offers support for children “in developing their own sense of purpose as individuals, as
members of their communities, and as knowledgeable citizens of the world” (Center for
Research on Learning & Technology, 2000b, para.1) Although the computer-based environments
that form the setting for Quest Atlantis missions may not be authentic, the problems and
activities within the environment are quite authentic. The designers identified and embedded
Walker & Shelton 8
seven dimensions, such as social responsibility and environmental awareness, to help children
achieve the goals listed above. It is our contention that the crucial element in authenticity is the
alignment between the educational task and eventual practice. If the learning environment
introduces abstraction that supports the authentic nature of the task then the authenticity
requirement is met. In this case, children may be reluctant to explore issues of social
responsibility in a real village with real people. Still, abstractions to the environment necessitate
a bridge to more authentic environment. In the case of Quest Atlantas, the developers also offer
teaching supports in the form of downloadable documents, which allow students to take their
game experiences home for additional work (Barab, 2006). In this way the missions within the
game serve as a jumping-off point for students to take what they learned from interactions within
the game context and apply them to their everyday lives.
Learner Centered
PBL is a self directed (Savery & Duffy, 1995) or learner centered approach to education.
Specifically, learners take on primary responsibility for examining a problem and then
determining what they know and what they need to learn in order to solve it. Note that
instructors still have a responsibility for defining learning objectives, but these are mostly kept
from students who generate their own learning objectives in response to a given problem
(Barrows & Tamblyn, 1980; Savery & Duffy, 1995). The next crucial step is locating
appropriate resources to fill in their knowledge gaps or investigate areas of personal interest
(Barrows, 1996). Resources are broadly defined as content experts, reference books, journal
articles, electronic sources and the like (Barrows, 1996, 2002).
In order for this to happen,
problems need to be presented prior to any pursuit of knowledge (Barrows, 1986) since a lecture
based approach relies on the instructor to decide what students need to learn. Barrows claims
Walker & Shelton 9
that instruction prior to presenting problems comes at a cost. Most notably it undermines the
learner's control and may also prevent content knowledge from being structured to aid in
problem solving (1986). As a final step, learners are responsible for self-assessing and peer
assessing problem solving performance (Barrows, 2002; Savery, 2006; Savery & Duffy, 1995).
An example of learner centered design can be found in the Virtual Reality Gorilla Exhibit
(Learning & Performance Support Laboratory, 2000), which teaches users about gorilla
behaviors and social interactions (Allison, Wills, Bowman, Wineman, & Hodges, 1997). The
Georgia Institute of Technology's Virtual Environments Group and Zoo Atlanta put students into
the gorilla's environment in a way that supports an authentic experience but without the
dangerous repercussions. The program, which allows students to roam in a simulated natural
gorilla habitat, also simulates certain types of interactions with the animals. The goal of the
project is to educate and help understand how gorillas function in their native environment and
with other gorillas (Walton, 1996).
Figure 2 – Virtual Gorilla
Activity centers on student movements and actions with the virtual system, acting as
gorillas from a first-person perspective. The construction of students’ own knowledge is built
through the students’ game-like activity. The creators of virtual gorilla have taken great pains to
Walker & Shelton 10
implement real gorilla-like behavior in the environment so that students, acting as a gorilla in the
environment, understand what it is like in the habitat. The gorillas in the environment exhibit
behaviors and actions extremely close to those in the natural environment, and it is up to the
student to determine how to satisfy the needs of the beast. These needs include issues of survival,
such as hunger and shelter, as well as group hierarchy and interaction with other gorillas. By
searching and identifying the needs for gorilla survival, students must identify the gaps in their
current understanding of what drives gorilla behaviors and attempt to resolve these issues
through the investigation of cause-and-effect relationships in the environment. By reflecting on
their newly acquired knowledge, students learn what it means to be a part of a social group that
would otherwise be unrecognizable to them (Hay, 2006).
Teachers as Facilitators
Rather than disseminating knowledge, PBL instructors act primarily as facilitators, a role
shift that can be quite challenging (Ertmer & Simons, 2006). Their focus is to serve as a
metacognitive aid, modeling the kinds of questions students need to ask themselves in order to
solve the problem (Barrows, 1996; Savery & Duffy, 1995). These questions correspond with the
activities discussed above under the definition of learner centered. For instance, facilitators
might ask students what their knowledge limitations or gaps are, where they could go to fill
them, how the knowledge they have gleaned will help them, or how effective the group was in
solving the problem (Barrows, 2002; Ertmer & Simons, 2006). In order to have the most impact,
facilitators need to be content experts and they need to be formally trained as facilitators (Moust,
Grave, & Gijselaers, 1990). In large part this training is necessary so that facilitators do not
engage in lectures during PBL sessions, or correct the lines of reasoning that students engage in.
Rather facilitators assure participation from all students, focus the discussion on the problem,
Walker & Shelton 11
make students’ knowledge transparent, and encourage self-directed learning (Ertmer & Simons,
2006). Over time, the goal of a facilitator is to be replaced by students—slowly fading their
metacognitive prompts and letting a student issue the same kinds of prompts to the group
(Barrows, 1996).
Examples of facilitation in educational games are rare, but one is the Multi-User Virtual
Environments or MUVEES (Harvard University, 2003). Within MUVEES, student action is
mediated by a progression of data gathering and analysis tasks within the environment. Progress
is monitored by an instructor who provides coaching. MUVEES are designed to be an engaging
way to improve educational outcomes using multimedia and virtual environments for teaching
and learning science. MUVEES enables multiple simultaneous participants to:
•
access virtual architectures configured for learning
•
interact with digital artifacts
•
represent themselves through graphical "avatars"
•
communicate both with other participants and with computer-based agents, enacting
collaborative learning activities
Figure 3 – MUVEES River City
Walker & Shelton 12
As part of the project data were collected from both students and teachers (Nelson et al.,
2007). Interestingly, the students who played the game indicated that they perceived the role of
their teachers as being more “facilitator-like” than those who did not play the game, in that they
did not press them for their understanding. This suggests that students who played the games
may have assumed more responsibility for their own learning. Further, the study indicated the
teacher prompts tended to decrease over time, indicating a desirable shift towards increased
student responsibility over time.
Small Group Interaction
PBL is typically conducted in a series of small group interactions with 5-9 students
(Barrows, 1996; Savery & Duffy, 1995). This can be considered a form of collaborative
independent study. After agreeing on the set of knowledge gaps collaboratively, group members
distribute and then tackle the task of filling them independently. Collaboration then comes back
into play when the knowledge is shared and applied to the problem (Savery, 2006). Group
sessions also provide an opportunity to reflect on and critique the utility of any acquired
knowledge. When a series of conceptually related problems is complete students are placed in
new groups so they are exposed to a wide variety of future colleagues. Although there are
certainly PBL implementations involving larger groups (Farquhar, J., & K., 1986; Jones, Beiber,
Echt, Scheifley, & Ways, 1984; Polglase, Parish, Buckley, Smith, & Joiner, 1989; Rangachari,
1996) the commonly accepted practice is to keep the groups smaller.
A strong example of small group interaction comes in the educational game Voices of
Spoon River 3D or VOSR 3D (Creative Learning Environments Lab, 2005). The VOSR 3D
project involves the design, development, implementation and research of an educational
interactive fiction game in a virtual 3D environment to teach poetry to 9th grade English
Walker & Shelton 13
students. The game play combines aspects of interactive text-adventures and multi-player virtual
environments that introduce students to the works of an American poet. At the same time, VOSR
3D provides an opportunity to solve puzzles, talk to ghosts, and engage in a computer-based
reading adventure. Each poem encountered in the environment is an epitaph of someone from the
fictional town of Spoon River. The many connections and stories to be discovered from a careful
reading of the individual poems require students to work together to advance in the game’s
context. So while this is a game about resolving conflicts and uncovering mysteries relating to
these epitaphs, progressing in the game requires a collaborative solving of these conflicts while
necessitating careful reading (Shelton, 2005, 2007a). The designers wanted to create an
environment where students have access to the actual epitaph text, a supportive framework to
augment the students' grasp of the material, and puzzles and challenges that encourage students
to critically examine the relationships and problems expressed in the Spoon River Anthology
epitaphs. A number of studies have addressed the kinds of learning that takes place in
collaborative virtual 3D environments and thoughtful game play (Steinkuehler, 2004, 2006).
Figure 4 – Voices of Spoon River
Walker & Shelton 14
PBL Outcomes
The intended outcomes for PBL include 1) effective self-directed learning skills, 1)
content knowledge that is integrated and structured around problems, 2) skills for solving
problems that are connected to content knowledge, and 4) an increased motivation for learning
(Barrows, 1986). Each of these outcomes will be described in more detail.
Effective Self-directed Learning Skills
To the extent that learners take the lead in defining their knowledge goals with respect to
a problem, identifying their knowledge gaps, and then selecting and pursuing resources to fill
those gaps, they will develop self-directed learning skills (Barrows, 1996; Savery & Duffy,
1995). These skills include the ability to identify, select and evaluate resources, and develop an
awareness for what they need to learn (Barrows, 1986). This outcome can be diminished, largely
in the name of efficiency through several means. For example: Facilitators or content experts
who lecture, listing narrow learning goals for students, or providing a constrained list of
resources for a single problem (Barrows, 1996).
Examples of encouraging self-directed learning skills can be found in Civilization III
(FIRAXIS Games, 2005). Civilization III is a computer game that leads players through a
simulation of world history, politics and geography. Each student makes decisions as a leader of
a large group of people that impact the course of that group within historical contexts. The
decisions the student makes are not necessarily “right” or “wrong,” rather, they are choices that
have real effects on the outcome of the computerized citizens of the environment. Opportunities
exist within the game to reflect on which decisions are made and the consequences of those
decisions—including opportunities for modifications of those decisions (Squire, 2004).
Walker & Shelton 15
Figure 5 – Civilization III
Integrated and Structured Content Knowledge
This outcome has two important features. First, integrated content knowledge arises from
the cross-disciplinary nature of PBL. Since problems are selected because they represent the
naturally occurring complexity of a given domain, students are forced to pull from all of the
relevant sub-disciplines in order to achieve a state of resolution. In order to fully derive this
benefit it is crucial for the curriculum to not drop out of PBL for various sub-disciplines
(Barrows, 1996). The second outcome, structured content knowledge, is the essential core of
PBL (Barrows, 1996). This concept has strong ties to three features of information processing
theory, specifically prior knowledge activation, encoding specificity, and elaboration (Albanese,
2000). By asking students to move as close as possible to the resolution state of a problem
before using their outside resources, they will be forced to activate their relevant prior
knowledge which will assist them greatly in structuring their newly acquired knowledge
(Albanese, 2000; Barrows, 1996). The concept behind encoding specificity is close alignment
between the tasks of learning and the tasks of practice. By achieving this alignment, learners
will be more likely to retrieve and apply (e.g. transfer) their content knowledge to future work
(Albanese, 2000; Barrows, 1986). The ability to engage in transfer also depends on elaboration
Walker & Shelton 16
of newly acquired knowledge structures. In the context of PBL this occurs through the
discussion of, reflection on and critique of knowledge that takes place in small group
interactions. Engaging in this kind of elaboration allows for increased levels of understanding
and more likely retrieval (Albanese, 2000).
The kind of complexity necessary for PBL to occur is certainly present in Civilization III.
As stated by Wong (2001), the game steeps learners in issues of building empires, city
improvements, researching technologies, and building and controlling military units. Multiple
solution paths exist, including political power, military power, or production through building a
spaceship. Each of these main solution states can be achieved by several possible means. While
it is advisable to focus their strategy, learners still need to pay attention to all facets of the
environment (e.g. military, politics, industrial production, research, etc . . . ). Although
Civilization III is an excellent example in terms of knowledge integration it does not appear to
embody the outcome of structured content knowledge. There are no real prompts for the kinds
of discussion, reflection, and critique of newly acquired knowledge within the game.
Problem Solving Skills Connected to Content Knowledge
Each discipline has constituent problem-solving skills that may or may not be present in
other disciplines. For the field of medical education, Barrows described these problem-solving
skills as “ . . . hypothesis generation, inquiry, data analysis, problem synthesis and decisionmaking” (Barrows, 1986, p. 481). A large part of this process is training learners to reason
backwards from a hypothesis, as opposed to forwards from the given data (Hmelo, Gotterer, &
Bransford, 1997). While data-driven reasoning is highly efficient, and typically used by experts,
it is subject to high levels of error when employed by novices who lack relevant content
Walker & Shelton 17
knowledge (Patel & Kaufman, 1993). Furthermore, hypothesis-driven reasoning is useful, even
by experts, when circumstances are novel.
In order to fully achieve this outcome, it is important that problems be presented in a
manner that is most reflective of eventual practice. For example, if students are handed
sufficient information to achieve problem resolution or the problems are simplified, then they
will not be able to engage in inquiry. It is crucial that students be asked to reason from partial
information or these problem solving process skills are unlikely to emerge (Barrows, 1996).
This outcome also highlights the need for facilitators to not be directive. If they lecture about a
problem they will likely model their own data-driven reasoning process, as opposed to the
hypothesis-driven process integral to promoting problem solving skills connected to content
knowledge.
As an example, reasoning skills are required to progress through facets of the game play
features within Artistotle’s Assassins (Learning Games Initiative & Utah State University, 2005).
Aristotle’s Assassins is a role-playing adventure set within the political turmoil of ancient Greece
designed to teach period culture, philosophy, and rhetoric. The gameplay revolves around a roleplaying character chosen by the player, who is guided through the environment by a series of
clues that points to specific choices. The choices the player makes change the story progression,
but only in ways that help make sense of the task-based activity and ancient Greek culture as a
whole. For example a player’s choice of gender before the game even begins will result in
radically different responses from other characters in the game. All players go though a similar
storyline and have the same instructional objectives, however they are given the freedom of time
and action within their own individual story, that helps make each student’s experience their own
(Frushtick, 2006).
Walker & Shelton 18
Figure 6 – Aristotle’s Assassins
Students navigate the world by their own accord, stopping for rest and reflection at
specific points. Learners are self-directed, have freedom of actions and live with their decisions
which impact the environment around them. The player's responsibility is to interact with the
history and mythology of Greece on a journey that in part involves a choice between democracy
and aristocracy for Greece. The ability to problem solve within the environment is directly tied to
the emerging complexity of their understanding of the game content, where the player is required
to navigate the landscape, collect information from NPCs that is necessary to solve several
puzzles contextualized around Greek culture and politics. Although Aristotle’s Assassins does
present unresolved problems and asks students to reason from partial knowledge it appears to fall
short in promoting hypothesis generation and the backwards driven reasoning process.
Increased Motivation for Learning
Motivation is a direct result of both the challenge of arriving at problem resolution as
well as the relevance to future practice (Barrows, 1986, 1996). In terms of relevance, students
are able to see more clearly why a specific reasoning process is necessary to learn. In addition,
they have an immediate need, the solution of their problem, for pursuing relevant content
Walker & Shelton 19
knowledge. Since the problem is tied to future practice, the relevance of acquiring necessary
content knowledge is far more transparent (Barrows & Tamblyn, 1980). In terms of challenge,
students become intrinsically motivated by their growing ability to achieve problem resolution.
This is a marked difference from the extrinsic rewards of a grade or standardized test, and will
persist when extrinsic rewards are no longer present (Barrows & Tamblyn, 1980).
The motivation for learning with games like Civilization III and Aristotle’s Assassins
mirrors those properties of traditional video gameplay. Aspects of challenge, proclivity, and
uncertainty, along with certain social factors all play a role in why students gravitate toward
computer gaming. Game-like activity that is aligned with learning objectives, whether it happens
to be ancient Greek rhetoric or international politics is an educationally meaningful embodiment
of this motivation. The moves the player makes have real consequences based on real decisions.
The player cares about the outcome, and can enjoy their experience at the same time.
In Civilization III, the focus is not to just on generic civilization battles, but actual
conquests from history, and to control every aspect of them. For example, an early conquest
involves the Phoenicians battling the Babylonians. Each game revolves around a conquest that
lets the player have access to specific maps and decisions that control the growth of the virtual
society. Players can go as far as creating a police force to prevent civil unrest in the cities, and
make choices about ruling with love and respect or fear (Kolumbic, 2006). The game certainly
holds interest through aspects of challenge. Thirdly, the game should hold levels of uncertainty,
and hold rewards for accomplishing progressive tasks. In Civilization III, if you are a good leader
your people will from time to time reward you by building and improving your palace, an
extrinsic reward. As your society grows and prospers, the player is imbued with a sense of power
and accomplishment, an intrinsic reward (Steinkuehler, 2003). Then there exists a level of social
Walker & Shelton 20
interaction within the game as well that many players find fascinating. The game allows for
team-like play or adversarial, with and against strangers or friends. Many players find the social
organization and relationships of online games extremely satisfying (Steinkuehler, 2003).
Although Civilization III certainly presents a challenging environment, the tie to future
practice is not immediately apparent. The primary task of ruling a civilization is one that
learners are unlikely to engage in. That said, relevance may be achieved through carefully
structuring the problems that students engage in within the game. For example, the focus may
shift from ruling a civilization to brokering peace between Lebanon and Israel in a custom
scenario about current events. This need to customize the game environment underscores the
discussion above about differences between Aristotle’s Assassins which is an educational game,
and Civilization III, a commercial game that can be used for educational purposes. While
Civilization III may do a better job of increasing motivation to engage in the activity, that
motivation may not always be directed at learning goals. Conversely, Aristotle’s Assassins may
focus on learning goals to the detriment of motivation.
Assessment Rubric & Discussion
The assessment rubric below (see Table 1) is built upon the characteristics and outcomes
of PBL discussed above in the context of educational game design. Alongside each trait is a
scoring sample initiated by an expert in PBL and confirmed by an expert in educational games.
For the purposes of uniformity, if an element from the rubric was not readily apparent from the
game or the available information about the game then non-conformance was assumed.
From looking at the sample scores a couple of things are apparent. First, the overall
score, an unweighted average of all eight sub-scores, is not all that flattering (48%). Although
some game examples were chosen because they helped elaborate a principle from PBL, most
Walker & Shelton 21
were selected because they were the best example known to the authors of a particular PBL trait.
If the assumption that educational games will benefit from a PBL design is true, then this
suggests some ripe areas for incorporation of PBL into future game designs. Most notably, the
creation of problems that are learner centered, promote self-directed learning, content knowledge
that is both integrated and structured, as well as increase motivation for learning. This last area
(motivation for learning) is particularly promising since, as noted above, educational games have
been described as boring and unfulfilling (Kirriemuir, 2002; Kirriemuir & McFarlane, 2003).
Perhaps a PBL structured educational game will be able to provide this motivational element in a
way that maintains a focus on learning. Note that it is possible these games are meeting the
criteria in ways that are not anticipated by PBL. From a PBL perspective Civilization III may
look poor with respect to self-directed learning because it does not explicitly prompt learners to
identify and then fill in their knowledge gaps. Still, this activity may emerge from play itself.
That is the highly motivational nature of the game (at least with respect to challenge if not
learning) may be enough to encourage learners to go through these steps even without
prompting. Thus, the best way to use the rubric may not be to make value judgments based on a
summary score, but instead to take a closer look at an educational game to see if a particular
characteristic is fulfilled in another way or an outcome is achieved through alternate means.
Further examination of the rubric reveals some additional recommendations for future
game designs. Although each characteristic is given equal weight in the rubric it should be clear
from reading the outcomes section that some portions have more far-reaching implications than
others. For example, if problems do not relate closely with future practice it will likely hinder
motivation for learning and the integration of content knowledge. If teachers are too directive in
Walker & Shelton 22
their role as facilitators, they may hamper the development of self directed learning skills and
problems solving skills.
Conclusions
This paper argues that educational games may benefit a great deal from incorporating the
traits of PBL. The combination of educational games and PBL may help bridge the gap between
the enticements of a commercial game designed purely for entertainment and an educational
game that holds no allure for learners. There are two uses for the rubric: 1) to evaluate the PBL
alignment of an educational game and 2) inform the design of educational games through the
inclusion of PBL features.
It is important to note that the rubric should be used to determine points at which an
educational game or a game design may not align with PBL, as an impetus for closer
examination. That examination may reveal that an outcome is achieved through alternate means.
For example, Civilization III fell well short on the outcome of structuring content knowledge
because there are no functions or prompts for discussion, reflection, and critique within the
game. However, these kinds of activities do take place external to the game, in player chats,
discussion and newsgroup sessions, and blogs about actions and strategies. Thus a game may be
able to overcome a lack of small group interactions by taking learners to an environment external
to the game.
Planned future work includes the use of this rubric as a framework for educational game
designs created by graduate students in an educational gaming class. This will provide a means
of validating the rubric from a practitioner perspective through participant reflection papers and
interviews. Further validation as well as reliability (intra-rater and inter-rater) analysis is
planned by applying the rubric to several existing educational games. Finally, the rubric will be
Walker & Shelton 23
applied to iterations of the Voices of Spoon River game. Each version will be used by students
and empirical data on learning outcomes will be collected in part to analyze the predictive
validity of the rubric.
Walker & Shelton 24
Table 1 Assessment/Design Rubric
Trait
Criteria
Characteristics
Authentic
__ Problems are complex (cross-disciplinary).
Problems
__ Problems have multiple solution paths.
__ Problems are unresolved and ill-structured.
5 possible
__ Problems are likely to be encountered in professional practice.
__ The environment should approximate that of eventual practice unless:
• The environment is abstracted as a design decision (e.g. a virtual world) to support the
authentic nature of the task
• And learning is followed up with opportunities for practice in a more realistic setting.
Learner
__ Learners generate objectives from given (and unresolved problems).
Centered
__ Learners locate and pursue resources (content experts, reference books, journals articles)
from that will assist in problem resolution.
3 Possible
__ Learners engage in self and peer assessment of problem solving performance.
Teachers
as
Facilitators
__
3 Possible
__
Small
Group
Interaction
__
__
4 Possible
__
__
__
Facilitators model and prompt students with meta-cognitive questions that assist in problem
resolution.
Over time, facilitators fade their assistance and prompts in favor of students taking on the
responsibility themselves.
Facilitators are not directive with respect to content (e.g. they do not lecture or correct the
misunderstandings of students).
Learners meet in small (5-9 person) groups.
Learners decide as a group what their knowledge gaps are in relation to the problem at hand
and divide up the pursuit of knowledge to fill those gaps.
After acquiring new knowledge, learners meet again to share and discuss their findings.
As part of sharing their findings, the group reflects on and critiques the utility of the
acquired knowledge in solving the problem.
Example
Quest Atlantis is complex (1) and involves
problems that will likely be encountered in
real life (1). Although the environment is
abstracted this was done in support of the
authentic nature of the task and alongside
opportunities for practice in a more realistic
setting(1). Total = 3/5 (60%).
Virtual Gorilla certainly presents an
unresolved problem (keeping the Gorillas
happy) that forces learners to generate their
own learning objectives (1). It appears to
fall short in a search of external resources
and assessment. Total = 1/3 (33%).
The Multi-User Virtual Environments do
appear to engage in a fading of their
coaching (1), although it is not clear if that
coaching involves meta-cognitive prompts.
Students did acknowledge their instructor
took a passive role with respect to content
(1). Total = 2/3 (67%).
Voices of Spoon River 3D offers an
opportunity for small group interaction (1)
by requiring learners to share and discuss
their findings in order to move forward in
the game (1). It does not, however, promote
a divide and conquer approach or enforce
the reflection and critique of achievements
or pitfalls. Total = 2/4 (50%).
Walker & Shelton 25
Outcomes
Effective
Selfdirected
Learning
Skills
3 Possible
Integrated
and
Structured
Content
Knowledge
4 Possible
Problem
Solving
Skills
Connected
to Content
Knowledge
3 Possible
Increased
Motivation
for
Learning
__
__
Learners take the lead in identifying knowledge goals.
Learners are responsible for identifying the gap between their knowledge goals, and their
current knowledge.
__ As a consequence of filling their knowledge gaps, learners improve their ability to identify,
select, and evaluate resources.
Tied with Small Group Interaction, but can only occur if Teachers as Facilitators don’t usurp
the learner’s activities.
__ Learners obtain a set of content knowledge that is integrated (e.g. cross-disciplinary). Note
this part of the outcome will be threatened if the game drops out of PBL for select content
areas.
__ Learners activate their prior knowledge by attempting to solve the problem before filling
their knowledge gaps.
__ Learners benefit from encoding specificity--close alignment between the tasks of learning
and the tasks of future practice.
__ Learners elaborate on their newly acquired knowledge through reflection, discussion, and
critique as it is shared with the group.
Tied with using complex (e.g. cross-disciplinary) Authentic Problems that relate closely to
future practice and to the reflection, discussion, and critique in Small Group Interaction.
__ Learners engage in backwards-driven reasoning (e.g. generating a preliminary hypothesis
and reasoning back through their existing knowledge set towards the data provided for the
problem).
__ Learners are presented with partial knowledge about unresolved problems, which allows
them to utilize inquiry.
__ Learners take the primary responsibility for data analysis, problem synthesis and decisionmaking.
Tied to Authentic Problems that are presented with partial information and in an unresolved state
as well as Teachers as Facilitators who are not directive (e.g. do not lecture or correct
misunderstandings).
__ Learners engage with problems that are relevant to their future practice as professionals.
__ Learners are challenged by the presented problems, and become intrinsically motivated by
growing ability to bring problems to resolution.
Tied to all facets of Authentic Problems.
2 Possible
The overall score is an unweighted average across all traits (48%) in this case.
Civilization III represents a very openended environment for learners. It is
certainly true that knowledge goals will
largely come from the learner (1) but it is
less clear if they are encouraged to do so, or
to identify knowledge gaps, much less fill
them. Total = 1/3 (33%).
Civilization III does quite well at presenting
complex problems and integrating content
knowledge across disciplines (1), it falls
well short in all three facets of structured
content knowledge—with the possible
exception of opportunities for reflection
(but this is not coupled with discussion and
critique). Total = 1/4 (25%).
Decision-making plays a key role in
Aristotle’s Assassins, as does problem
synthesis and data analysis (1). Learners
gain the bulk of the knowledge needed to
resolve various problems in the game only
during the course of play (1).
Encouragement to engage in hypothesis
formation and backwards-driven reason
does not appear to be present. Total = 2/3
(67%).
Civilization III presents challenging
problems that allow a progression in
problem solving ability (1). Since the
primary task of leading a society is not
likely to be the future practice of the
learners, it is not all that relevant. Total =
1/2 (50%).
Walker & Shelton 26
References
Albanese, M. (2000). Problem-based learning: why curricula are likely to show little effect on
knowledge and clinical skills. Medical Education, 34(9), 729-738.
Aldrich, C. (2005). Learning by doing: A comprehensive guide to simulations, computer games,
and pedagogy in e-learning and other educational experiences. San Francisco: Pfeiffer.
Allison, D., Wills, B., Bowman, D., Wineman, J., & Hodges, L. F. (1997). The virtual reality
gorilla exhibit. IEEE Computer Graphics and Applications, 17(6), 30-38.
Appelman, R. A. (2005). Designing experiential modes: A key focus for immersive learning
environments. Tech Trends, 49(3), 64-75.
Arts, J. A. R., Gijselaers, W. H., & Segers, M. S. R. (2002). Cognitive Effects of an Authentic
Computer-Supported, Problem-Based Learning Environment. Instructional Science,
30(6), 465-495.
Barab, S. (2006, June 15-16). Gandalf enters the classroom: Learn how we have made the
transition with Quest Atlantis. Paper presented at the Games, Learning & Society
conference, Madison, WI.
Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzan, H. (2005). Making learning fun:
Quest Atlantis, a game without guns. Educational Technology Research & Development,
53(1), 86-107.
Barrows, H. S. (1985). How to design a problem-based curriculum for the preclinical years.
New York: Springer Publishing Company.
Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical education,
20(6), 481-486.
Barrows, H. S. (1996). Problem-based learning in medicine and beyond: A brief overview. New
directions for teaching and learning(68), 3-12.
Barrows, H. S. (2002). Is it Truly Possible to Have Such a Thing as dPBL? Distance Education,
23(1), 119-122.
Barrows, H. S., & Tamblyn, R. M. (1980). Problem-based learning: An approach to medical
education. New York: Springer Publishing Company.
Breton, G. (1999). Some empirical evidence on the superiority of the problem-based learning
(PBL) method. Accounting Education, 8(1), 1-12.
Center for Research on Learning & Technology. (2000a). Quest Atlantis. Retrieved January 5,
2007, from http://atlantis.crlt.indiana.edu/start/qaworlds.html
Center for Research on Learning & Technology. (2000b). Quest Atlantis: Mission. Retrieved
August 29th, 2006, from http://atlantis.crlt.indiana.edu/start/mission.html
Crawford, C. (1984). The art of computer game design. Berkeley, CA: McGraw-Hill/Osborne
Media.
Creative Learning Environments Lab. (2005). The Voices of Spoon River project. Retrieved
January 5, 2007, from http://cle.usu.edu/cle_inst_games_learning.html
Csíkszentmihályi, M. (1997). Finding flow. New York: Basic Books.
DeCastell, S., & Jenson, J. (2003). Serious play. Journal of Curriculum Studies, 35(6), 649-665.
Dede, C., Salzman, M. C., Loftin, R. B., & Sprague, D. (1999). Multisensory immersion as a
modeling environment for learning complex scientific concepts. In W. Feurzeig & N.
Roberts (Eds.), Computer Modeling and Simulation in Science Education. New York:
Springer-Verlag.
Walker & Shelton 27
Ertmer, P. A., & Simons, K. D. (2006). Jumping the PBL implementation hurdle: Supporting the
efforts of K-12 teachers. The interdisciplinary Journal of Problem-based Learning, 1(1),
40-54.
Farquhar, J., H., & K., K. (1986). Effect of two preclinical curricular on NMBE part 1
examination performance. Journal of Medical Education, 61(5), 368-373.
FIRAXIS Games. (2005). Sid Meier's Civilization III. Retrieved January 5, 2007, from
http://www.civ3.com/
Frushtick, R. (2006). Civilization III Conquests. Retrieved August, 2008, from
http://www.civ3.com/conq_features.cfm
Garland, N. J. (1995). Peer group support in economics: Innovations in problem-based learning.
In W. Gijselaers, D. Tempelaar, P. Keizer, E. Bernard & H. Kasper (Eds.), Educational
innovation in economics and business administration: The case of problem-based
learning (pp. 331-337).
Harvard University. (2003). Introducing MUVEES. Retrieved January 5, 2007, from
http://www.gse.harvard.edu/~dedech/muvees/index.htm
Hay, K. (2006, June 15-16). Building a wrong model: Developing the learning affordances of 3D
environments. Paper presented at the Games, Learning & Society 2006, Madison, WI.
Heycox, K., & Bolzan, N. (1991). Applying problem-based learning in first-year social work. In
D. Boud & G. Feletti (Eds.), The challenge of problem-based learning in work (pp. 186193). New York: St. Martin's Press.
Hmelo, C. E., Gotterer, G. S., & Bransford, J. D. (1997). A theory-driven approach to assessing
the cognitive effects of PBL. Instructional Science, 25(6), 387-408.
Jonassen, D. H. (2000). Toward a Design Theory of Problem Solving. Educational Technology
Research and Development, 48(4), 63-85.
Jones, J. W., Beiber, L. L., Echt, R., Scheifley, V., & Ways, P. O. (1984). A Problem-based
curriculum, ten years of experience. In H. G. Schmidt & M. L. d. Volder (Eds.), Tutorials
in problem-based learning. New direction in training for health professions.
Kirkley, S. E., Tomblin, S., & Kirkley, J. (2005). Instructional design authoring support for the
development of serious games and mixed reality training. Paper presented at the
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC)
November 28-December 1, 2005, Orlando, FL.
Kirriemuir, J. (2002). Video gaming, education and digital learning technologies. D-Lib
Magazine, 8.
Kirriemuir, J. (2003). The relevance of video games and gaming consoles to the higher and
further education learning experience. Retrieved August 18, 2008, from
http://www.jisc.ac.uk/uploaded_documents/tsw_02-01.rtf
Kirriemuir, J., & McFarlane, A. (2003). Use of computer and video games in the classroom.
Paper presented at the DiGRA, Utrecht, Holland.
Kolumbic, K. (2006). We play Civilization III... Retrieved July, 2006, from
http://www.strategyinformer.com/pc/civilizationiii/review.html
Learning & Performance Support Laboratory. (2000). The virtual gorilla modeling project.
Retrieved January 5, 2007, from
http://lpsl.coe.uga.edu/projects/projects_detail.asp?projectName=The%20Virtual%20Gor
illa%20Modeling%20Project
Learning Games Initiative, & Utah State University. (2005). Aristotle's Assassins. Retrieved
January 5, 2007, from http://lgi.usu.edu/projects/aristotle.htm
Walker & Shelton 28
Macklin, B. (2006). Video games: Where to now? (electronic report).
Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic
motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning and
Instruction: Conative and Affective Process Analysis (Vol. 3). Englewood Cliffs, NJ:
Erlbaum.
Mestre, J., Dufrense, R. J., Gerace, W. J., & Hardiman, P. T. (1993). Promoting skilled problemsolving behavior among beginning physics students. journal of Research in Science
Teaching, 30(3), 303-317.
Moust, J. H. C., Grave, W. S. D., & Gijselaers, W. H. (1990). The tutor role: A neglected
variable in the implementation of problem-based learning. In Z. H. Nooman, H. G.
Schmidt & E. S. Ezzat (Eds.), Innovation in medical education: An evaluation of its
present status (pp. 135-151). New York: Springer.
Nelson, B. C., Ketelhut, D. J., Clarke, J., Dieterle, E., Dede, C., & Erlandson, B. (2007). Robust
design strategies for scaling educational innovations: The River City MUVE case study.
In B. E. Shelton & D. Wiley (Eds.), The Educational Design and Use of Simulation
Computer Games. Rotterdam, The Netherlands: Sense Publishers.
Paras, B., & Bizzocchi, J. (2005, June 16-20). Game, motivation, and effective learning: An
integrated model for educational game design. Paper presented at the Digital Games
Research Association (DiGRA): Changing Views -- Worlds in Play, Vancouver, BC.
Patel, V. L., & Kaufman, D. R. (1993). Development of knowledge-based reasoning strategies
with medical training (No. CME93-CS3): Centre for Medical Education.
Polglase, R. F., Parish, D. C., Buckley, R. L., Smith, R. W., & Joiner, T. A. (1989). Problembased ACLS instruction: A model approach for undergraduate emergency medical
education. Annals of Emergency Medicine, 18(9), 997-1000.
Prensky, M. (2001). Digital Game-based Learning. New York: McGraw-Hill.
Rangachari, P. K. (1996). Twenty-up: Problem-based learning with a large group. New
directions for teaching and learning(68), 63-81.
Rieber, L. P. (1992). Computer-based microworlds: A bridge between constructivism and direct
instruction. Educational Technology Research and Development, 40(1), 93-106.
Rieber, L. P. (1996). Seriously considering play: Designing interactive learning environments
based on the blending of microworlds, simulations, and games. Educational Technology
Research and Development, 44(2), 43-58.
Rohrs, H., & Lenhart, V. (Eds.). (1995). Progressive education across the continents: A
handbook (Vol. 44). Frankfurt am Main: Peter Lang.
Ross, S. M., & McNeil Hurlbert, J. (2004). Problem-Based Learning: An Exercise on Vermont's
Legalization of Civil Unions. Teaching Sociology, 32(1), 79 (15 pages).
Savery, J. R. (2006). Overview of problem-based learning: Definitions and distincts. The
interdisciplinary Journal of Problem-based Learning, 1(1), 9-20.
Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its
constructivist framework. Educational Technology, 35(5), 31-38.
Shelton, B. E. (2005, December 7-8). Designing and creating interactive fiction for learning.
Paper presented at the New Media Consortium (NMC) Online Conference on
Educational Gaming, Internet.
Shelton, B. E. (2007a). Designing instructional games for activity-goal alignment. In B. E.
Shelton & D. Wiley (Eds.), The Design and Use of Simulation Computer Games in
Education. Rotterdam, The Netherlands: Sense Publishers.
Walker & Shelton 29
Shelton, B. E. (2007b). The practices of designing educational games for activity-goal alignment.
In B. E. Shelton & D. Wiley (Eds.), The Educational Design and Use of Simulation
Computer Games. Rotterdam, The Netherlands: Sense Publishers.
Son, B., & Van Sickle, R. (2000). Problem-solving instruction and students' acquisition,
retention and structuring of economic knowledge. Journal of Research and Development
in Education, 33(2), 95-105.
Spaulding, W. (1969). The undergraduate medical curriculum (1969 model): McMaster
University. Canadian Medical Association Journal, 100(14), 659-664.
Squire, K. (2002). Cultural framing of computer/video games. Game Studies, 2(1).
Squire, K. (2004). Sid Meier's Civilization III. Simulations and Gaming, 35(1), 135-140.
Squire, K., Barnett, M., Grant, J. M., & Higginbotham, T. (2004). Electromagnetism
supercharged! Learning physics with digital simulation games. Paper presented at the
International Conference of the Learning Sciences, Santa Monica, CA.
Steinkuehler, C. A. (2003, March 25). Videogaming as participation in a discourse. Paper
presented at the Annual Conference of the American Association for Applied Linguistics.
Steinkuehler, C. A. (2004, March 25). Learning in massively multiplayer online games. Paper
presented at the Sixth International Conference of the Learning Sciences (ICLS), Santa
Monica, CA.
Steinkuehler, C. A. (2006). Massively multiplayer online videogaming as participation in a
discourse. Mind, Culture & Activity, 13(1), 38-52.
Steinkuehler, C. A., Derry, S. J., Hmelo-Silver, C. E., & Delmarcelle, M. (2002). Cracking the
Resource Nut With Distributed Problem-Based Learning in Secondary Teacher
Education. Distance Education, 23(1), 23-39.
Van Treuren, K. W., & Havener, A. G. (1997). Teaching introductory engineering: A problem
based learning experience. Biomedical Sciences Instrumentation, 34, 1-6.
VanEck, R. (2006, June). Where do we go from here? Ten critical areas to guide future research
in digital games-based learning. Paper presented at the conference on Games, Learning
& Society, Madison, WI.
Walton, M. (1996, June 12). Virtual gorilla puts students in the wild. cnn.com.
Wong, T. (2001). Civilization 3 Review. Retrieved July, 2006, from
http://www.firingsquad.com/games/civ3
Walker & Shelton 30
Footnote
1
While concepts from PBL can certainly be connected to the progressive education movement
the specifics of those connections are subject to debate. Despite conceptual similarities, Barrows
tends not to cite these theoreticians, philosophers, and practitioners when discussing the origins
of his own work. We admit to side-stepping the issues around the origins of PBL and
educational philosophy, in order to avoid a broader discussion that is perhaps best left to another
article.