Cognitive Tutor Tools for Advanced Instructional Strategies
Kenneth R. Koedinger
Vincent A.W.M.M. Aleven
Neil T. Heffernan
Human-Computer Interaction Institute
Carnegie Mellon University
Contact Information for PI
Phone: 412 268 7667
Fax: 412 268 1266
Email: [email protected]
Abstract
Cognitive Tutors have been demonstrated to yield dramatic improvements in student
learning. Classroom evaluations of our Algebra Cognitive Tutor have demonstrated that
students in tutor classes outperform students in control classes by 50-100% on targeted
real-world problem-solving skills and by 10-25% on standardized tests. This tutor is now
being marketed nationwide and is use in over 725 schools across the country. Despite the
great potential of Cognitive Tutors to improve student learning in other areas,
development of such systems is currently costly and has been limited to just a few
research teams. We propose to create a family of "cognitive model editors" that will
employ advanced user interface techniques to make Cognitive Tutor development and
debugging easier, faster, and more error free. Such tools would not only speed the
development process, but by allowing psychologists and education researchers to build
these models directly, such tools would improve quality, reduce development cost, and
open access to this effective educational technology. The proposed project is an initial
step in a larger research agenda designed to test: 1) hypotheses about key features of
Cognitive Tutors that yield greater learning yet require a greater development cost than
traditional computer-aided instruction and 2) hypotheses about development and
debugging tools that will dramatically lower the costs of Cognitive Tutor development.
The goal of the current project is to create the initial set of tools that will facilitate these
experiments.
2
Problem
Cognitive Tutors have been demonstrated to yield dramatic improvements in student
learning. As a particular approach to intelligent tutoring systems, Cognitive Tutors
combine cognitive science theory, HCI methods, and particular AI algorithms for
modeling student thinking. Classroom evaluations of our Algebra Cognitive Tutor have
demonstrated that students in tutor classes outperform students in control classes by 50100% on targeted real-world problem-solving skills and by 10-25% on standardized tests
(Koedinger, Anderson, Hadley, & Mark, 1997; Koedinger, Corbett, Ritter, & Shapiro,
2000). This tutor is now being marketed nationwide and is use in over 725 schools
across the country (Corbett, Koedinger, & Hadley, 2001; see also
www.carnegielearning.com).
Despite the great potential of Cognitive Tutors to improve student learning in
other areas, development of such systems is currently costly and has been limited to just a
few research teams. In addition to experts in the education domain, such teams currently
require PhD level expertise in cognitive task analysis and advanced symbolic AI
programming skills (e.g., LISP and rule-based production system skills).
Opportunity
The project investigators of this proposal are particularly well positioned to tackle this
problem. Our research group as a whole, and each one of us individually, has extensive
experience in developing and evaluating Cognitive Tutors. Further, we have
considerable experience training both programmers and non-programmers to develop
Cognitive Tutors using our existing tool set.
3
In addition to the high school algebra Cognitive Tutor course described above, we
have created complete courses for high school geometry and algebra II, three years of
middle school math (grades 6-8), and, at the college level, quantitative literacy, LISP
programming, and introductory statistics (Aleven & Koedinger, in press; Anderson,
Conrad, & Corbett, 1989; Corbett, McLaughlin, & Scarpinatto, 2000; Koedinger &
Sueker, 1996; Rittle-Johnson & Koedinger, 2001). All of these systems are in active use
by students here at CMU and/or at schools and colleges across the country. All of these
systems have been subject to substantial empirical evaluation, both formative and
summative. The mathematics Cognitive Tutors have been licensed to a spin-off
company, Carnegie Learning (see www.carnegielearning.com).
The Cognitive Tutor Technology
Cognitive Tutors are based on the ACT theory of learning and performance
(Anderson & Lebiere, 1998). ACT is a complex and broad “unified theory of cognition”
(Newell, 1990). The theory distinguishes between tacit performance knowledge, socalled “procedural knowledge” and static verbalizable knowledge, so-called “declarative
knowledge”. According to ACT, performance knowledge can only be learned by doing,
not by listening or watching. Such performance knowledge is represented in the notation
of if-then production rules.
To build a Cognitive Tutor, we create a cognitive model of student problem
solving by writing production rules that characterize the variety of strategies and
misconceptions students acquire. For simplicity of illustration, we provide an example
from the domain of equation solving:
4
Strategy 1: IF the goal is to solve a(bx+c) = d
THEN rewrite this as bx + c = d/a
Strategy 2: IF the goal is to solve a(bx+c) = d
THEN rewrite this as abx + ac = d
Misconception: IF the goal is to solve a(bx+c) = d
THEN rewrite this as abx + c = d
The first two productions illustrate alternative strategies for the same problemsolving goal. By representing alternative strategies for the same goal, the Cognitive
Tutor can follow different students down different problem-solving paths of the students’
own choosing. The third “buggy” production represents a common misconception (cf.,
Sleeman, 1986). Buggy production rules allow the Cognitive Tutor to recognize such
misconceptions and thus, provide appropriate assistance. A complete cognitive model for
linear equation solving would have hundreds of production rules, dealing with the
different types of subgoals that arise in equation solving (e.g. to reduce a particular form
of equation to a simpler form) and the different operations that can be applied to each
subgoal (e.g., add or subtract to both sides, multiply or divide both sides, combine like
terms, distribute, etc.). The Cognitive Tutor makes use of the cognitive model to follow
students through their individual approaches to a problem. It does so using a technique
called “model tracing”.
The cognitive model is also used to trace students’ knowledge growth across
problem-solving activities. The “knowledge tracing” technique is dynamically updating
estimates of how well the student knows each production rule (Corbett & Anderson,
1995). These estimates are used to select problem-solving activities and to adjust pacing
to adapt to individual student needs.
Experience Teaching Cognitive Tutor Development
To make it feasible for others to create Cognitive Tutors, starting in the fall of 2000, we
have had four extended experiences in teaching Cognitive Tutor development to over 60
5
students. Two of these experiences were instances of a semester long course, “Cognitive
Modeling and Intelligent Tutoring System”, taught to Carnegie Mellon graduates and
undergraduates by project investigators Koedinger and Aleven. One was a four-week
version of this course taught by Koedinger and Dr. Tanja Mitrovic at the University of
Christchurch, New Zealand. The fourth was a week-long full-time summer course taught
to 20 researchers, graduate students, and programmers from around the US
(http://www.pitt.edu/~circle/SummerSchool/Report2001.html). These courses have
involved students with a wide variety of backgrounds, including cognitive psychology or
computer science, but also individuals with instructional domain expertise but with little
programming experience and little to no cognitive science background. These courses
have had important instructional outcomes besides Cognitive Tutor development, like
teaching cognitive task analysis and data-driven software design. However, except for 6
students who were engaged in a local project at CMU or Carnegie Learning, only 1 other
student has gone on to successfully create a Cognitive Tutor and only with extensive
consulting with us. In other words, given the current state of our development tools, it is
currently difficult to impossible to learn to create Cognitive Tutors without an extensive
apprentice experience.
It has become abundantly clear that a real barrier to success is that our current set
of tools for developing Cognitive Tutors is particularly hard to use. In a survey at the end
of the summer course, many students commented on the difficulty of learning our
existing “Tutor Development Kit” (TDK), which is embedded in the LISP programming
language (Anderson & Pelletier, 1991). Students' stated dislikes included "challenge of
the TDK", “learning to use TDK”, and “amount of programming skill is the main barrier
6
for full, immediate enjoyment of the unique things this course has to offer”. Another
student commented “being involved in the development of a tutor has been quite
interesting, but it's very difficult to cope with all the [topics] because of the huge effort of
facing the LISP programming”. Cognitive modeling in TDK challenges even students
with extensive programming background. One such student’s list of “likes” included:
“thinking about the problem & translating it into production rules & wme's (although
difficult)".
Our teaching experiences have not only made clear the need for better tools. They
have also given us data both on the variety of specific difficulties beginning cognitive
modelers experience and the kinds of instructional supports that help. Armed with this
information, we are in the position to create tools that reduce these difficulties and
automate aspects of the instructional support that has proven effective.
Related Work on Tools for Authoring Instructional Software
Instructional authoring software that is available commercially (e.g., MacroMind
Director) is limited to the production of multi-media enhancements of traditional textbased instruction. The benefits of multimedia presentations over traditional approaches is
limited at best (e.g., Miller, Lehman, & Koedinger, 1999; Palmiter & Elkerton, 1993;
Pane, Corbett, & John, 1996; Stasko, Badre, Lewis, 1993). In contrast, more
sophisticated intelligent tutoring systems (ITS) have been shown to yield dramatic
instructional improvements over traditional methods, as discussed above, but see also
Bloom, Sparks, Dooley, Meiskey, Bell, & McClard (1995) and Lajoie (1993) for
examples in industry and military settings. Some ITS authoring systems have been
created but they are used primarily by research specialists with AI programming skills.
7
More widespread and effective use of ITS authoring tools will be possible only if such
systems are made much easier to use and to learn to use. In a widely cited article, Murray
(1999) surveyed the state of the art in authoring tools for building Intelligent Tutoring
Systems. He reviews ITS authoring tools for curriculum sequencing and simulationbased tutors (like RIDES). However, his review did not identify any currently available
authoring tools for building more sophisticated ITSs, like Cognitive Tutors. We review
some existing ITS authoring tools that are most related to our effort.
Our proposed project differs from Demonstr8 (pronounced “Demonstrate”;
Blessing, 1997), an earlier prototype authoring tool for Cognitive Tutors. Demonstr8 was
designed to let educators craft a production rule model with little or no support from
programmers or cognitive modelers. It used techniques of programming by
demonstration: the author demonstrated how to solve a problem in the tutor’s user
interface (which would be developed as a first step). Demonstr8 then proposed
production rules that abstracted from the specifics of the demonstrated solution to capture
a general approach that would work for other problems of the same type. If needed, the
author could adjust the level of generality of the proposed rules using a GUI tool, for
which no knowledge of production rule syntax was needed, only a good understanding of
the task.
Attractive as this may sound, Demonstr8 did not make it past the prototype stage
and has a number of limitations. First, the production rule models it produced did not
cover the full complexity of TDK rules (e.g., negations, slot-value patterns, rule
chaining). Second, use of this system required the user to first construct an interface and
create a tight mapping between user interface elements and elements of the production
8
systems "working memory". Third, the system tried to make all generalizations
automatically rather than the mixed intelligence approach we are proposing whereby the
system eases the process of generalization through suggestions and the structure of the
interface. The difference is that it is not possible for a system to make reliable
generalizations from a single example without significant background knowledge. Thus,
the full generalization approach either makes too many generalization errors or requires
significant expert knowledge coding, but such coding is just what we are trying to avoid.
In short, our proposed approach, while less "automatic", is much more robust and
general.
A system called RIDES is "an application program for authoring and delivering
graphical simulations and intelligent tutoring in the context of simulations" (Munro,
1994, p. iii). In previous work (Koedinger & Corbett, 1997), we performed detailed
usability inspections (Nielsen & Mack, 1994) of the RIDES authoring environment and
demonstrated significant usability gains from a redesign of the lesson authoring
component (Mathan, Koedinger, Corbett, Hyndman, 2000). Our proposed Working
Memory Editor (see below) borrows from successful user interface techniques employed
for authoring the simulation component phase of RIDES. More generally, we will take
advantage of other lessons derived from our usability analysis of RIDES and redesign
success.The design of previous authoring tools has not been driven by the combination of
cognitive theory (Anderson & Lebiere, 1998; Card, Moran, and Newell, 1983) and
Human-Computer Interaction (HCI) methods for data-driven design, including usability
inspection methods (Nielsen & Mack, 1994), think-aloud user studies (Ericsson & Simon,
1984; Flanagan, 1954; Gomoll, 1990), contextual inquiry (Beyer & Holtzblatt, 1998) that
9
are available to us today. Our team has extensive expertise in employing cognitive
theory and HCI methods in system design.
Proposed Solution
We will create a family of "cognitive model editors" that will employ advanced user
interface techniques to make cognitive modeling easier, faster, and more error free.
These cognitive model editors/viewers will include the following:
1. Working Memory Editor/Viewer. Each cognitive model comprises a set of
production rules and a set of working memory elements, data structures that at any point
in time capture the state of problem solving. The Working Memory Editor allows the
Cognitive Tutor author to create working memory representations of problem states
(especially the initial state) by using structured editor techniques that make contextsensitive inferences to fill in obvious elements and provide context-specific options (e.g.,
pop-up menus). (See design principle #2 below.)
2. Instance-based Production Editor/Viewer. This tool will enable the author to
create and modify a production rule in two steps. First, the author selects (via drag-anddrop) working memory elements from the Working Memory Editor and placing them into
if-part and then-part slots of a production rule template. Next the author generalizes
these instances into an abstract production rule by deleting clauses and changing
constants to variables.
(See design principles #1 and #3 below.)
3a. Execution Space Editor/Viewer as planning tool. This tool lets the Cognitive
Tutor author construct a diagram that depicts the "execution space" for a given problem,
which shows (possibly multiple) sequences of solution steps. Such a graph is useful early
on in the development of a cognitive model. As an author demonstrates a solution to a
10
problem, successive states of the solution are displayed in a directed graph (as thumb nail
images). The author indicates alternative possible solutions (branches in the search
space) by clicking on a prior state in which case the interface is placed in that state and
the author proceeds to demonstrate an alternative solution path, which is added to the
graphic display. Once an "execution space" diagram is created, the links between states
are labeled by the production rules that make the transition. (Links to error states
correspond with "buggy" production rules.)
3b. Execution Space Editor/Viewer as debugging tool. As described above, the
Execution Space Editor is used as a planning tool to assist the Cognitive Tutor author in
deciding what production rules need to be written. It can also be used in debugging
mode. Once some productions have been written, the execution space graphic display
can be automatically generated. Authors can then check whether the generated space is
consistent or not with their plan.
4. Cognitive Model Visualizer. We currently have an initial prototype of a tool,
which provides a dynamic view of a cognitive model as it is run on a given problem. The
tool shows alternative production rule paths including conflict sets at multiple choice
points. This visualizer allows the author to do “what if” and “why not” analysis to
understand what the model can do and why it is not doing what is expected.
Some key design principles that will be employed and tested in the design and
evaluation of these editors include:
1. Inductive Support. It is easier to formulate a general rule by first generating a
specific instance, in the desired syntax, and then substituting variables for constants as
appropriate. (See Koedinger & Anderson, 1998; Gluck, 1999.)
11
2. Reference By Pointing. It is easier and more efficient to let the Cognitive Model
author reference objects by clicking (e.g., in pop-up menus) rather than by typing in
symbolic labels, which are subject to error and take time to write and rewrite.
3. Cautious Generalization. It is a robust and efficient strategy to have the system
make guesses as to appropriate generalizations, but check these guesses with the user.
For instance, when an author changes an instance of a constant to a variable, the system
suggests that other instances of that constant also be so changed, but it gets the author's
OK on each. User interfaces designed following this strategy require less knowledge of
syntax or even production rule "logic" than authoring from scratch.
The four tools described above represent near term goals for Phase 1 of our
project, which is the phase for we are currently requesting funding. For the longer-term,
we plan to develop tools for formative evaluation and iterative improvement of Cognitive
Tutors, as our experience indicates that data-driven iterative design is crucial in
developing cognitive models. The planned tools will assist authors in analyzing log files
of student-tutor interactions and using these log files to do automated testing. Second, a
Testing Tracking Tool will allow an author to compile a set of test problems plus their
correct solutions, to run a cognitive-model under-development on these problems, and be
notified of correct and incorrect behavior produced by the model. This tool will make
testing far easier than it is currently and will addresses a common problem Cognitive
Tutor authors experience: As production rules are added or modified in the course of
model development (i.e., in order to make the model work on new test problems), the
model now fails on test problems that it previously handled correctly. Third, we plan to
incorporate machine learning to improve cognitive models and/or instructional strategies.
12
We are currently creating a machine learning method called “learning factors analysis”
that uses log files to automatically evaluate alternative cognitive models based on fitting
student learning curves. Finally, we plan to create tools to facilitate deploying Cognitive
Tutors on the internet and facilitate the collection of logs of student-tutor interactions.
Deploying a tutor on the internet is not only valuable for student accessibility, but also
makes it easier to collect data on the tutor’s effectiveness.
We now describe our tool development plans in more detail using examples in
both a military domain and a school domain.
Illustrating the Working Memory and Production Rule Editor
Learning to use a current military command and control system such as the U.S. Navy
Combat Information Center takes a great deal of practice. The Aegis Simulation Program
(shown in Figure 1) is a simulation training system for an Anti-Air Warfare Coordinator
(AAWC). The task of the AAWC is complex and involves classifying planes on a radar
screen, and taking action when appropriate. . Here we illustrate how a cognitive model
for this task might be built using the proposed tools. (In fact, our colleagues (Sohn,
Douglass, Chen, & Anderson, in submission) have studied the AAWC task and built a
cognitive model in ACT-R, a related system.)
13
Figure 1: Screen shot of the Aegis Simulation Program showing (in the circle on the
right) in half square boxes the unidentified "tracks" that the AACW needs to
classify (e.g., friendly, hostile).
To build a Cognitive Tutor for the AAWC task, the author (such as a person who
oversees training of AAWC) needs to develop a cognitive model that captures the ways
in which humans perform this task. The cognitive model will include both the correct
steps experts perform, as well as common errors.
Having selected one or more appropriate scenarios on which to focus during the
development of the cognitive model, the Cognitive Tutor author begins by defining the
working memory structures (WMEs) that represent the evolving state of problem solving.
These structures hold the information that the AAWC trainee presumably knows at any
point in time as well as information about the state of the simulation as it unfolds. The
14
author starts by defining the WMEs that represent the initial state of the scenario, using
the Working Memory Editor. Once this is done, the author focuses on a concrete step the
AAWC trainee is supposed to perform in that state and creates a production rule that
generates that step. The author creates a production rule by dragging and dropping the
appropriate WMEs into a rule template in the Production Rule Editor. Then the author
generalizes the production rule by removing unnecessary conditions, as well as replacing
constants with appropriate variables. Once a rule is authored, the author can tell the
system to execute that rule. The production system makes the appropriate changes to
working memory to update the state of the scenario. The author can then work on the
next rule to be added to the system, in the same manner as before: start with concrete
instances in the rule condition and action and generalize them as needed. In this way, the
author can gradually define the rules, as well as the WMEs that model the task the
AAWC performs. Next, we illustrate this in more detail.
15
Figure 2: A mock-up of the Working Memory Elements Editor
showing the slot-value pairs for a wme named "aircraft04", which
represents an aircraft on the AAWC radar screen.
The first tool the author uses is called the Working Memory Editor, shown in
Figure 2. In the top left corner of the Working Memory Editor is a list of the different
WME types that the author has defined so far (e.g., AAWC-Scenario, types for aircraft,
and goal structures like identify-goal, or intercept-goal). The top right frame shows the
some of the WME instances that exist in the given state of the scenario, namely, those of
type "aircraft." There are 7 such WME instances, each of which represents a particular
plane on the radar screen. The bottom frame of the Working Memory Editor shows the
details of one particular aircraft WME instance, named "aircraft04". ThisWME has slots
and associated values for "Radar-Signature", "Altitude", "Course", "Speed," etc. It also
has slots that will hold information that the AAWC trainee has to identify, including the
16
airplane's primary-intent (e.g., friendly, hostile, neutral), the type of aircraft (F-15, B1bomber, Tomahawk, etc), and whether the plane has been evaluated yet.
Once the author has selected the working memory representation, the next step is
for the author to specify the production rules that represent the problem-solving steps
used by an AAWC expert. Table 1 shows a few examples of English rules that the author
needs to encode as production rules.
Table 1: A small sample of some of the rules that would be encoded in the intelligent
tutoring system for the AAWC task
1) In identifying, select the closest radar track to the ship (a “track" is an object that
appears on the radar screen like a plane or missile).
2) To select a track you move your mouse to it and click.
3) First, try identifying a track by checking whether it fits the altitude and speed profile
of a commercial aircraft and if it does classify it as such.
4) Second, try identifying a track by requesting an EWS signal and if it gives one
classify it accordingly.
5) If the radar signal indicates a type of plane that only friendly forces are know to have,
then classify the target as friendly.
6) A commercial aircraft is classified with a primary intent of "assumed friendly" and an
air ID of "non-military".
7) To classify a plane with primary intent X and air ID Y, select the following sequence
of menu options: "track", "update", "primary", X, "air", Y, "save".
8) Commercial airlines fly between 20,000 and 40,000 feet
Some of the rules in Table 1 result in directly observable actions (mouse clicks or key
clicks) in the interface, such as rule 2 and rule 7. Other rules correspond to cognitive
steps that do not result in observable actions. For example, rule 1, which models the
AAWC operator's setting a goal to identify a plane, has no concrete action in the
interface. We model the setting of goals by the creation of WMEs that represent those
goals. Therefore, the author has defined a WME type called "identify -goal".
17
Figure 4: The rule after the author
1) deleted unnecessary slots and 2)
generalized the WMEs in the
Condition and Action part by
changing constants into variables
(variables are shown in italics).
Figure 3: A mock-up of the Rule
Editor after 1) wmes have been
dropped into the Condition and
Action frames, and 2) slot values
(shown in bold) have been changed.
Let us now assume that the author has already authored rules 1-4, has defined
WMEs that represent a state in which it is appropriate to apply rule 5, and has decided to
name this state "State03 - Unidentified Aircraft". The WMEs for this state include a
WME representing a goal, named identify -goal08, to identify a particular plane, namely,
aircraft04. (When running the model, this WME would be created by rule 1.) To create
the code for rule 5 , the author creates a new Rule-Editor window and names the rule
"Identify-Friend-By-Radar". Next, the author selects the WMEs in State03 to which the
new rule should apply. In particular, the author selects a plane (aircraft04) with a radar
18
signature that only friendly forces are known to have (B1-Bomber). The author drags
and drops the aircraft04 WME as well as the idenfity-goal8 WME into the "Condition"
frame of the Rule Editor window. Since the author wants the new production rule to
change information in these WMEs, the author drags and drops the same WMEs into the
"Action" frame. Next, the author needs to generalize the WME instances in the Action
and Condition part of the rule into more abstract "WME patterns." The author first works
on the WMEs in the Action part, changing the primary-intent slot to "friendly, type-ofaircraft to B1-Bomber, and evaluated status to yes. To assist the author in quickly
understanding the meaning of the rule, the system shows these changed fields in bold.
The author also changes the "status" slot of WME idenfiy-goal08 from "pending" to
"evaluated". Figure 3 shows a mock-up of the state of the Rule-Editor at this point.
19
Since the author wants to have this rule apply to B1-bombers, regardless of the altitude,
speed and course, the author uses the mouse to delete these slots in the Condition part of
the rule. To further generalize the production so that it applies to all B1-Bombers, and not
just the one instance "aircraft04", the author replaces the occurrences of "aircraft08" with
a new variable named "=aircraft." (Terms that start with an equal sign are variables,
rather than constants. Variables are shown by the system in italics.) The author converts
a constant to a variable by clicking on an occurrence of the constant (e.g., the text
"aircraft04") and selecting from the resulting pop-up menu an option to "generalize from
constant." As a result, the system, after prompting the user, changes the other two
instances of aircraft04 to "=aircraft". The author performs similar steps to generalize the
Figure 6: The author can see both
the template versions (e.g., top
row) of their hints and also check
to see how there hints will appear
when used (e.g., second row).
Figure 5: This rule will work for all
types of aircraft that at known to be
possessed exclusively by friendly
forces.
20
constant "identify -goal08" to become the variable "=identify-goal". The resulting general
production rule is shown in Figure 4.
This production in Figure 4 can be used to classify all B1-Bombers as friendly,
but what about the other types of planes that the enemy does not have? The author could
write a separate production for each type of plane, or better yet, generalize this rule
further by matching the type of aircraft against a list of friendly aircraft. This latter
option is shown in Figure 5. The pattern matching employed at the line that reads "($
=aircraft-type $)" will attempt to bind the variable =aircraft-type to any member of the
list of friendly-aircraft.
Finally, the author clicks on the "Hints" button to write the hint templates that will
be used to generate on-demand hints. The first hint template shown in Figure 6 would be
used to generate the hint "Do the hostile forces have B1-bombers?" Because it is often
hard to image what text will be produced when a hint template is instantiated, the user
can toggle between the template and an English version. The English version presents
one concrete example of the hint that would be generated given a particular state of
working memory (e.g., the state named "state03-Unidenfified Aircraft" as selected in the
pull-down menu.)
Illustrating the Cognitive Model Visualizer
Now we illustrate our plans for a tool, called the Cognitive Model Visualizer, for which
we have already built a working prototype. In developing production rule models, one
must pay attention to how the rules work together to achieve the desired behavior.
However, keeping track of the flow of control in a production rule model or the algorithm
implemented by a production rule model is difficult to do, even for experienced cognitive
21
model developers, especially when the model grows to contain hundreds of rules.
Production systems are said to be “opaque” in this regard (Barr & Feigenbaum, 1989,
Sec. III.C.4). The Cognitive Model Visualizer is meant to help a cognitive model author
get an overview of how a set of production rules work together and interact.
Figure 7: Using the Cognitive Model Visualizer to diagnose why an expected
production rule firing did not occur.
We have developed a first version of the Cognitive Model Visualizer, illustrated
in Figure 7 using a small production rule model for place-value addition. The tool gives
the Cognitive Tutor author a dynamic view of a production rule model, as that model is
being used in the context of a given problem. After every production system cycle, the
tool gives an overview of the production rule instantiations that were generated in that
cycle. This information is rendered as a tree of production rule instantiations. The nodes
with lower case names represent production rule instantiations, the CHAIN nodes
represent choice points. Each path through the tree represents a sequence of production
rules, one of which will be selected for execution. This tree is thus analogous to the
22
“conflict set” in other production rule systems1. The Cognitive Tutor author can request
to see each production rule instantiation in more detail, simply by clicking on it. This
information helps an author to keep close tabs on what happens when a model is being
executed.
The Cognitive Model Visualizer also makes it easier to debug production rule
models: It shows incomplete production rule paths that were considered but abandoned
by the production rule interpreter. Such paths end in nodes marked by “***”. Incomplete
paths can be a sign that there are errors in one or more production rules. The Cognitive
Model Visualizer makes it easy to obtain further information that helps in diagnosing the
errors. For example, the tool makes it easy to investigate situations where a production
rule that was expected to fire did not fire. This is a common situation confronting
developers of cognitive models.
When this happens, the Cognitive Tutor author must first determine at which
point in the tree (i.e., at which CHAIN node) the rule was supposed to have fired. When
the author clicks on that CHAIN node, a menu pops up listing all production rules that
were considered at that particular point in the chain. The author can select the rule that
was expected to fire at that point. The tool prints out all partial instantiations of the
selected rule. This information is almost always sufficient to diagnose the error in the
production rule. Previously, the same information could be obtained only in a much
more cumbersome and time-consuming manner.
1
The conflict resolution mechanism in the TDK production system is somewhat unusual in that it considers
a set of production rule paths rather than simply a set of production rules, as do other production rule
systems. The need to consider paths stems from the requirements of model tracing. In model tracing, one
asks, given an observed action by the student in the system’s user interface, which production rule firings
could account for the action. An observable action may involve a significant amount of cognitive effort that
is best modeled by means of multiple production rules. Therefore, one does not want to make the
assumption that each observable action corresponds to the firing of a single production rule.
23
For the future, our plans are first, to extend the Cognitive Model Visualizer
further so that it provides further information, second, to integrate it with the other tools
in particular, the Execution Space Editor, and finally, to extend the Cognitive Model
Visualizer so that it provides a static view of a production rule model. This will be a
great help in interpreting the information currently presented.
Formative Evaluation and Iterative Design
Our design will be informed by formative evaluation data on student usage that we have
already started to collect and has driven the existing ideas. We will use contextual
inquiry and think aloud user study techniques (Beyer & Holtzblatt, 1998; Gomoll, 1990).
One source of feedback from potential new Cognitive Tutor authors is the upcoming
repeat of the Summer School discussed above. A second source of feedback is from
physics instructors at the Naval Academy who have been working to create intelligent
tutors.
Conclusion: Goals and Benefits of This Project
This project is an initial step in a broader research agenda designed to test two classes of
hypotheses. First, we plan to test hypotheses about key features of Cognitive Tutors that
yield greater learning yet require a greater development cost than traditional computeraided instruction. Second, we plan to test hypotheses about development and debugging
tools that will dramatically lower the costs of cognitive tutor development. The goal of
the current Phase 1 of project is to create the initial set of tools that will facilitate these
experiments (see the Statement of Work below).
24
Statement of Work for Phase 1 of Project
Time span
Activity
April 1 to
May 1
April 15 to
May 15
May 15 to
July 1
June 1 to
July 15
July 15 to
July 29
July 29 to
August 15
July 15 to
September 1
August 15 to
September 1
September 1
to Sept. 30
Design of integrated suite of all four tools using
storyboards and perform usability inspections.
Implementation of Working Memory Editor
Initial implementation of the Production Rule
Editor in preparation for Summer School
Further implementation and integration of the
Cognitive Model Visualizer
Initial use and formative evaluation of tools at
CIRCLE Summer School
Redesign of Production Rule Editor based on
Summer School feedback
Implementation of Execution Space Editor
Further formative evaluation of tools and
iterative redesign
Demonstration of the tools on the example
domains selected initially plus one domain
selected post-hoc
Project Members
Involved
KK, NH, VA
KK, NH,
programmers
KK, NH,
programmers
VA, programmers
All
KK, NH,
programmers
VA, programmers
All
All
We are looking forward to writing a grant renewal to pursue our broader research
agenda. We intend to engage in both formative and summative evaluation of the tools we
develop and to disseminate the tools to other educational technology developers in
universities, industry, and the military. Our evaluation and dissemination goals will be
facilitated by Koedinger's Cognitive Tutor course at CMU, the on-going CIRCLE
Summer School, and a new ITS course that Heffernan will start Spring 2003 at Worcester
Polytechnic Institute. Our long-term goals are to 1) make it possible for undergraduate
cognitive science and education students to create cognitive tutors and 2) to cut Cognitive
Tutor development time by an order of magnitude.
25
References
Aleven, V. & Koedinger, K.R. (in press). An effective metacognitive strategy: Learning by doing and
explaining with a computer-based Cognitive Tutor. Cognitive Scienc 26(2).
Anderson, J.R., Conrad, F.G. & Corbett, A.T. (1989). Skill acquisition and the LISP Tutor. Cognitive
Science, 13, 467-505.
Anderson, J. R., & Lebiere, C. (1998). The Atomic Components of Thought. Hillsdale, NJ: Erlbaum.
Anderson, J.R., & Pelletier, R. (1991). A development system for model-tracing tutors. In Proceedings of
the International Conference of the Learning Sciences, 1-8.
Barr, A., & Feigenbaum, E.A. (Eds.). (1989). The Handbook of Artificial Intelligence, Vol 1. Reprint.
Reading, MA: Addison-Wesley.
Beyer, H. & Holtzblatt, K. (1998). Contextual Design: Defining Customer-Centered Systems. San
Francisco, CA: Morgan Kaufman.
Bloom, C. P., Sparks, R., Dooley, S., Meiskey, L., Bell, B., & McClard, A. (1995). Putting intelligent
tutoring systems technology into practice. In Proceedings of the 7th World Conference on Artificial
Intelligence in Education, (pp. 405-412). Charlottesville, VA: Association for the Advancement of
Computing in Education.
Card, S.K., Moran, T.P., & Newell, A (1983). The Psychology of Human-Computer Interaction. Hillsdale,
NJ: Lawrence Erlbaum Associates.
Corbett, A.T. & Anderson, J.R. (1995). Knowledge tracing: Modeling the acquisition of procedural
knowledge. User modeling and user-adapted interaction, 4, 253-278.
Corbett, A. T., Koedinger, K. R., & Hadley, W. H. (2001). Cognitive Tutors: From the research classroom
to all classrooms. In Goodman, P. S. (Ed.) Technology Enhanced Learning: Opportunities for Change,
(pp. 235-263). Mahwah, NJ: Lawrence Erlbaum Associates.
Corbett, A. T., McLaughlin, M. & Scarpinatto, K.C. (2000). Modeling student knowledge: Cognitive
tutors in high school and college. User modeling and user-adapted interaction, 10, 81-108.
26
Ericsson, K. A., & Simon, H. A. (1984). Protocol Analysis: Verbal Reports as Data. Cambridge, MA:
The MIT Press.
Flanagan (1954), Psychological Bulletin, 51 (4), 327-358.
Gluck, K. (1999). Eye movements and algebra tutoring. Doctoral dissertation. Psychology Department,
Carnegie Melon University.
Gomoll, K., (1990). Some techniques for observing users. In Laurel, B. (ed.), The Art of Human-Computer
Interface Design, Addison-Wesley, Reading, MA, pp. 85-90.
Koedinger, K. R., & Anderson, J. R. (1998). Illustrating principled design: The early evolution of a
cognitive tutor for algebra symbolization. In Interactive Learning Environments, 5, 161-180.
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to
school in the big city. International Journal of Artificial Intelligence in Education, 8, 30-43.
Koedinger, K. R. & Corbett, A. T. (1997). RIDES Usability Evaluation. Final Report for Office of Naval
Research.
Koedinger, K. R., Corbett, A. T., Ritter, S., Shapiro, L. J. (2000). Carnegie Learning’s Cognitive Tutor:
Summary Research Results. White Paper. Pittsburgh, PA: Carnegie Learning.
Koedinger, K. R., & Sueker, E. L. F. (1996). PAT goes to college: Evaluating a cognitive tutor for
developmental mathematics. In Proceedings of the Second International Conference on the Learning
Sciences, (pp. 180-187). Charlottesville, VA: Association for the Advancement of Computing in
Education.
Koedinger, K. R., Suthers, D. D., & Forbus, K. D. (1998). Component-based construction of a science
learning space. In Goettl, B. P., Halff, H. M., Redfield, C. L., & Shute, V. J. (Eds.) Intelligent
Tutoring Systems, Proceedings of the Fourth International Conference, (pp. 166-175). Lecture Notes
in Computer Science, 1452. Berlin: Springer-Verlag.
Lajoie, S. P. (1993). Computer environments as cognitive tools for enhancing learning. In Lajoie, S., &
Derry, S. (Eds.) Computers as Cognitive Tools. Hillsdale, NJ: Erlbaum.
27
Mathan, S., Koedinger, K. R., Corbett, A., & Hyndman, A. (2000). Effective strategies for bridging gulfs
between users and computer systems. In Proceedings of HCI-Aero 2000: International Conference on
Human Computer Interaction in Aeronautics. (pp 197 – 202). Toulouse, France.
Miller, C. S., Lehman, J. F., & Koedinger, K. R. (1999). Goals and learning in microworlds. Cognitive
Science, 23, (3), 305-336.
Munro, A. (1994). RIDES Authoring Reference. Behavioral Technology Laboratories, USC
(http://btl.usc.edu/rides/)
Murray, T.(1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International
Journal of Artificial Intelligence in Education, 10, pp. 98-129.
Newell, A. (1990). Unified Theories of Cognition. Cambridge, MA: Harvard University Press.
Nielsen J. & Mack, R.L. (1994). Usability Inspection Methods. New York: John Wiley & Sons, 1994.
Palmiter, S., & Elkerton, J. (1993). Animated demonstrations for learning procedural computer-based
tasks. Human-Computer Interaction, 8, 193-216. Hillsdale, NJ: Erlbaum.
Pane, J.F., Corbett, A.T., and John, B.E. (1996). Assessing dynamics in computer-based instruction. In
Proceedings of ACM CHI'96 Conference on Human Factors in Computing Systems. Vancouver, pp.
197-204.
Rittle-Johnson, B. & Koedinger, K. R. (2001). Using cognitive models to guide instructional design: The
case of fraction division. In Proceedings of the Twenty-Third Annual Conference of the Cognitive
Science Society, (pp. 857-862). Mahwah,NJ: Erlbaum.
Sleeman, D. (1986). Introductory algebra: A case study of student misconceptions. Journal of
Mathematical Behavior, 5, 25-52.
Sohn, M., Douglass, S. A., Chen, C. & Anderson, J. R. (in submission). Unit task execution with a dynamic
display in a complex problem-solving situation.
Stasko, J., Badre, A., & Lewis, C. (1993). Do algorithm animations assist learning? An empirical study
and analysis. Proceedings of ACM INTERCHI'93 Conference on Human Factors in Systems,
Amsterdam, Netherlands, 61-66.
28