43
Cognitive Task Analysis
Richard E. Clark
University of Southern California, Los Angeles, California
David F. Feldon
University of South Carolina, Columbia, South Carolina
Jeroen J. G. van Merriënboer
Open University of The Netherlands, Heerlen, The Netherlands
Kenneth A. Yates
University of Southern California, Los Angeles, California
Sean Early
University of Southern California, Los Angeles, California
CONTENTS
Introduction .....................................................................................................................................................................578
Types of Cognitive Task Analysis Currently in Use ......................................................................................................579
Cognitive Task Analysis Families .........................................................................................................................579
Varieties of CTA Methods and Their Applications ........................................................................................................579
Collect Preliminary Knowledge ............................................................................................................................580
Document Analysis ......................................................................................................................................580
Observations.................................................................................................................................................580
Unstructured Interviews ...............................................................................................................................580
Identify Knowledge Representations ....................................................................................................................580
Learning Hierarchy Analysis .......................................................................................................................581
Apply Focused Knowledge Elicitation Methods.........................................................................................581
Concepts, Processes, and Principles ............................................................................................................581
Critical Decision Method.............................................................................................................................582
Analyze and Verify Data Acquired .......................................................................................................................582
Format Results for the Intended Application........................................................................................................582
577
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
Current Research Evidence for the Impact of Cognitive Task Analysis .......................................................................583
Declarative Knowledge and CTA..........................................................................................................................583
Procedural Knowledge and CTA...........................................................................................................................584
Cues........................................................................................................................................................................584
Decision Points ......................................................................................................................................................585
Cognitive Skills......................................................................................................................................................585
Instructional Evidence ...........................................................................................................................................586
Generalizability of CTA-Based Training Benefits ................................................................................................586
Cost–Benefit Studies of CTA ................................................................................................................................586
Integrating Cognitive Task Analysis and Training Design.............................................................................................587
Optimal Integration of CTA and Training Design................................................................................................587
Learning Tasks .............................................................................................................................................587
Supportive Information ................................................................................................................................587
Just-in-Time information .............................................................................................................................588
Part-Task Practice.........................................................................................................................................588
Cognitive Task Analysis and 4C/ID Model....................................................................................................................588
Decomposition of the Complex Skill....................................................................................................................588
Sequencing Task Classes .......................................................................................................................................588
Analyze the Nonrecurrent Aspects of the Complex Skill ....................................................................................588
Analyze the Recurrent Aspects of the Complex Skill ..........................................................................................589
The Next Generation of Research on Cognitive Task Analysis.....................................................................................589
First Principles of Cognitive Task Analysis..........................................................................................................589
Research on Automated, Unconscious Expert Knowledge ..................................................................................590
Cost Effectiveness and Cost–Benefit Research.....................................................................................................590
Conclusion.......................................................................................................................................................................591
References .......................................................................................................................................................................591
ABSTRACT
This chapter presents an overview of the current state
of cognitive task analysis (CTA) in research and practice. CTA uses a variety of interview and observation
strategies to capture a description of the explicit and
implicit knowledge that experts use to perform complex tasks. The captured knowledge is most often
transferred to training or the development of expert
systems. The first section presents descriptions of a
variety of CTA techniques, their common characteristics, and the typical strategies used to elicit knowledge
from experts and other sources. The second section
describes research on the impact of CTA and synthesizes a number of studies and reviews pertinent to
issues underlying knowledge elicitation. In the third
section, we discuss the integration of CTA with training design. Finally, in the fourth section, we present a
number of recommendations for future research and
conclude with general comments.
KEYWORDS
Automated knowledge: About how to do something;
with repetition, it operates outside of conscious
awareness and executes much faster than conscious
processes.
578
Cognitive task analysis: Interview and observation
protocols for extracting implicit and explicit
knowledge from experts for use in instruction and
expert systems.
Complex tasks: Tasks where performance requires the
integrated use of both controlled and automated
knowledge to perform tasks that often extend over
many hours or days.
Declarative knowledge: Knowledge about what or
why; hierarchically structured; formatted as propositional, episodic, or visuospatial information that
is accessible in long-term memory and consciously
observable in working memory.
Subject-matter expert (SME): A person with extensive
experience who is able to perform a class of tasks
rapidly and successfully.
INTRODUCTION
Cognitive task analysis is the extension of traditional task
analysis techniques to yield information about the knowledge, thought processes and goal structures that underlie
observable task performance. [It captures information
about both] … overt observable behavior and the covert
cognitive functions behind it [to] form an integrated
whole. (Chipman et al., 2000, p. 3)
Cognitive Task Analysis
Cognitive task analysis (CTA) uses a variety of interview and observation strategies to capture a description
of the knowledge that experts use to perform complex
tasks. Complex tasks are defined as those for which
their performance requires the integrated use of both
controlled (conscious, conceptual) and automated
(unconscious, procedural, or strategic) knowledge to
perform tasks that often extend over many hours or
days (van Merriënboer et al., 2002). CTA is often only
one of the strategies used to describe the knowledge
required for performance. It is a valuable approach
when advanced experts are available who reliably
achieve a desired performance standard on a target task
and the goal is to capture the cognitive knowledge used
by them (Clark and Estes, 1999). Analysts use CTA to
capture accurate and complete descriptions of cognitive processes and decisions. The outcome is most
often a description of the performance objectives,
equipment, conceptual knowledge, procedural knowledge, and performance standards used by experts as
they perform a task. The descriptions are formatted so
they can be used as records of task performance and
to inform novices in a way that helps them achieve the
performance goals in any context. CTA is most often
performed before (or as an integral part of) the design
of instruction, work, job aids, or tests. The descriptions
are then used to develop expert systems, tests to certify
job or task competence, and training for acquiring new
and complex knowledge for attainment of performance
goals (Chipman et al., 2000; Jonassen et al., 1999).
TYPES OF COGNITIVE TASK
ANALYSIS CURRENTLY IN USE
Researchers have identified over 100 types of CTA
methods currently in use, which can make it difficult
for the novice practitioner to choose the appropriate
method (Cooke, 1994). The number and variety of CTA
methods are due primarily to the diverse paths that the
development of CTA has taken. It has origins in behavioral task analysis, early work in specifying computer
system interfaces, and in military applications—each
with its own demands, uses, and research base. Over
the past 20 years, CTA has been increasingly informed
by advances in cognitive science and has become an
important component for the design of systems and
training in many domains. The growing body of literature describing CTA methods, applications, and
results mirrors the diverse application and development
of CTA methods; however, reviews and classifications
are available to guide those interested in exploring and
applying CTA, including a comprehensive review of
reviews provided by Schraagen et al. (2000).
Cognitive Task Analysis Families
Cooke (1994) conducted one of the more extensive
reviews of CTA. She identified three broad families of
techniques: (1) observation and interviews, (2) process
tracing, and (3) conceptual techniques. Observations
and interviews involve watching experts and talking
with them. Process tracing techniques typically capture an expert’s performance of a specific task via
either a think-aloud protocol or subsequent recall. In
contrast, conceptual techniques produce structured,
interrelated representations of relevant concepts within
a domain.
Cooke’s (1994) three families of techniques differ
in terms of their specificity and formality. Generally,
observations and interviews are informal and allow
knowledge elicitors much flexibility during elicitation.
Process tracing methods have more structure and specificity, although some analysis decisions are left to the
elicitor. Conceptual techniques are well specified and
formal, with few judgments on the part of the elicitor.
As a further comparison, more formal methods require
greater training on the mechanisms and produce more
quantitative data compared to the informal methods,
which focus on interview skills and generate qualitative output. Because different techniques may result in
different aspects of the domain knowledge, Cooke recommends the use of multiple methods, a recommendation often echoed throughout the CTA literature no ref
for
(Ericsson and Simon, 1993; Russo et al., 1989; Vos- Vosniadou
niadou, 1994).
Wei and Salvendy’s (2004) review of CTA methods introduced a fourth family—formal models—
which uses simulations to model tasks in the cognitive
domain. Their review further differs from others in that
they provided practical guidelines on how to use the
classifications of CTA methods to select appropriate
techniques to accomplish various objectives. One
guideline, for example, suggests that when tasks or
jobs do not have a defined domain, observations and
interviews are especially useful in the initial phase of
CTA to generate a more explicit context and identify
boundary conditions.
VARIETIES OF CTA METHODS
AND THEIR APPLICATIONS
These reviews provide a starting point to explore the
numerous varieties of CTA methods and their applications. We examine the overall CTA process and describe
in depth some methods that have particular application
to instructional design. Although there are many varieties of CTA methods, most knowledge analysts follow a
579
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
five-stage process (Chipman et al., 2000; Clark, 2006;
Coffey and Hoffman, 2003; Cooke, 1994; Crandall et
al., 2006; Hoffman et al., 1995; Jonassen et al., 1999).
The five common steps in most of the dominant CTA
methods are performed in the following sequence:
• Collect preliminary knowledge.
• Identify knowledge representations.
• Apply focused knowledge elicitation methods.
• Analyze and verify data acquired.
• Format results for the intended application.
The following sections contain descriptions of common CTA methods and brief explanations of each type
as it is used during each stage of the general process.
Collect Preliminary Knowledge
In this initial stage, the analyst identifies the sequence
of tasks that will become the focus of the CTA. Analysts attempt to become generally familiar with the
knowledge domain and identify experts to participate
in the knowledge elicitation process. Although knowledge analysts and instructional developers do not need
to become subject-matter experts (SMEs) themselves,
they should be generally familiar with the content,
system, or procedures being analyzed. If possible, two
or more subject-matter experts should be selected to
participate in the process (Chao and Salvendy, 1994;
Lee and Reigeluth, 2003). Although specific criteria
for identifying experts may change depending on circumstances,* all SMEs must have a solid record of
successful performance at the tasks being analyzed.
Experts are most often interviewed separately to avoid
premature consensus regarding the knowledge and
skills necessary for effective performance. Techniques
typically used during this phase include document
analysis, observation, and interviews (structured or
unstructured). The analyst uses the results of this stage
to identify the knowledge types and structures involved
in performing the tasks.
Document Analysis
Analysts often begin their reviews by collecting any
available written resources describing the tasks and
related subject matter. This can include a wide variety
of documents, such as promotional literature, bro* See extensive discussions of appropriate definitions of expertise and
criteria for identifying experts in Cooke (1992), Dawes (1994), Ericsson
and Smith (1991), Glaser and Chi (1988), Mullin (1989), and Sternberg
and Horvath (1998).
580
chures, manuals, employee handbooks, reports, glossaries, course texts, and existing training materials. These
documents are analyzed for orientation on the tasks,
preparation of the in-depth analysis, and confirmation
of preliminary ideas (Jonassen et al., 1999). This orientation prepares analysts for subsequent task analysis
activities; for example, the information elicited during
structured interviews may be more robust when analysts are already familiar with experts’ terminology.
Documentation analysis also allows comparison of
existing materials on a procedure with accounts of
expert practitioners to identify any immediate discrepancies between doctrine and typical implementation.
Observations
Observation is one of the most frequently used and
most powerful tools of knowledge elicitation. It can
be used to identify the tasks involved, possible limitations and constraints for subsequent analysis, and
available information necessary to perform the task. It
also allows analysts to compare an expert’s description
of the task with actual events. In many CTA systems,
analysts will unobtrusively observe experts while they
are performing the tasks under examination to expand
their understanding of the domain. Analysts observe
and record the natural conditions and actions during
events that occur in the setting (Cooke, 1994).
Although definitive identification of an expert’s mental
operations cannot be accomplished through observation, analysts may note the occasions on which it
seems that experts must make decisions, assess situations, or engage in analysis.
Unstructured Interviews
“The most direct way to find out what someone knows
is to ask them” (Cooke, 1999, p. 487). In addition to
observation, unstructured interviews are also common
early in the CTA process to provide an overview of
the domain and to raise issues and questions for exploration in subsequent structured interviews. In unstructured interviews, analysts may not dictate the content
or sequence of conversation. In other instances, however, they may ask an expert to focus on a task, event,
or case with instructions to “tell me everything you
know about….”
Identify Knowledge Representations
Using the information collected during the preliminary
stage, analysts examine each task to identify subtasks
and types of knowledge required to perform the task.
Most CTA approaches are organized around knowl-
Cognitive Task Analysis
edge representations appropriate for the task, such as
concept maps, flow charts, semantic nets, and so forth.
These representations provide direction and order to
latter stages in the CTA process because knowledge
elicitation methods map directly to knowledge types.
Some are best used to elicit procedural knowledge,
while others are more successful for capturing declarative knowledge (Chipman et al., 2000). A learning
hierarchy is one example of a method to organize the
types of knowledge required to perform a task.
or the use of repertory grids (Bradshaw et al., 1993).
It is the variation in these specific techniques that
defines the major differences between specific CTA
models. Although the methods may differ in focus,
they share a common purpose in capturing the conditions and cognitive processes for necessary for complex problem solving. Following are descriptions of
two CTA models that have been documented to effectively elicit experts’ knowledge in a manner that is
particularly effective for instruction (Crandall and
Getchell-Reiter, 1993; Velmahos et al., 2004).
Learning Hierarchy Analysis
A learning hierarchy analysis represents the content of
skills ordered from more complex problem-solving
skills at the top to simpler forms of learning (Gagné,
1962, 1968; Jonassen et al., 1999), so, for example,
problem solving is followed by rule learning, which is
followed by concepts. Thus, the basic idea is that people can only learn rules if they have already mastered
the prerequisite concepts necessary to learn the rules.
Analyzing a learning hierarchy begins by identifying
the most complex (highest) learning outcome and then
determining the underlying skills that must be mastered to achieve the target outcome. A hierarchy of
skills is represented as a chart of tasks for each intellectual skill that is acquired to progress to increasingly
complex skills. The learning hierarchy constructed at
this stage of the CTA process provides the guide to
structure the next stage of knowledge elicitation by
identifying the information that must be captured from
the SMEs. Thus, it reflects the reiterative nature of the
CTA process, in which the details of the knowledge,
skills, and cognitive strategies necessary for complex
learning are revealed, refined, and confirmed.
Apply Focused Knowledge Elicitation Methods
During knowledge elicitation, the analyst applies various techniques to collect the knowledge identified in
the prior stage. Past research indicates that different
elicitation methods yield different types of knowledge
and that knowledge is rarely articulated without being
the focus of elicitation (Crandall et al., 2006; Hoffman
et al., 1998). Analysts attempt to choose methods
appropriate to the targeted knowledge type as determined by the knowledge representations identified for
each task; consequently, most elicitation efforts entail
multiple techniques. Among the many types of knowledge elicitation methods, variations of structured and
semi-structured interviews are most commonly
involved in CTA because they are relatively easy to
use and require less training than more formal methods
such as protocol analysis (Ericsson and Simon, 1993)
Concepts, Processes, and Principles
Gathering concepts, processes, and principles (CPPs)
(Clark, 2004, 2006) involves a multi-stage interview
technique that captures the automated and unconscious
knowledge acquired by experts through experience and
practice. Multiple SMEs describe the same procedure,
followed by cycles of expert self-review and peer
review. The initial, semi-structured interview begins
with a description of the CTA process by the analyst.
The SME is then asked to list or outline the performance sequence of all key subtasks necessary to perform the larger task being examined. SMEs are also
asked to describe (or help the interviewer locate) at
least five authentic problems that an expert should be
able to solve if they have mastered the task. Problems
should range from routine to highly complex whenever
possible. The resulting sequence of tasks becomes the
outline for the training to be designed or the job
description produced after the CTA is completed.
Starting with the first subtask in the sequence, the
analyst asks a series of questions to collect:
• The sequence of actions (or steps) necessary
to complete the subtask
• The decisions that have to be made to complete the subtask, when each must be made,
the alternatives to consider, and the criteria
used to decide among the alternatives
• All concepts, processes, and principles that
are the conceptual basis for the experts’
approach to the subtask
• The conditions or initiating events that must
occur to start the correct procedure
• The equipment and materials required
• The sensory experiences required (e.g., the
analyst asks if the expert must smell, taste,
or touch something in addition to seeing or
hearing cues in order to perform each subtask)
• The performance standards required, such as
speed, accuracy, or quality indicators
581
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
The interview is repeated for each SME; each interview is recorded and transcribed verbatim for later
analysis.
Critical Decision Method
The critical decision method (CDM) (Klein et al.,
1989) is a semi-structured interview method that uses
a set of cognitive probes to determine the bases for
situation assessment and decision making during critical (nonroutine) incidents (for a full procedural
description, see Hoffman et al., 1998). CDM is based
on the concept of expert decision making as the recognition of cue patterns in the task environment without conscious evaluation of alternatives; thus, situational awareness plays a dominant role in experts’
selection of courses of action. The speed with which
such decisions are made suggests that experts unconsciously assess feasible goals, important cues, situational dynamics, courses of action, and expectancies.
To elicit this knowledge, CDM uses a retrospective,
case-based approach with elicitation occurring in multiple sweeps to gather information in progressively
deepening levels of detail.
The technique begins by selecting a critical incident from the expert’s task experience that was unusual
in some way. The experts involved provide unstructured accounts of the incident, from which a timeline
is created. Next, the analyst and the experts identify
specific points in the chronology at which decisions
were made. These decision points are defined as
instances when other reasonable alternative courses of
action were possible. The decision points are then
probed further using questions that elicit: (1) the perceptual cues used in making the decision, (2) prior
knowledge that was applied, (3) the goals considered,
(4) decision alternatives, and (5) other situation assessment factors. The reports are recorded and transcribed
verbatim.
Following coding, the formatted output is presented to the participating SMEs for verification,
refinement, and revision to ensure that the representations of tasks and their underlying cognitive components are complete and accurate. Once the information
in the formatted output has been verified or revised by
the expert, the analyst should then compare it with the
output of other experts to verify that the results accurately reflect the desired knowledge representation.
The analysis stage in CPP (Clark, 2004, 2006)
begins with the analyst preparing a summary of the
interview in a standard format that includes the task,
a list of subtasks, and the conditions, standards, equipment, and materials required. For each subtask, the
analyst then writes a procedure that includes each
action step and decision step required to perform the
task and gives the procedure to the SME to review. To
verify the individual CTAs, the analyst gives each subject-matter expert’s product to one of the other SMEs
and asks that person to edit the document for accuracy
and efficiency (that is, to determine the fewest steps
necessary for a novice with appropriate prior knowledge to perform the task). In the final stage, the analyst
edits the individual CTAs into one formatted description of how to accomplish all tasks. After final approval
by the SMEs, this final, formatted document provides
the information for the instructional design process.
Clark (2006) provides the format of the protocol.
The critical decision method prescribes no single
method for coding the transcripts that are transcribed
verbatim from the recorded interviews, as each specific
research question defines how the transcripts are coded
(Klein et al., 1989). The coding scheme, however,
should be domain relevant and have cognitive functionality; in other words, it should tag information that
represents perceptual cues, decision points, and situational assessments. A sample of a coded protocol can
be found in Hoffman and colleagues (1998).
Format Results for the Intended Application
Analyze and Verify Data Acquired
As noted above, CTA methods vary in structure, formality, and results. Because the knowledge elicitation
techniques described here are less formal, they require
that the analyst code and format the results for verification, validation, and applicability for use in their
intended application. When conducting interviews
with experts, practitioners recommend recording the
interviews and transcribing them for review at a later
time, rather than trying to take detailed notes during
the interview, which may distract from the process.
Transcripts may be coded to summarize, categorize,
or synthesize the collected data.
582
The results of some highly structured CTA methods
(e.g., cognitive modeling) are readily applied to expert
systems or computer-assisted tutoring applications.
For less formal CTA methods, such as those described
here, the results must be translated into models that
reveal the underlying skills, mental models, and problem-solving strategies used by experts when performing highly complex tasks. Further, these models inform
the instructional design of curriculum, training and
other performance applications. The concepts, processes, and principles (Clark, 2004, 2006) method generates a description of the conceptual knowledge, conditions, and a detailed list of the actions and decisions
Cognitive Task Analysis
necessary to perform a task. These products can be
incorporated into an instructional design system. Similarly, products resulting from the application of the
critical decision method have been used for a variety
of instructional applications, including building and
evaluating expert systems and identifying training
requirements. CDM can provide case studies and
information regarding which aspects of a task depend
on explicit knowledge and which depend on tacit
knowledge (Klein et al., 1989).
CURRENT RESEARCH EVIDENCE
FOR THE IMPACT OF
COGNITIVE TASK ANALYSIS
Modern CTA evolved from a behavioral approach to
analyzing performance. As the understanding of occupational demands evolved from a focus on physical
performance to a focus on cognitive performance, evidence suggested that key aspects of performance
entailed knowledge that was not directly observable
(Ryder and Redding, 1993; Schneider, 1985). Applications of behavioral task analysis to training resulted
in incomplete descriptions that led to decision errors
during job performance (Schraagen et al., 2000). Early
versions of CTA were designed to capture the decisions and analyses that could not be directly observed
as well as the deeper conceptual knowledge that served
as the basis for analytical strategies and decisions
(Clark and Estes, 1999). Thus, training shifted from
the reinforcement of associations between perceptual
stimuli and behaviors to the development of declarative and procedural knowledge.
Research evidence indicates that the accurate identification of experts’ cognitive processes can be
adapted into training materials that are substantially
more effective than those developed through other
means (Merrill, 2002; Schaafstal et al., 2000; Velmahos et al., 2004). When content is inaccurate or incomplete, any instruction based on that knowledge will be
flawed (Clark and Estes, 1996; Jonassen et al., 1999).
Such flaws interfere with performance and with the
efficacy of future instruction (Lohman, 1986; Schwartz
and Bransford, 1998). Resulting misconceptions resist
correction, despite attempts at remediation (Bargh and
Ferguson, 2000; Chinn and Brewer, 1993; Thorley and
Stofflet, 1996).
Declarative Knowledge and CTA
Declarative knowledge is hierarchically structured,
propositional, episodic, visuospatial information that
is accessible in long-term memory and consciously
observable in working memory (Anderson, 1983;
Anderson and Lebiere, 1998; Gagné et al., 1992). This
type of knowledge supports performance through the
conceptual understanding of processes and principles
related to a task and the role that the task plays within
its broader context (Gagné, 1982). SMEs possess
extensive declarative knowledge of their domains in
the form of principled frameworks of abstract,
schema-based representations. These frameworks
allow experts to analyze complex problems efficiently
(Glaser and Chi, 1988; Zeitz, 1997). These elaborate
schemas enable experts to retain and recall information, events, and problem states with a high degree of
accuracy (Cooke et al., 1993; Dochy et al., 1999;
Ericsson and Kintsch, 1995). Further, broad, principled understandings of their domains facilitate skill
transfer to solve related novel and complex problems
(Gagné and Medsker, 1996; Hall et al., 1995; van
Merriënboer, 1997).
When communicated to novices, the organization
of experts’ knowledge also impacts training outcomes.
In an examination of experts’ instructions to novices,
Hinds et al. (2001) found that trainees who received
explanations from experts performed better on transfer
tasks than trainees who received their explanations
from non-experts. The experts provided explanations
that were significantly more abstract and theoretically
oriented than those of the non-experts, so learners in
the expert-to-novice instructional condition were able
to solve transfer problems more quickly and effectively
than their counterparts in the non-expert-to-novice
instructional condition.
Conceptual knowledge alone, however, is insufficient for generating effective performance. The nonexpert instructors in the study provided more concrete,
procedural explanations, which facilitated higher performance by trainees when they attempted to perform
the original target task. The abstractions provided by
the experts lacked key details and process information
necessary for optimal performance. This finding is
consistent with many others in the training literature,
suggesting that the most effective learning occurs
when all necessary information is available to the
learner in the form of instruction and/or prior knowledge (for a review, see Kirschner et al., 2006).
Findings from a variety of studies indicate that
without CTA to facilitate knowledge elicitation,
experts in many fields unintentionally misrepresent the
conceptual knowledge on which they base their performance. In a study by Cooke and Breedin (1994),
for example, expert physicists attempted to predict the
trajectories of various objects and provided written
explanations of the methods by which they reached
their conclusions; however, when the researchers
583
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
attempted to replicate the physicists’ predictions on
the basis of the explanations provided, they were
unable to attain the same results. The calculated trajectories were significantly different from those provided by the experts.
In a similar study, expert neuropsychologists evaluated hypothetical patient profiles to determine their
theoretical levels of intelligence (Kareken and Williams, 1994). Participants first articulated the relationships between various predictor variables (e.g., education, occupation, gender, and age) and intelligence.
They then estimated IQ scores on the basis of values
for the predictor variables they identified; however,
their estimates differed significantly from the correlations they provided in their explanations of the relationships among predictor variables. Many were completely uncorrelated. Clearly, the experts’ performance
relied on processes that were very different from their
declarative knowledge of their practice.
Procedural Knowledge and CTA
Procedural knowledge is required for all skilled performance. Skill acquisition often begins with learning
declarative knowledge about discrete steps in a procedure; yet, the development of automaticity occurs as
we practice those procedures. The automatization process involves learning to recognize important environmental cues that signal when the skill is to be applied
and the association of the cues to the discrete covert
(cognitive) and overt (action) steps required to attain
a goal or subgoal (Neves and Anderson, 1981).
Through practice, these associations and steps increase
in reliability and speed of performance. Over time, the
procedures require diminishing levels of mental effort
or self-monitoring to perform until they utilize very
few, if any, cognitive resources (Wheatley and Wegner,
2001). This consistent, repeated mapping of conditional cues and steps manifests as an integrated if–then
decision rule between the cue (if) and the procedure
(then) necessary to attain a goal from a particular problem state (Schneider and Shiffrin, 1977). This representation is a production within the ACT-R cognitive
model of learning proposed by Anderson (Anderson,
1995; Anderson and Lebiere, 1998).
During complex tasks, multiple if–then productions are strung together to generate more sophisticated
hierarchies of performances. Each individual production attains a subgoal that is a component of the overall
goal. To move from one production to the next in a
sequence, the new subgoal must be identified and an
appropriate production selected. For novices, the identification and selection process for nearly every subgoal is a conscious, deliberate decision; however,
584
experts automate this process, so they cannot consciously identify many of these decision points (Blessing and Anderson, 1996).
Automaticity has two primary properties that limit
the effectiveness of unassisted explanations by experts.*
First, automated knowledge operates outside of conscious awareness, and executes much faster than conscious processes (Wheatley and Wegner, 2001). As such,
it is not available for introspection or accurate selfmonitoring. Second, automated processes are typically
uninterruptible, so they cannot be effectively changed
once they are acquired (Hermans et al., 2000); consequently, experts’ unaided self-reports of their problemsolving processes are typically inaccurate or incomplete** (Chao and Salvendy, 1994; Feldon, 2004).
Cues
Each element of an if–then production has great importance for effective training. For learners to develop
effective procedures, they must attend to relevant cues
to determine correctly which subgoals and procedures
are appropriate. Thus, incorporating experts’ knowledge of these cues is important for optimal instruction
(Fisk and Eggemeier, 1988; Klein and Calderwood,
1991). Crandall and Getchell-Reiter (1993), for example, investigated the procedural knowledge of expert
nurses specializing in neonatal intensive care for newborn or premature babies. The participants were 17
registered nurses who averaged 13 years of overall
experience and 8.1 years of specialization. Without a
formal knowledge elicitation technique, they
attempted to recall highly detailed accounts of critical
incidents or measures they had implemented that they
believed had positively influenced a baby’s medical
condition. After completing a free recall phase, the
researchers used CTA to identify additional relevant
information that the nurses did not articulate. Analysis
of the transcripts revealed that the CTA probes elicited
significantly more indicators of medical distress in the
babies than were otherwise reported. Before CTA, the
nurses’ explanations of the cues they used were either
* The literature on expertise has not reached a consensus on the role
of automaticity. however, much empirical evidence suggests that it plays
a defining role. See Feldon (in press) for an extensive review. Until the
article reaches publication, it can be found on the SpringerLink website
under digital object identifier (DOI) 10.1007/s10648-006-9009-0. A
prepublication draft can also be located at http://www.cogtech.usc.edu/
recent_publications.php.
** When experts attempt to solve novel problems, the elements of their
decision-making processes that are newly generated are less likely to
be reported inaccurately; however, preexisting processes that were
applied to those problems will continue to be subject to self-report errors
(Betsch et al., 1998).
Cognitive Task Analysis
omitted or articulated vaguely as “highly generalized
constellations of cues” (Crandall and Getchell-Reiter,
1993, p. 50).
Comparison of the elicited cues to those described
in the available medical and nursing training literature
of the time revealed that more than one third of the
cues (25 out of 70) used by the expert nurses in the
study to correctly diagnose infants were absent from
that literature. These cues spanned seven previously
unrecognized categories that were subsequently
incorporated into standard training for novice nurses
entering neonatal intensive care (Crandall and Gamblian, 1991).
Decision Points
In addition to knowing which cues are important for
decision making, it is also necessary to correctly identify the points at which those decisions must be made.
Much of the research on decision making suggests that
many decisions are made prior to awareness of the
need to make a decision (Bargh et al., 2001; Wegner,
2002). Abreu (1999) found that practicing psychotherapists evaluated fictitious case studies more negatively
when they were primed with material about AfricanAmerican stereotypes than when they rated the same
information without priming. Similarly, when Bargh
et al. (2001) subconsciously primed participants with
goals of either cooperation or high performance, the
actions of the participants in a variety of tasks typically
conformed to the subliminal goal despite their being
completely unaware of either the content of the prime
or the fact that they held the goal itself.
In professions, automaticity presents significant
problems for training if experts are relied upon to
explain the points at which decisions must be made.
In medicine, for example, studies of the reliability of
diagnoses by expert physicians for identical symptoms
presented at different times only correlated between
.40 and .50 (Einhorn, 1974; Hoffman et al., 1968).
Despite self-reports suggesting that the participants
considered extended lists of symptoms, analysis of the
symptoms in the cases presented indicated that only
one to four symptoms actually influenced diagnosis
decisions (Einhorn, 1974).
Some experts freely acknowledge that they are
unable to accurately recall aspects of their problemsolving strategies. Johnson (1983) observed significant
discrepancies between an expert physician’s actual
diagnostic technique and the technique that he articulated to medical students. Later, he discussed with the
physician why his practice and his explanation differed. The physician’s explanation for the contradiction was, “Oh, I know that, but you see, I don’t know
how I do diagnosis, and yet I need things to teach
students. I create what I think of as plausible means
for doing tasks and hope students will be able to convert them into effective ones” (Johnson, 1983, p. 81).
Cognitive Skills
Correctly identifying and explaining the sequences
of cognitive and psychomotor actions that are triggered by cues at decision points are likewise crucial
to effective instruction. Although psychomotor
aspects of a task are relatively simple for learners to
observe, cognitive operations require articulation for
a learner to successfully replicate an expert’s performance; however, automaticity often impairs this process. As an example, a team of engineers and technicians with expertise in the assembly of
sophisticated research equipment attempted unsuccessfully to generate a complete set of assembly
instructions, despite extensive and repeated efforts to
include every relevant fact, process, and heuristic
(Collins et al., 1985). When scientists who purchased
the equipment attempted to assemble it according to
the instructions, the equipment did not function. After
many discussions with the engineers, the scientists
eventually discovered that the expert team had accidentally omitted a necessary step from the instructions. The step turned out to be a universally implemented practice among the engineers and technicians
that they had failed to articulate.
Chao and Salvendy (1994) systematically documented the rates at which experts omit cognitive skills
from self-reports. Six expert programmers were asked
to complete a series of challenging troubleshooting
tasks, and all of their actions were recorded. The
programmers were then asked to explain their procedures using a variety of different knowledge elicitation methods. No single expert was able to report
more than 41% of their diagnostic actions, 53% of
their debugging actions, or 29% of their interpretations, regardless of the knowledge elicitation method
used; however, when the researchers began compiling
the elicited explanations from different experts, they
found that the percentage of actions explained
increased. When explanations from all six experts
were aggregated, the percentages of verbalization for
each category of actions increased to 87%, 88%, and
62%, respectively. The improvement in information
elicited reflects the experts’ individual differences in
which subgoal productions had been automated to
greater and lesser extents. Thus, one promising practice for instruction based on expert knowledge is to
employ CTA methods with multiple experts prior to
developing instruction.
585
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
Instructional Evidence
Several studies provide direct evidence for the efficacy
of CTA-based instruction. In a study of medical school
surgical instruction, an expert surgeon taught a procedure (central venous catheter placement and insertion)
to first-year medical interns in a lecture/demonstration/practice sequence (Maupin, 2003; Velmahos et al.,
2004). The treatment group’s lecture was generated
through a CTA of two experts in the procedure. The
control group’s lecture consisted of the expert instructor’s explanation as a free recall, which is the traditional instructional practice in medical schools. Both
conditions allotted equal time for questions, practice,
and access to equipment. The students in each condition completed a written post-test and performed the
procedure on multiple human patients during their
internships. Students in the CTA condition showed
significantly greater gains from pretest to post-test than
those in the control condition. They also outperformed
the control group when using the procedure on patients
in every measure of performance, including an observational checklist of steps in the procedure, number of
needle insertion attempts required to insert the catheter
into patients’ veins, frequency of required assistance
from the attending physician, and time to completion
for the procedure.
Similarly, Schaafstal et al. (2000) compared the
effectiveness of a preexisting training course in radar
system troubleshooting with a new version generated
from cognitive task analyses. Participants in both versions of the course earned equivalent scores on knowledge pretests; however, after instruction, students in the
CTA-based course solved more than twice as many malfunctions, in less time, as those in the traditional instruction group. In all subsequent implementations of the
CTA-based training design, the performance of every
student cohort replicated or exceeded the performance
advantage over the scores of the original control group.
Merrill (2002) compared CTA-based direct
instruction with a discovery learning (minimal guidance) format and a traditional direct instruction format
in spreadsheet use. The CTA condition provided direct
instruction based on strategies elicited from a spreadsheet expert. The discovery learning format provided
authentic problems to be solved and made an instructor
available to answer questions initiated by the learners.
The traditional direct instruction format provided
explicit information on skills and concepts and guided
demonstrations taken from a commercially available
spreadsheet training course. Scores on the post-test
problems favored the CTA-based instruction group
(89% vs. 64% for guided demonstration vs. 34% for
the discovery condition). Further, the average times to
586
completion also favored the CTA group. Participants
in the discovery condition required more than the allotted 60 minutes, the guided demonstration participants
completed the problems in an average of 49 minutes,
and the participants in the CTA-based condition
required an average of only 29 minutes.
Generalizability of CTA-Based
Training Benefits
Lee (2004) conducted a meta-analysis to determine
how generalizable CTA methods are for improving
training outcomes across a broad spectrum of disciplines. A search of the literature in ten major academic
databases (Dissertation Abstracts International, Article
First, ERIC, ED Index, APA/PsycInfo, Applied Science Technology, INSPEC, CTA Resource, IEEE,
Elsevier/AP/Science Direct) using keywords such as
“cognitive task analysis,” “knowledge elicitation,” and
“task analysis” yielded 318 studies. Seven studies
qualified, based on the qualifications of (1) training
based on CTA methods with an analyst, (2) conducted
between 1985 and 2003, and (3) reported pre- and
post-test measures of training performance. A total of
39 comparisons of mean effect size for pre- and posttest differences were computed from the seven studies.
Analysis of the studies found effect sizes between .91
and 2.45, which are considered to be large (Cohen,
1992). The mean effect size was d = +1.72, and the
overall percentage of post-training performance gain
was 75.2%. Results of a chi-square test of independence on the outcome measures of the pre- and posttests (χ2 = 6.50, p < 0.01) indicated that CTA most
likely contributed to the performance gain.
Cost–Benefit Studies of CTA
There are few published cost-effectiveness or
cost–benefit studies that compare CTA with other task
analysis approaches. One exception, reported by Clark
and Estes (1996), described a field-based comparison
of traditional task analysis and cognitive analysis by a
large (10,000+ employees) European organization that
redesigned a required training course in emergency
and safety procedures for approximately 500 managers. The old and new versions of the course continued
to be offered after the new version of the course was
designed so the relative efficacy of the two approaches
could be compared. All objectives and test items were
similar in both the old and new versions. As Table 43.1
indicates, the use of CTA required a greater front-end
investment of time (the organization refused to release
salary data) both for the CTA itself and the training of
instructors for the course (data on the time required to
Cognitive Task Analysis
TABLE 43.1
Cost Comparison of Behavioral and Cognitive Task Analysis
Comparison Activities
Behavioral Task Analysis
and Design Daysa
Cognitive Task Analysis
and Design Days
7
0
80
87
1000
1087
38
18
34
90
500
590
Task analysis and design
Training of presenters
Delivery by trainers
Subtotal
Total time for 500 trainers
Total training daysb
a
b
Day = person work day to design and present safety course.
Total savings with CTA: 1087 days—590 days = 497 days, or 2.5 person years.
Source: Clark, R.E. and Estes, F., Int. J. Educ. Res., 25, 403–417, 1996. With permission.
train instructors for the old course were not available).
Yet, even with the approximately 85% more front-end
time invested in design, development, and instructor
training, the new course resulted in 2.5 person-years
of time savings, because it could be offered in 1 day
(compared with 2 days for the previous course) with
equal or greater scores on the performance posttest.
Although these data are only suggestive, the time savings reported by Clark and Estes (1996) reflect similar
time savings reported above by Velmahos et al. (2004)
and Merrill (2002).
INTEGRATING COGNITIVE TASK
ANALYSIS AND TRAINING DESIGN
Optimal Integration of CTA
and Training Design
For optimal application to instruction, CTA methods
should be fully integrated with a training design model
to facilitate the alignment between learning objectives,
knowledge (declarative and procedural) necessary for
attaining the objectives, and instructional methods
appropriate to the required knowledge. Currently, three
major systems take this approach: the Integrated Task
Analysis Model (ITAM) (Redding, 1995; Ryder and
Redding, 1993), Guided Experiential Learning (GEL)
(Clark, 2004, 2006), and the Four-Component Instructional Design (4C/ID) system (van Merriënboer, 1997;
van Merriënboer and Kirschner, 2007; van Merriënboer et al., 2002). Of these, the 4C/ID model is the
most extensively developed. It can be distinguished
from other instructional design models in three ways.
First, the emphasis of the model is on the integrated
and coordinated performance of task-specific constituent skills rather than specific knowledge types or
sequenced performance of tasks. Second, a distinction
is made between supportive information, which helps
learners perform the nonrecurrent aspects of a complex
skill, and procedural or just-in-time (JIT) information,
which is presented to learners during practice and
helps them to perform the recurrent aspects of a complex skill. Third, the 4C/ID model is based on learners
performing increasingly complex skills as a whole
task, with part-task practice only of the recurrent skills;
in contrast, traditional design methods emphasize the
deconstruction of a complex task into part tasks,
which, once learned separately, are compiled as wholetask practice. The assumption of the 4C/ID model is
that environments supporting complex skill learning
can be described in terms of four interrelated components: learning tasks, supportive information, just-intime information, and part-task practice.
Learning Tasks
Learning tasks are concrete, authentic, whole-task
experiences that are organized sequentially from easy
to difficult. Learning tasks at the same level of difficulty comprise a task class, or group of tasks that draw
upon the same body of knowledge. Learning tasks
within a class initially employ scaffolding that fades
gradually over subsequent tasks within the class.
Learning tasks foster schema development to support
nonrecurrent aspects of a task. They also facilitate the
development of automaticity for schemata used during
recurrent aspects of a task.
Supportive Information
Supportive information assists the learner with interpreting, reasoning, and problem-solving activities that
comprise the nonrecurrent aspects of learning tasks. It
587
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
includes mental models demonstrated through case
studies, cognitive strategies modeled in examples, and
cognitive feedback. Through elaboration, supportive
information helps learners to apply their prior knowledge when learning new information they need to perform the task.
Just-in-Time information
Just-in-time information consists of rules, procedures,
declarative knowledge, and corrective feedback
required by learners to perform recurrent aspects of
the task. JIT information is presented in small units as
“how-to” instruction, with demonstrations of procedures and definitions of concepts illustrated with
examples. As learners perform the recurrent aspects of
a task and acquire automaticity, the amount of JIT
information provided diminishes.
Part-Task Practice
Part-task practice opportunities are provided for repetitive performance of the recurrent aspects of a task when
a high degree of automaticity is required. Part-task practice is repeated throughout instruction and mixed with
other types of practice. Part-task practice includes items
that vary from very familiar to completely novel.
COGNITIVE TASK ANALYSIS
AND 4C/ID MODEL
The 4C/ID model utilizes CTA to accomplish four
tasks: (1) decomposing complex skills into skill hierarchies, (2) sequencing the training program within task
classes, (3) analyzing nonrecurrent aspects of complex
skills to identify cognitive strategies and mental models,
and (4) analyzing recurrent aspects of the complex skill
to identify rules or procedures and their prerequisite
knowledge that generate effective performance. In general, these activities occur within the framework of the
five-stage CTA process; however, because this process
is highly integrated with the 4C/ID model, the instructional design model guides the CTA activities. This
integration with the instructional design process tends
to highlight the reiterative nature of the CTA process.
Objectives are classified as nonrecurrent if the desired
behavior varies from problem to problem and is guided
by the use of cognitive strategies or mental models.
Objectives are recurrent if the desired behavior is
highly similar from problem to problem and is guided
by rules or procedures. Sometimes recurrent constituent skills require a high degree of automaticity; these
skills are identified for additional part-task practice.
Documentation analysis, observation, and unstructured interviews with SMEs provide the information
for building a preliminary skills hierarchy to guide
further knowledge elicitation efforts. Data collection,
verification, and validation of the skills hierarchy
require multiple iterations of knowledge elicitation
using multiple SMEs. The verified skills hierarchy then
serves as a guide for deeper CTA techniques, such as
Clark’s concepts, principles, and processes (Clark,
2004, 2006). The CPP data identify constituent skills
and their interrelationships, performance objectives for
each constituent skill, and the classification of the skill
as recurrent or nonrecurrent. The CPP method also
identifies problems ranging from easy to difficult to
assist in sequencing task classes.
Sequencing Task Classes
The second group of task analysis activities involves
categorizing learning tasks into task classes. The skills
hierarchy and classified performance objectives determine the sequence of training for individual constituent skills. The 4C/ID-model employs a whole-task
approach, in which trainees learn all constituent skills
at the same time. In the first task class, learners perform
the simplest version of the whole task. As the conditions under which the task is performed become
increasingly complex, the whole task scenarios
become more authentic and reflective of those encountered by experts in the real world. CTA processes are
used both to verify the skills hierarchy and to confirm
the sequencing of task classes from simple to complex.
Analyze the Nonrecurrent
Aspects of the Complex Skill
The third set of analytic activities identifies the supportive information necessary for each task class in the
form of mental models (how is the problem domain
Decomposition of the Complex Skill
In the first group of task analysis activities, complex
skills are broken down into constituent skills, and their
interrelationships are identified.* Performance objectives are specified** for all constituent skills, and the
objectives are classified as recurrent or nonrecurrent.
588
* The three categories of interrelationships are coordinate (performed
in temporal order), simultaneous (performed concurrently), and transposable (performed in any order).
** Performance objectives reflect the performance as a result of learning
and include an action verb, a description of tools used, conditions, and
standards for performance.
Cognitive Task Analysis
organized?) and cognitive strategies (how to approach
problems in the domain?). Knowledge elicitation
methods commonly used with SMEs to capture data
for nonrecurrent aspects of a complex skill include
interviews and think-aloud protocols. The CTA methods are repeated for both simple versions and complex
versions of the task to capture the knowledge required
for performing the nonrecurrent aspects of the task.
Analyze the Recurrent
Aspects of the Complex Skill
The final set of task analysis activities in the 4C/ID
model is an in-depth analysis of the recurrent constituent skills. These are identified during the skill decomposition process to identify the JIT information required
for the recurrent aspects of the learning tasks. Each
constituent skill that enables the performance of another
constituent skill is identified in a reiterative process,
until the prerequisite knowledge already mastered by
learners at the lowest level of ability is identified.
Analysts employ CTA techniques to identify task
rules and generate highly specific, algorithmic descriptions of task performance. Next, the prerequisite
knowledge required to apply the procedure is identified. The analysis of concepts occurs through the creation of feature lists that identify the characteristics of
all instances of a concept. At a lower level, facts (which
have no prerequisites) are identified. At a higher level,
processes and principles are identified. When completed, the analyst incorporates these prerequisite
knowledge components into the rules or procedures
for performing the task.
In sum, the results of the four sets of CTA activities
in the 4C/ID model provide detailed and in-depth information about the skills, sequence, cognitive strategies,
mental models, rules, and prerequisite knowledge
required for complex skill learning through the instructional design of its four interrelated components: (1)
learning tasks, (2) supportive information, (3) JIT
information, and (4) part-task practice. Combined, they
form a fully integrated system for problem-based learning in complex domains. A complete description and
procedure for implementing the 4C/ID model can be
found in van Merriënboer and Kirschner (2007).
THE NEXT GENERATION
OF RESEARCH ON
COGNITIVE TASK ANALYSIS
Although CTA appears to have significant potential to
improve various kinds of performance, it shares many
of the challenges reported in studies of instructional
design theories and models (Glaser, 1976; Salas and
Cannon-Bowers, 2001). We need many more welldesigned studies that systematically compare the
impact of different forms of CTA on similar outcome
goals and measures. We also need to understand the
efficacy of different CTA methods when used with
different training design models and theories.
So many types of CTA have been used and
reported, and variation in the application of methods
is so overwhelming that it is doubtful that any generalization about CTA will satisfy basic standards of
construct validity. Researchers are cautioned to look
carefully at the description of the methods used to
implement CTA to classify the family origin of the
technique being replicated. We attempted to describe
five common elements of most CTA methods in the
first part of this chapter, but the specific strategies used
to implement each of these elements varies across studies. The elements we described are focused on the
common steps used to implement CTA. This is a
sequence model and is similar to the Analysis, Design,
Development, Implementation, and Evaluation
(ADDIE) model for instructional design. Schraagen et
al. (2000) and Cooke (1994) have discussed this problem in detail and have attempted to organize the various methods into families based on the type of outcome being pursued (e.g., training, job design, and
assessment). Wei and Salvendy (2004) have suggested
11 very useful guidelines for selecting the best CTA
method to achieve a goal (see Table 43.2).
First Principles of Cognitive Task Analysis
A different and equally valuable strategy for tackling
the multiplicity of CTA methods would be to apply
Merrill’s (2002) first principles approach to a similar
problem with instructional design models. Merrill
classified what appeared to be the most psychologically active instructional methods in a group of popular, evidence-based instructional design models. One
of the principles he suggested is that designs that help
learners connect with prior knowledge are more successful. An attempt to identify first principles of CTA
would be a benefit to researchers and practitioners by
identifying the active ingredients in key CTA methods;
for example, nearly all CTA methods seem to place a
heavy premium on the identification of the environmental or contextual cues that indicate the need to
implement a skill. The study of neonatal nurses by
Crandall and Getchell-Reiter (1993) involved generating more accurate diagnostic symptoms (cues)
expressed by very sick babies. Because the recognition
of conditional cues may be automated and unconscious
for many SMEs, the need for accurate and exhaustive
589
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
TABLE 43.2
Guidelines for Selecting CTA Methods
Families of CTA Methods
When To Use Various CTA Methods
Tasks and domain are not well defined in the initial stages.
Procedures to perform a task are not well defined.
Tasks are representative, and the process is clear.
Task process and performance require tracking.
Verbal data are easily captured without compromising performance.
Domain knowledge and structures require defining.
Multiple task analyzers are used, and task requires less verbalization.
Task requires quantitative predication, and task models change little
when the scenario changes.
Task performance is affected or distracted by interference.
Task analyzers lack significant knowledge and techniques.
Tasks are:
Skill-based
Rule-based
Knowledge-based
Observations
and Interviews
Process Tracing
Conceptual
Techniques
Formal Models
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Source: Adapted from Wei, J. and Salvendy, G., Behav. Inform. Technol., 23(4), 273–299, 2004.
identification of important cues may be one of the most
important principles of CTA. In the case of the neonatal nursing studies, the cues captured during CTA
have changed the textbook instructions for future neonatal nurses. Other principles may be associated with
the identification of the sequence in which productions
must be performed and the decisions that must be made
(including the alternatives that must be considered and
the criteria for selecting alternatives). Principles may
also be related to the protocols that are used to observe
and interview experts to capture the most accurate and
exhaustive description of their task-based knowledge.
It is also likely that a separate set of principles would
be needed to characterize team or organizational CTAs
(Schraagen et al., 2000).
Research on Automated, Unconscious
Expert Knowledge
Concerns about experts’ awareness of their own expertise and the strategies used to capture unconscious
knowledge are arguably the most important research
issues associated with CTA. The body of research on
unconscious, automated knowledge has yet to be widely
integrated into instructional design or the practice of
educational psychologists. Most of the research in this
area has been conducted by those interested in psychotherapy and the dynamics of stereotypes and bias in
decision making (Abreu, 1999; Bargh and Ferguson,
2000; Wheatley and Wegner, 2001) and motivation
590
(Clark et al., 2006). Yet, we have ample evidence of the
importance of this issue in CTA and training from the
results of current research by, for example, Velmahos
et al. (2004) and Chao and Salvendy (1994). We need
to know much more about how unconscious expertise
influences the accuracy of task analysis. We also need
to know much more about how to modify automated,
unconscious knowledge when people must learn to
modify skills. Clark and Elen (2006) have reviewed past
research and made suggestions for further study.
Cost Effectiveness and Cost–Benefit Research
Another promising area of future research is costeffectiveness and cost–benefit analysis (Levin and
McEwan, 2000). Existing studies have not explored
this issue systematically, but very promising preliminary analyses indicate significant learner time savings
and decreases in significant performance errors (Clark
and Estes, 1996; Merrill, 2002; Schaafstal et al., 2000;
Velmahos et al., 2004). These data are important in
part because many key decision makers have the
impression that CTA is an overly complex process that
requires a great deal of time to conduct and should be
avoided due to its cost (Cooke, 1994, 1999). It is accurate to state that CTA increases the time and effort
required for front-end design—particularly when a
number of experts who share the same skill must be
observed and interviewed; yet, it is also possible that
these costs may be offset by delivery-end savings due
Cognitive Task Analysis
to increased learner accuracy and decreased learning
time. People in formal school settings seldom consider
decreased learning time as a benefit, but in business
and government settings time is a valuable commodity.
The conditions under which savings are, and are not,
available would be a valuable adjunct to continued
development of CTA. Many other suggestions are possible but are beyond the scope of this chapter.
CONCLUSION
Cognitive task analysis is one of the major contributions to instructional technology that have resulted
from the cognitive revolution in psychology and education beginning in the 1970s. CTA does not seek to
replace behavioral task analysis (or the analysis of documents and research to support training) but instead
adds to existing methods that help capture the covert
mental processes that experts use to accomplish complex skills. The importance of CTA is based on compelling evidence that experts are not fully aware of
about 70% of their own decisions and mental analysis
of tasks (Clark and Elen, 2006; Feldon and Clark, 2006)
and so are unable to explain them fully even when they
intend to support the design of training, assessment,
job aids, or work. CTA methods attempt to overcome
this problem by specifying observational and interview
strategies that permit designers to capture more accurate and complete descriptions of how experts succeed
at complex tasks. Research evidence described in this
chapter strongly suggests huge potential benefits for
designers and learners when CTA-based performance
descriptions are used in training and job aids.
Many designers are apparently not aware of or are
not using CTA. In January 2007, we searched Google™ Scholar for the terms “task analysis” or “task
analysis models” and then “cognitive task analysis” or
“cognitive task analysis models.” The former terms
returned about nine to ten times more hits than the
cognitive task analysis terms. We looked at a number
of the texts used to teach instructional design and could
not find any references to CTA.
Cognitive task analysis has been the subject of
research more often than it has been applied in practice, so we suspect that few designers have been trained
to conduct effective cognitive task analyses. It is also
possible that the assumptions underlying CTA conflict
with the assumptions that underlie some of the currently popular design theories such as constructivism
and problem-based learning (Kirschner et al., 2006).
Educators who avoid direct instruction in favor of
expert-supported group problem solving or communities of practice would not be inclined to conduct CTA
to support a constructivist context for learning new
skills or to teach CTA in graduate programs. Our
review of the research evidence for CTA strongly indicates that if it is adopted it might make a huge contribution to learning and performance. It is also clear that
many questions about CTA remain to be answered.
REFERENCES
Abreu, J. M. (1999). Conscious and nonconscious AfricanAmerican stereotypes: impact on first impression and diagnostic ratings by therapists. J. Consult. Clin. Psychol., 67,
387–393.
Anderson, J. R. (1983). The Architecture of Cognition. Cambridge, MA: Harvard University Press.
Anderson, J. R. (1995) ACT: a simple theory of complex cognition. Am. Psychol., 51, 355–365.
Anderson, J. R., and Lebiere, C. (1998). The Atomic Components of Thought. Mahwah, NJ: Lawrence Erlbaum Associates.*
Bargh, J. A. and Ferguson, M. J. (2000). Beyond behaviorism:
on the automaticity of higher mental processes. Psychol.
Bull., 126, 925–945.*
Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K.,
and Trötschel, R. (2001). The automated will: activation and
pursuit of behavioral goals. J. Pers. Soc. Psychol., 81,
1014–1027.
Betsch, T., Fiedler, K., and Brinkmann, J. (1998). Behavioral
routines in decision making: the effects of novelty in task
presentation and time pressure on routine maintenance and
deviation. Eur. J. Soc. Psychol., 28, 861–878.
Blessing, S. B. and Anderson, J. R. (1996). How people learn
to skip steps. J. Exp. Psychol. Learn. Mem. Cogn., 22,
576–598.
Bradshaw, J. M., Ford, K. M., Adams-Webber, J. R., and Agnew,
N. M. (1993). Beyond the repertory grid: new approaches
to constructivist knowledge acquisition tool development. In
Knowledge Acquisition as Modelling, edited by K. M. Ford
and J. M. Bradshaw, pp. 9–32. New York: John Wiley &
Sons.
Chao, C.-J. and Salvendy, G. (1994). Percentage of procedural
knowledge acquired as a function of the number of experts
from whom knowledge is acquired for diagnosis, debugging
and interpretation tasks. Int. J. Hum.–Comput. Interact., 6,
221–233.*
Chinn, C. A. and Brewer, W. F. (1993). The role of anomalous
data in knowledge acquisition: a theoretical framework and
implications for science education. Rev. Educ. Res., 63(1),
1–49.
Chipman, S. F., Schraagen, J. M., and Shalin, V. L. (2000)
Introduction to cognitive task analysis. In Cognitive Task
Analysis, edited by J. M Schraagen, S. F. Chipman, and V.
J. Shute, pp. 3–23. Mahwah, NJ: Lawrence Erlbaum Associates.*
Clark, R. E. (2004). Design Document for a Guided Experiential Learning Course, Final Report on Contract DAAD 1999-D-0046-0004 from TRADOC to the Institute for Creative
Technology and the Rossier School of Education.
Clark, R. E. (2006). Training Aid for Cognitive Task Analysis,
Technical report produced under contract ICT 53-08210137-W911NF-04-D-0005 from the Institute for Creative
Technologies to the Center for Cognitive Technology, University of Southern California.
591
Richard E. Clark, David F. Feldon, Jeroen J. G. van Merriënboer, Kenneth A. Yates, and Sean Early
Clark, R. E. and Elen, J. (2006). When less is more: research
and theory insights about instruction for complex learning.
In Handling Complexity in Learning Environments:
Research and Theory, edited by J. Elen and R. E. Clark, pp.
283–297. Oxford: Elsevier.*
Clark, R. E. and Estes, F. (1996) Cognitive task analysis. Int.
J. Educ. Res., 25, 403–417.*
Clark, R. E., Howard, K., and Early, S. (2006). Motivational
challenges experienced in highly complex learning environments. In Handling Complexity in Learning Environments:
Research and Theory, edited by J. Elen and R. E. Clark, pp.
27–43. Oxford: Elsevier.
Coffey, J. W. and Hoffman, R. R. (2003). Knowledge modeling
for the preservation of institutional memory. J. Knowl. Manage., 7(3), 38–52.
Cohen, J. (1992). A power primer. Psychol. Bull., 112, 155–159.
Collins, H. M., Green, R. H., and Draper, R. C. (1985). Where’s
the expertise? Expert systems as a medium of knowledge
transfer. In Proceedings of the Fifth Technical Conference
of the British Computer Society Specialist Group on Expert
Systems ’85, edited by M. Merry, pp. 323–334. New York:
Cambridge University Press.
Cooke, N. J. (1992). Modeling human expertise in expert systems. In The Psychology of Expertise: Cognitive Research
and Empirical AI, edited by R. R. Hoffman, pp. 29–60.
Mahwah, NJ: Lawrence Erlbaum Associates.
Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. Int. J. Hum.–Comput. Stud., 41, 801–849.*
Cooke, N. J. (1999). Knowledge elicitation. In Handbook of
Applied Cognition, edited by F. T. Durso, pp. 479–509. New
York: Wiley.
Cooke, N. J. and Breedin, S. D. (1994). Constructing naive
theories of motion on-the-fly. Mem. Cogn., 22, 474–493.
Cooke, N. J., Atlas, R. S., Lane, D. M., and Berger, R. C. (1993).
Role of high-level knowledge in memory for chess positions.
Am. J. Psychol., 106, 321–351.
Crandall, B. and Gamblian, V. (1991). Guide to Early Sepsis
Assessment in the NICU. Fairborn, OH: Klein Associates.
Crandall, B. and Gretchell-Leiter, K. (1993). Critical decision
method: a technique for eliciting concrete assessment indicators from the ‘intuition’ of NICU nurses. Adv. Nurs. Sci.,
16(1), 42–51.
Crandall, B., Klein, G., and Hoffman, R. R. (2006) Working
Minds: A Practitioner’s Guide to Cognitive Task Analysis.
Cambridge, MA: MIT Press.*
Dawes, R. M. (1994). House of Cards. New York: Free Press.
Dochy, F., Segers, M., and Buehl, M. M. (1999). The relation
between assessment practices and outcomes of studies: the
case of research on prior knowledge. Rev. Educ. Res., 69(2),
145–186.
Einhorn, H. (1974). Expert judgment: some necessary conditions and an example. J. Appl. Psychol., 59, 562–571.
Ericsson, K. A. and Kintsch, W. (1995). Long-term working
memory. Psychol. Rev., 102, 211–245.*
Ericsson, K. A. and Simon, H. A. (1993). Protocol Analysis:
Verbal Reports as Data, rev. ed. Cambridge, MA: Bradford.
Ericsson, K. A. and Smith, J. (1991). Towards a General Theory
of Expertise: Prospects and Limits. New York: Cambridge
University Press.
Feldon, D. F. (2004) Inaccuracies in Expert Self Report:
Errors in the Description of Strategies for Designing Psychology Experiments. Ph.D. dissertation. Los Angeles,
CA: Rossier School of Education, University of Southern
California.
592
Feldon, D. F. (in press). Implications of research on expertise
for curriculum and pedagogy. Educ. Psychol. Rev.
update?
Feldon, D. F. and Clark, R. E. (2006). Instructional implications
of cognitive task analysis as a method for improving the
accuracy of experts’ self-report. In Avoiding Simplicity, Confronting Complexity: Advances in Studying and Designing
(Computer-Based) Powerful Learning Environments, edited
by G. Clarebout and J. Elen, pp. 109–116. Rotterdam: Sense
Publishers.*
Fisk, A. D. and Eggemeier, R. T. (1988). Application of automatic/controlled processing theory to training of tactical
command and control skills: I. Background and task analytic
methodology. In Proceedings of the Human Factors Society
33rd Annual Meeting, pp. 281–285. Santa Monica, CA:
Human Factors Society.
Gagné, R. M. (1962). The acquisition of knowledge. Psychol.
Rev., 69, 355–365.
Gagné, R. M. (1968). Learning hierarchies. Educ. Psychol., 6,
1–9.
Gagné, R. M. (1982). Developments in learning psychology:
implications for instructional design and effects of computer
technology on instructional design and development. Educ.
Technol., 22(6), 11–15.
Gagné, R. M. and Medsker, K. L. (1996). The Conditions of
Learning: Training Applications. New York: Harcourt Brace.
Gagné, R. M., Briggs, L. J., and Wager, W. W. (1992). Principles
of Instructional Design. Fort Worth, TX: Harcourt Brace
Jovanovich.
Glaser, R. (1976). Components of a psychology of instruction:
toward a science of design. Rev. Educ. Res., 46(1), 1–24.
Glaser, R. and Chi, M. T. H. (1988). Overview. In The Nature
of Expertise, edited by M. T. H. Chi, R. Glaser, and M. J.
Farr, pp. xv–xxviii. Mahwah, NJ: Lawrence Erlbaum Associates.
Hall, E. M., Gott, S. P., and Pokorny, R. A. (1995). A Procedural
Guide to Cognitive Task Analysis: The PARI Methodology.
Brooks Air Force Base, TX: Manpower and Personnel Division, U.S. Air Force.
Hermans, D., Crombez, G., and Eelen, P. (2000). Automatic
attitude activation and efficiency: the fourth horseman of
automaticity. Psychol. Belg., 40(1), 3–22.
Hinds, P. J., Patterson, M., and Pfeffer, J. (2001). Bothered by
abstraction: the effect of expertise on knowledge transfer
and subsequent novice performance. J. Appl. Psychol., 86,
1232–1243.
Hoffman, P., Slovic, P., and Rorer, L. (1968). An analysis of
variance model for the assessment of configural cue utilization in clinical judgment. Psychol. Bull., 69, 338–349.
Hoffman, R. R., Shadbolt, N. R., Burton, A. M., and Klein, G.
(1995). Eliciting knowledge from experts: a methodological
analysis. Org. Behav. Hum. Decis. Processes, 62(2),
129–158.*
Hoffman, R. R., Crandall, B., and Shadbolt, N. (1998). Use of
the critical decision method to elicit expert knowledge: a
case study in the methodology of cognitive task analysis.
Hum. Factors, 40, 254–277.*
Johnson, P. E. (1983). What kind of expert should a system be?
J. Med. Philos., 8, 77–97.
Jonassen, D. H., Tessmer, M., and Hannum, W. H. (1999). Task
Analysis Methods for Instructional Design. Mahwah, NJ:
Lawrence Erlbaum Associates.*
Kareken, D. A. and Williams, J. M. (1994). Human judgment
and estimation of premorbid intellectual function. Psychol.
Assess., 6(2), 83–91.
Cognitive Task Analysis
Kirschner, P., Sweller, J., and Clark, R. E. (2006). Why minimally guided learning does not work: an analysis of the
failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educ. Psychol.,
41(2), 75–86.*
Klein, G. A. and Calderwood, R. (1991). Decision models: some
lessons from the field. IEEE Trans. Syst. Man Cybernet., 21,
1018–1026.
Klein, G. A., Calderwood, R., and MacGregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Trans.
Syst. Man Cybernet., 19, 462–472.*
Lee, J.-Y. and Reigeluth, C. M. (2003). Formative research on
the heuristic task analysis process. Educ. Technol. Res. Dev.,
51(4), 5–24.
Lee, R. L. (2004). The Impact of Cognitive Task Analysis on
Performance: A Meta Analysis of Comparative Studies.
Ed.D. dissertation. Los Angeles, CA: Rossier School of Education, University of Southern California.
Levin, H. M. and McEwan, P. J. (2000). Cost Effectiveness
Analysis: Methods and Applications, 2nd ed. Beverly Hills,
CA: SAGE.
Lohman, D. F. (1986). Predicting mathemathantic effects in the
teaching of higher-order thinking skills. Educ. Psychol.,
21(3), 191–208.
Maupin, F. (2003). Comparing Cognitive Task Analysis to
Behavior Task Analysis in Training First Year Interns To
Place Central Venous Catheters. Ph.D. dissertation. Los
Angeles, CA: University of Southern California.
Merrill, M. D. (2002). A pebble-in-the-pond model for instructional design. Perform. Improv., 41(7), 39–44.
Mullin, T. M. (1989). Experts estimation of uncertain quantities
and its implications for knowledge acquisition. IEEE Trans.
Syst. Man Cybernet.,19, 616–625.
Neves, D. M. and Anderson, J. R. (1981). Knowledge compilation: mechanisms for the automatization of cognitive
skills. In Cognitive Skills and Their Acquisition, edited by
J. R. Anderson, pp. 335–359. Hillsdale, NJ: Lawrence
Erlbaum Associates.
Redding, R. E. (1995). Cognitive task analysis for instructional
design: applications in distance education. Dist. Educ., 16,
88–106.
Russo, J. E., Johnson, E. J., and Stephens, D. L. (1989). The
validity of verbal protocols. Mem. Cogn., 17(6), 759–769.
Ryder, J. M. and Redding, R. E. (1993). Integrating cognitive
task analysis into instructional systems development. Educ.
Technol. Res. Dev., 41, 75–96.
Salas, E. and Cannon-Bowers, J. A. (2001). The science of
training: a decade of progress. Annu. Rev. Psychol., 52,
471–497.
Schaafstal, A., Schraagen, J. M., and van Berlo, M. (2000).
Cognitive task analysis and innovation of training: the case
of the structured troubleshooting. Hum. Factors 42, 75–86.
Schneider, W. (1985). Training high-performance skills: fallacies and guidelines. Hum. Factors, 27, 285–300.
Schneider, W. and Shiffrin, R. M. (1977). Controlled and automatic human information processing. 1. Detection, search,
and attention. Psychol. Rev., 84, 1–66.
Schraagen, J. M., Chipman, S. F., and Shute, V. J. (2000) Stateof-the-art review of cognitive task analysis techniques. In
Cognitive Task Analysis, edited by J. M Schraagen, S. F.
Chipman, and V. J. Shute, pp. 467–487. Mahwah, NJ:
Lawrence Erlbaum Associates.
Schwartz, D. and Bransford, J. D. (1998). A time for telling.
Cogn. Instruct., 16, 475–522.
Sternberg, R. J. and Horvath, J. A. (1998). Cognitive conceptions of expertise and their relations to giftedness. In Talent
in Context, edited by R. C. Friedman and K. B. Rogers, pp.
177–191. Washington, D.C.: American Psychological Association.*
Thorley, N. and Stofflet, R. (1996). Representation of the conceptual change model in science teacher education. Sci.
Educ., 80, 317–339.
van Merriënboer, J. J. G. (1997). Training Complex Cognitive
Skills: A Four-Component Instructional Design Model for
Technical Training. Englewood Cliffs, NJ: Educational
Technology Publications.*
van Merriënboer, J. J. G. and Kirschner, P. A. (2007). Ten Steps
to Complex Learning: A Systematic Approach to Four-Component Instructional Design. Mahwah, NJ: Lawrence
Erlbaum Associates.
van Merriënboer, J. J. G., Clark, R. E., and de Croock, M. B.
M. (2002). Blueprints for complex learning: the 4C/IDmodel. Educ. Technol. Res. Dev., 50(2), 39–64.*
Velmahos, G. C., Toutouzas, K. G., Sillin, L. F., Chan, L., Clark,
R. E., Theodorou, D., and Maupin, F. (2004). Cognitive task
analysis for teaching technical skills in an inanimate surgical
skills laboratory. Am. J. Surg., 18, 114–119.*
Wegner, D. M. (2002). The Illusion of Conscious Will. Cambridge, MA: MIT Press.
Wei, J. and Salvendy, G. (2004). The cognitive task analysis
methods for job and task design: review and reappraisal.
Behav. Inform. Technol., 23(4), 273–299.*
Wheatley, T. and Wegner, D. M. (2001). Automaticity of action,
psychology of. In International Encyclopedia of the Social
and Behavioral Sciences, edited by N. J. Smelser and P. B.
Baltes, pp. 991–993. Oxford: Elsevier.
Zeitz, C. M. (1997). Some concrete advantages of abstraction:
how experts’ representations facilitate reasoning. In Expertise in Context, edited by P. J. Feltovich, K. M. Ford, and
R. R. Hoffman, pp. 43–65. Menlo Park, CA: American Association for Artificial Intelligence.
* Indicates a core reference.
593