Journal of Surgical Research 173, e37–e42 (2012)
doi:10.1016/j.jss.2011.09.003
The Use of Cognitive Task Analysis to Improve Instructional Descriptions
of Procedures
Richard E. Clark, Ed.D.,* Carla M. Pugh, M.D., Ph.D.,† Kenneth A. Yates, Ed.D.,* Kenji Inaba, M.D., F.A.C.S.,‡
Donald J. Green, M.D., F.A.C.S.,‡ and Maura E. Sullivan, Ph.D.§,1
*Center for Cognitive Technology, Rossier School of Education; Department of Surgery, Keck School of Medicine, University of Southern
California, Los Angeles, California; †Center for Advanced Surgical Education, Northwestern University, Feinberg School of Medicine,
Chicago, Illinois; ‡Division of Trauma and Surgical Critical Care, Department of Surgery, Keck School of Medicine, LACþUSC Medical
Center, Los Angeles, California; and §Department of Surgery, Keck School of Medicine, Rossier School of Education, University of
Southern California, Los Angeles, California
Originally submitted May 24, 2011; accepted for publication September 2, 2011
Background. Surgical training relies heavily on the
ability of expert surgeons to provide complete and accurate descriptions of a complex procedure. However,
research from a variety of domains suggests that
experts often omit critical information about the judgments, analysis, and decisions they make when solving
a difficult problem or performing a complex task. In
this study, we compared three methods for capturing
surgeons’ descriptions of how to perform the procedure
for inserting a femoral artery shunt (unaided freerecall, unaided free-recall with simulation, and cognitive task analysis methods) to determine which method
produced more accurate and complete results. Cognitive task analysis was approximately 70% more complete and accurate than free-recall and or free-recall
during a simulation of the procedure.
Methods. Ten expert trauma surgeons at a major urban trauma center were interviewed separately and
asked to describe how to perform an emergency shunt
procedure. Four surgeons provided an unaided freerecall description of the shunt procedure, five surgeons provided an unaided free-recall description of
the procedure using visual aids and surgical instruments (simulation), and one (chosen randomly) was
interviewed using cognitive task analysis (CTA)
methods. An 11th vascular surgeon approved the final
CTA protocol.
Results. The CTA interview with only one expert
surgeon resulted in significantly greater accuracy
1
To whom correspondence and reprint requests should be addressed at Department of Surgery, Keck School of Medicine, Rossier
School of Education, University of Southern California, 1520 San
Pablo Street, Ste 4000, Los Angeles, CA 90033. E-mail: mesulliv@
usc.edu.
and completeness of the descriptions compared with
the unaided free-recall interviews with multiple expert surgeons. Surgeons in the unaided group omitted
nearly 70% of necessary decision steps. In the freerecall group, heavy use of simulation improved surgeons’ completeness when describing the steps of the
procedure.
Conclusion. CTA significantly increases the completeness and accuracy of surgeons’ instructional
descriptions of surgical procedures. In addition, simulation during unaided free-recall interviews may
improve the completeness of interview data. Ó 2012
Elsevier Inc. All rights reserved.
Key Words: trauma surgery training; cognitive task
analysis; surgical skills training; surgical expertise;
medical education.
INTRODUCTION
It is well known that surgeons must master highly
complex and rapidly changing knowledge and skills.
However, due to new demands in surgical education
over the past decade, specifically the regulation of
duty hours, the increasing complexity of cases, and
the decreased availability of operating room time, it
has become increasingly challenging to transmit this
knowledge and teach essential skills to surgeons-intraining. It is even more challenging for trauma surgeons who must perform and teach in high stress,
time-pressured, and uncontrolled environments. In
a trauma setting, there is limited time for instruction
during a procedure and, due to the nature of the specialty, there is little time to pre-plan and discuss
e37
0022-4804/$36.00
Ó 2012 Elsevier Inc. All rights reserved.
e38
JOURNAL OF SURGICAL RESEARCH: VOL. 173, NO. 1, MARCH 2012
patients prior to arrival. For these reasons, many programs have moved a significant portion of training out
of the clinical arena and into a simulated environment.
For this instruction to be successful, educators must develop a sound curriculum that supports the simulated
modality and teach the essential action and decisions
steps that accompany each task.
Traditionally, the creation of procedural skills curricula has relied on the ability of experts to describe
specific procedures. This is problematic because recent
research has shown that when experts describe how
they perform a difficult task, they may unintentionally
omit up to 70% of the critical information novices need
to learn to successfully perform the procedure [1]. This
has been termed the 70% rule and is a serious problem
because it forces novices to ‘‘fill in the blanks’’ using
less efficient and error-prone trial-and-error methods.
Moreover, as these errors are practiced over time, they
become more difficult to ‘‘unlearn’’ and correct [2].
There appear to be two reasons for this problem. First,
as physicians gain expertise, their skills become automated and the steps of the procedure blend together.
Experts perform tasks largely without conscious
knowledge as a result of years of practice and experience [3]. This causes experts to omit specific steps
when trying to describe a procedure because this information is no longer accessible to conscious processes
[3, 4]. Secondly, many physicians are not able to share
the complex thought processes that accompany the behavioral execution of technical skills. Even physicians
who make an attempt to ‘‘think out loud’’ during a procedure often omit essential information because their
knowledge is automated. Consequently, it is difficult
to identify the points during a procedure where an expert makes decisions [5].
When asked to describe a procedure, many surgeons
will rely on self-recall of the specific skill. Studies from
the field of cognitive psychology suggest that the use of
standard self-report or interview protocols to extract
descriptions of events, decision making, and problem
solving strategies can lead to inaccurate or incomplete
reports [6, 7]. These errors are not often recognized by
experts because of the automated and unconscious nature of the knowledge described [8]. Moreover, errors
are likely to increase in number and impact under
stressful situations [9].
A potential solution to the dilemma described above
is the use of cognitive task analysis (CTA) to extract
implicit and explicit knowledge from experts to better
inform surgical instruction. CTA uses a variety of
interview and observation methods to capture the
knowledge, goals, strategies, and decisions that underlie observable task performance in complex situations [10]. This interview strategy was first developed
by the military to increase the learning curve and
acquisition of expertise of complex technical skills
and is gaining attention and application in surgical
education. The common goal among CTA methods is
to assist a subject matter expert in the retrieval
and recounting of a procedure that may be highly automated and, therefore, not generally available for
conscious inspection.
The purpose of this study was to compare the percentage of knowledge that experts omitted while
describing a femoral artery shunt procedure and to
validate the 70% rule, which is that experts unintentionally leave out approximately 70% of the information that novices need to perform a task successfully.
More specifically, we examined the accuracy of experts’ traditional unaided free-recall of the procedure
compared with a gold standard protocol derived from
the results of cognitive task analysis. In addition, we
investigated whether task simulation (use of equipment and visual aids) increased the accuracy of the
traditional free-recall descriptions. We hypothesized
that (1) surgeons who give unaided free-recall descriptions of the femoral shunt procedure will omit 670% of
the critical steps compared with the CTA gold standard protocol; (2) the completeness of unaided, selfreported surgical protocol information will vary with
the use of simulation during the description of the
shunt procedure; (3) the use of CTA methods will increase the accuracy and completeness of surgical protocols by between 12% and 40%.
METHODS
Subjects
A total of 11 surgeons were recruited for this study. Nine trauma
surgeons with experience in placing Argyle type femoral shunts in
emergency trauma environments were recruited from the medical
school of a large urban research university. Four out of nine of these
surgeons were faculty in the Navy Trauma Training Program, and all
had battlefield experience as part of a Navy Surgical Team deployed to
Iraq. A 10th trauma surgeon, who had experience performing shunt
placements as part of an Army Forward Surgical Team in Iraq, was
recruited through the offices of the study sponsor. The 11th surgeon,
who was recruited from the Division of Vascular Surgery and had femoral shunt experience in both urban and battlefield settings, reviewed, finalized, and approved the gold standard protocol used to
check the accuracy and completeness of all descriptions of the shunt
procedure.
Procedure
A flow diagram of the research methods is presented in Fig. 1. Nine
trauma surgeons were interviewed and asked to give a free-recall, unaided description of the procedure (no CTA condition). Each surgeon
was interviewed separately and asked to describe how to perform
the shunt procedure under emergency conditions. Five surgeons
were randomly assigned to the ‘‘yes simulation condition’’ and were
asked to describe their surgical protocol as they manipulated a set
of surgical instruments and viewed depictions of the anatomy surrounding the femoral artery. The remaining surgeons were randomly
CLARK ET AL.: COGNITIVE TASK ANALYSIS IN SURGERY
e39
FIG. 1. Flow diagram of research methods.
assigned to the ‘‘no simulation condition’’ and were interviewed without access to instruments or anatomy. Follow-up questions were
asked to clarify statements made in the unaided section of the
interview.
A complete CTA interview was conducted with a 10th trauma surgeon. A CTA-trained interviewer (RC), who was not a surgeon, conducted the semi-structured interview using the methods described
in Clark et al. [11]. The interview consisted of a series of questions
pertaining to (1) conditions (indications and contraindications for
performing the procedures); (2) processes (who does what, when,
and where); (3) steps (action steps and decision steps accompanied
by alternatives and the criteria for deciding); (4) standards (time
and quality); (5) equipment (instruments); (6) reasons (principles
of science, i.e., why do this, and not that); and (7) concepts (names,
symbols, or events that a surgeon would need to know to perform the
procedure).
All interviews were recorded with audio and video. The interviews
were transcribed verbatim, and were coded using a scheme based on
the interview questions. Two coders worked independently to code the
transcripts, and then met to compare the results and resolve the discrepancies. An inter-rater reliability of 0.87 was achieved in the
coding.
A protocol of each surgeon’s description of the procedure was developed from the coded transcripts by a CTA analyst and reviewed
for accuracy by a second analyst. The surgeons were then given
the opportunity to review and correct the protocol developed from
their transcribed interview. The surgeons’ corrected protocols were
aggregated to create a preliminary ‘‘gold standard’’ protocol. The
11th surgeon, an independent vascular surgeon, reviewed and corrected the preliminary protocol, which then became the final ‘‘gold
standard.’’ The gold standard contained a sequential list of all of
the action and decision steps that are essential to successfully insert
a femoral shunt.
Once the gold standard protocol was developed, the statements
made in each surgeon’s CTA interview were compared with the gold
standard to determine the accuracy and completeness of their interview data. Surgeons’ protocols were compared and analyzed prior to
being corrected (Round 1), and after review and correction (Round
2). Additional data included the level of experience, measured by the
surgeons’ report of the number of shunt procedures performed.
The amount of interaction that each surgeon in the ‘‘yes simulation
condition’’ had with the equipment and visual aids provided was determined. Surgeons were rated as having minimal, occasional, or
heavy interaction. ‘‘Minimal’’ referred to a surgeon’s reference to the
simulated aids verbally, visually, or by pointing; ‘‘Occasional’’ referred
to a surgeon’s occasional touching of the simulation materials; and
‘‘Heavy’’ referred to surgeon’s who handled the materials to demonstrate the procedure.
RESULTS
Hypothesis 1
Surgeons who give unaided free-recall descriptions
of the femoral shunt procedure will omit 670% of
the critical steps compared with a gold standard
CTA protocol. The total percentage of agreement between surgeons’ free-recall description of the shunt
procedural steps and the gold standard protocol in
the unaided interview condition were 25.00% in Round
1 with a 6.25% improvement in Round 2 for a total of
31.25% agreement (Fig. 2). Thus, surgeons omitted an
e40
JOURNAL OF SURGICAL RESEARCH: VOL. 173, NO. 1, MARCH 2012
FIG. 2. Total percentage of agreement between surgeons’ unaided free-recall description of shunt procedure and gold standard protocol.
average of 68.75 % of the standard procedural steps in
support of Hypothesis 1.
Hypothesis 2
The completeness of unaided, self-reported surgical
protocol information will vary with the use of simulation during the description of the shunt procedure.
As shown by Fig. 3, there is no difference between the
Minimal and Occasional use of simulation in the accuracy and completeness of recall by surgeons as they
were describing the shunt procedure when compared
with the CTA-based gold standard. However, heavy
use of simulation increased the level of agreement in
both Round 1 and Round 2. These data may support
the hypothesis that the increased use of simulation augments surgeons’ description of the procedural steps
needed by students during training. It also appeared
that those surgeons who made use of instruments and
pictures during the interview experienced greater recall, which supports Hypothesis 2. Although the differences among the simulation groups were not found to be
significant, it is likely that lack of significance is attributable to a type II error due to low sample size.
FIG. 3. Percentage of agreement of unaided surgeons’ description of procedural steps by the level of interaction with simulation materials.
CLARK ET AL.: COGNITIVE TASK ANALYSIS IN SURGERY
Hypothesis 3
The use of CTA methods will increase the accuracy
and completeness of surgical protocols by between
12% and 40%. As shown in Fig. 4, a comparison between
the unaided free-recall interviews and CTA interview
against the gold standard demonstrates the increase
in the percentage of agreement for the CTA condition.
The results show that the total percentage of agreement
with CTA is 68.75% compared with 31.25% for the freerecall group. With only one surgeon having CTA, the
average of the unaided free-recall was compared with
his score using a one sample t-test on all three dependent variables (total steps, Round 1, Round 2) with
the results being highly significant (P < 0.001). These
results support Hypothesis 3.
DISCUSSION
This study suggests that consideration should be
given to adopting CTA methods to increase accuracy,
decrease recall errors, and capture the underlying judgments, analyses, decisions, and other cognitive processes that experts use to perform complex medical
procedures. By applying CTA methodology, we were
able to capture more action and decision steps from surgical experts than the use of unaided free-recall
methods alone. This study supports previous research,
which shows that most forms of CTA show tangible
benefits with respect to the accuracy and completeness
of knowledge obtained. For example, Sullivan and
colleagues [12] demonstrated that expert surgeons
omitted an average of 65% of procedural steps and cognitive decisions when teaching a colonoscopy to surgical
residents compared with a CTA gold standard protocol.
In another study, Chao and Salvendy [13] showed
e41
a 28% increase in information captured from experts
during a task by using CTA methods, while Clark and
Estes [3] showed a 40% gain in knowledge captured
from experts by using CTA. When multiple experts
are interviewed, the percent of information captured increases proportionately [13]. [Bartholio C. Unpublished
doctoral dissertation, Rossier School of Education, University of Southern California, Crispen P. Unpublished
doctoral dissertation, Rossier School of Education, University of Southern California.]
This study suggests that training curricula based on
free-recall, unaided (non-CTA) expert interviews may
be lacking important knowledge and skills required
by novices to successfully perform a medical procedure,
and may lead to less effective training and more ‘‘trial
and error’’ learning. There have been studies in the
surgical literature that suggest performance improvements in training can be attributed to CTA-based
instruction for central line placement [14], open cricothyrotomy [15], percutaneous tracheostomy placement
[16], and flexor tendon repair [17]. With this in mind, it
may be desirable to use ‘‘gold standard’’ protocols developed through the CTA process for surgical skills training and assessment.
CTA was first developed by the military as a method
for capturing highly complex performance expertise
where many covert decisions are linked with difficult
overt actions. Although developed for complex tasks,
its application in surgery has mostly focused on more
standard procedures. Current projects are underway
to investigate the use of CTA for more complex surgical
procedures (i.e., AVR) where more complex decisions
must be made that may alter the subsequent course of
the operation. In addition, it has been shown that
CTA can increase the learning curve and accelerate
the acquisition of expertise [3, 6, 18]. Training
FIG. 4. Total percentage of agreement between surgeons’ description of shunt procedure with CTA and without CTA, compared with gold
standard procedure.
e42
JOURNAL OF SURGICAL RESEARCH: VOL. 173, NO. 1, MARCH 2012
programs that utilized CTA methods have been able to
decrease the training time [3, 6, 18] and improve the
performance of tasks [14, 16]. One of the most dramatic
claims was made by Means and Gott [18], who speculated that the equivalent of 5 y of job knowledge could
be transmitted in 50 h of training based on CTA. In light
of the new work-hour restrictions for residents, CTA
may prove to be a tool that can be used to accelerate
the learning curve and improve the performance of surgical trainees. Future research efforts should be targeted at the effectiveness of CTA to improve the
efficiency of training in surgical programs.
Similarly, the use of CTA methods to capture highly
experienced physician’s decision-making strategies
may provide more accurate and complete information
for use in the design of medical simulation technologies.
The application of CTA methods with subject matter experts have been shown to produce high quality representations of expert performance in increasingly complex
and difficult settings. This information can be developed
into simulations that provide efficient and effective practice of surgical skills. Although the cognitive decisions
that accompany technical skills are essential to teach
novices, many surgical simulators focus largely on technical skills. Future research should be directed towards
using CTA methods to expose the critical decisionmaking skills necessary to perform difficult operations
and surgical tasks. Incorporating ‘‘gold standard’’
CTAs as the basis for assessing technical and surgical
decision-making skills should also be examined.
REFERENCES
1. Feldon DF, Clark RE. Instructional implications of cognitive
task analysis as a method for improving the accuracy of experts’
self-report. In: Clarebout G, Elen J, Eds. Avoiding simplicity,
confronting complexity: Advances in studying and designing
(computer-based) powerful learning environments. Rotterdam,
The Netherlands: Sense Publishers, 2006:109–16.
2. Clark RE. Resistance to change: Unconscious knowledge and the
challenge of unlearning. In: Berliner DC, Kupermintz H, Eds.
Changing institutions, environments and people. New York:
Rutledge, 2008:75–94.
3. Clark RE, Estes F. Cognitive task analysis. Int J Educ Res 1996;
25:403.
4. Clark RE, Elen J. When less is more: Research and theory insights about instruction for complex learning. In: Elen J,
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
Clark RE, Lowick J, Eds. Handling complexity in learning environments: Research and theory. Oxford: Elsevier, 2006:283–97.
Schraagen JM, Chipman SF, Shalin VL. Cognitive task analysis.
New Jersey: Lawrence Erlbaum Associates, 2000.
Glaser R, Lesgold A, Lajoie S, et al. Cognitive task analysis to enhance technical skills training and assessment. (Final Report to
the Air Force Human Resources Laboratory on Contract No.
F41689–8V3-C-0029). Pittsburgh, PA: Learning Research and
Development Center, University of Pittsburgh, 1985.
Besnard D, Bastien-Toniazzo M. Expert error in troubleshooting: An exploratory study in electronics. International Journal
of Human-Computer Interaction 1999;50:391.
Wheatley T, Wegner DM. Automaticity of action. In: Smelser NJ,
Baltes PB, Eds. International encyclopedia of the social and behavioral sciences. Oxford, IK: Elsevier Science Limited, 2001:
991–3.
Hunt E, Joslyn S. A functional task analysis of time-pressured
decision making. In: Schraagen JM, Chipman SF, Shalin VL,
Eds. Cognitive task analysis. Mahwah, New Jersey: Lawrence
Erlbaum Associates, 2000:119–32.
Clark RE, Feldon DF, Van Merri€enboer JJG, et al. Cognitive task
analysis. In: Spector JM, Merrill MD, Van Merri€enboer JJG,
Driscoll MP, Eds. Handbook of Research on Educational Communications and Technology. 3rd ed. Mahwah, NJ: Lawrence
Erlbaum Associates, Inc., 2008:577–93.
Clark RE, Pugh CM, Yates KA, et al. The use of cognitive task
analysis and simulators for after action review of medical events
in Iraq. Technical Report 5-21-2008 developed for the Center for
Cognitive Technology, Rossier School of Education, University of
Southern California, 2008.
Sullivan ME, Ortega A, Wasserman N, et al. Assessing the
teaching of procedural skills: Can cognitive-task-analysis add
to our traditional teaching methods? Am J Surg 2008;195:20.
Chao CJ, Salvendy G. Percentage of procedural knowledge
acquired as a function of the number of experts from whom
knowledge is acquired for diagnosis, debugging and interpretation tasks. IJMHCI 1994;6:221.
Velmahos G, Toutouzas K, Sillin L, et al. Cognitive task analysis
for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg 2004;187:114.
Campbell J, Tirapelle L, Yates KA, et al. The effectiveness of
a cognitive task analysis informed curriculum to increase selfefficacyand improve performance for an open cricothyrotomy.
J Surg Educ 2011;68:403.
Sullivan ME, Brown CV, Peyre SE, et al. The use of cognitive
task analysis to improve the learning of percutaneous tracheostomy placement. Am J Surg 2007;193:96.
Luker KR, Sullivan ME, et al. The use of a cognitive task
analysis-based multimedia program to teach surgical decision
making in flexor tendon repair. Am J Surg 2008;195:11.
Means B, Gott S. Cognitive task analysis as a basis for tutor development: Articulating abstract knowledge representations. In:
Psotka J, Massey LD, Mutter SA, Eds. Intelligent tutoring systems: Lessons learned. Hillsdale, NJ: Lawrence Erlbaum, 1988:
35–58.