11-­‐04-­‐18 Cognitive Task Analysis
Carolina Wannheden,
Doctoral student
Health Informatics Center, LIME
[email protected]
”The expert’s bill”
Protagonists: Henry Ford, Nikola Tesla
$ 10,000
Marking wall:
Knowing where to mark:
[email protected]
$1
$ 9,999
2011-04-18
1 11-­‐04-­‐18 Discuss in pairs…
1. What is it that Tesla knows, and how
does he know it?
2. How can we grasp this knowledge?
3. Why would we want to grasp this
knowledge?
[email protected]
2011-04-18
Learning Outcomes
 You should be able to
 Explain when and why to perform a Cognitive Task Analysis
 Explain what distinguishes an expert and when knowledge-based
systems may be useful
 Explain different knowledge structures
 Apply the Critical Decision Method for the aquisition of expert
knowledge to inform the design of a CDSS
[email protected]
2011-04-18
2 11-­‐04-­‐18 The purpose of CTA…
…is to understand how people think and what
they know (cognitive) to achieve some
particular goal (task)…
…in order to
 Analyze incidents
 Develop training material
 Develop memory aids, decision aids, expert
systems
[email protected]
2011-04-18
When to use knowledge-based systems
 When real experts are
 Scarce
 Expensive
 Inconsistent
 Unavailable on a routine basis
[email protected]
2011-04-18
3 11-­‐04-­‐18 Case from week 1
[email protected]
2011-04-18
Read the case and reflect on…
1. What makes the anesthesiologist’s decision challenging?
2. Why does the anesthesiologist choose the general anesthesia?
3. How would you describe the context? Which factors affect her
decision-making?
4. What type of knowledge does the anesthesiologist need in
order to make the right decision? Can you distinguish different
types of knowledge?
[email protected]
2011-04-18
4 11-­‐04-­‐18 Experts know how, not just what!
 Conceptual/factual knowledge (what)
 General anesthesia is for…
 Regional anesthesia is for…
 Procedural knowledge (how)
 IF X, then provide the appropriate anesthesia and operate.
 (Structural knowledge
 Guideline A is applicable for patients aged 15 or more)
[email protected]
2011-04-18
[email protected]
2011-04-18
5 11-­‐04-­‐18 Key aspects of CTA
1. Knowledge acquisition (KA)
2. Data analysis
3. Knowledge representation
[email protected]
2011-04-18
What CTA tries to capture
      What people are thinking about
What they are paying attention to
The strategies they are using in making decisions
What they are trying to accomplish
What information they discard
What they know about the way a process works
(Crandall, Klein, Hoffman, 2006)
[email protected]
2011-04-18
6 11-­‐04-­‐18 Techniques for knowledge acquisition
 Interviews
 + Require a minimum level of resources
 + Can be performed in a relatively short time frame
 + Can yield a significant amount of qualitative knowledge
 - Lack of quantitative data
 - Bias due to selection of questions by researcher
 - Elicited knowledge may not correspond to what expert actually
does
 Think-aloud protocols
 Observations
 Ethnographic evaluations to collect information in context
 Group techniques (e.g. brainstorming)
[email protected]
2011-04-18
Data analysis methods
 Protocol and discourse analysis
 Elicit knowledge from individuals while they are engaged in
problem-solving or reasoning tasks
 Determine conceptual entities and relationships between them
 Concept mapping
 Node-link structures of knowledge
 Concept maps can support the formation of consensus among
experts
 Verification and validation
 Verification: fulfillment of perceived requirements (to define design)
 Validation: fulfillment of realized requirements (upon
implementation)
[email protected]
2011-04-18
7 11-­‐04-­‐18 Knowledge Representation
     Narrative formats
Chronologies
Data organizers
Process diagrams
Concept maps
(Crandall, Klein, Hoffman, 2006)
[email protected]
2011-04-18
Decision Requirements Table
Treatment
phase
Decision
challenge
Cue/
Information
Giving
anesthesia
before
surgery
Choose
adequate
type of
anesthesia
Age, medical
history
Strategy or
practice
Novice
Traps
(Crandall, Klein, Hoffman, 2006)
[email protected]
2011-04-18
8 11-­‐04-­‐18 Challenges in acquiring expert knowledge
 Complex and resource-intensive
 Identification and access to domain expert with
 Sufficient domain knowledge
 Interest in participating in knowledge acquisition process
 Minimal bias
(Greenes, 2006)
[email protected]
2011-04-18
Reasoning biases
 Poor estimation of probabilities (Probability bias)
 Use terms like ”suggests”, ”supports”, ”goes against”, ”often”,
”evokes the possibility” to describes uncertainty
 Estimation bias
 Recency bias (mistaking for frequeny)
 Anchor judgments on initial estimates
 Familiarity or stereotypic frequency over objective frequency
 Overestimate frequency of rare events
(Greenes, 2006)
[email protected]
2011-04-18
9 11-­‐04-­‐18 Critical Decision Method (CDM)
 CDM was created to learn from
specific incidents (Hoffman et al.,
1998)
 Described well in chapter 5,
Working Minds: A Practitioner's
Guide to Cognitive Task
Analysis (Crandall, Klein, Hoffman,
2006)
[email protected]
2011-04-18
The CDM interview
 Intensive in-depth interview (duration ~2 hrs) to elicit cognitive
functions such as decision making, planning, sensemaking
within a specific challenging incident
 Conducted by 2 researchers
1. Primary facilitator (and note-taker)
2. Note-taker and time-keeper
 Conducted in 4 sweeps (phases)
1. 2. 3. 4. Incident identification
Constructing a timeline
Deepening
”What if” queries
[email protected]
2011-04-18
10 11-­‐04-­‐18 Sweep 1: Incident identification
 Goal: Try to identify an incident that will contain cognitive
components beyond background and routine procedural
knowledge
 Nonroutine, challenging events
 The participant has to have a role as a ”doer/decision maker”
 The participant’s decision making should have had a direct impact
on the outcome
 Critical event, time pressure
 Ask the participant to provide a brief account of the story, from
beginning to end
[email protected]
2011-04-18
Sweep 2: Constructing a timeline
 Goal: Get a clear, refined, and verified overview of the incident
structure, identifying key events and segments.
 The interviewer diagrams the sequence of events on a timeline
 Identify critical points/”decision points”
 Try to note sequence and duration of events, actions, perceptions,
thoughts, decisions
[email protected]
2011-04-18
11 11-­‐04-­‐18 Sweep 3: Deepening
 Goal: Get inside the expert’s head: ”[W]hat did they know, when
did they know it, how did they know, and what did they do with
what they knew?” (Crandall et al., 2006)
 Based on the timeline probe critical points for the participant’s
 Perceptions
 Expectations
 Goals
 Judgments
 Confusions, uncertainties, concerns
 Options
 Information needed and used
[email protected]
2011-04-18
Sweep 4: ”What if” queries
 Goal: Illuminate expert-novice differences and potential
vulnerabilities for error in the domain
 The interviewer poses hypotheticals about the event
 What if a novice had been at charge?
 What if [key feature] had been different?
 What training might have been an advantage?
 What knowledge, information, tools/technologies could have
helped?
[email protected]
2011-04-18
12 11-­‐04-­‐18 Discuss in your groups…
 Would the Critical Decision Method be an appropriate method to
elicit knowledge for the CDSS you intend to develop?
 Would it be a feasible method?
[email protected]
2011-04-18
A final quote
”For now, […], it is the direct interaction among experts,
and between experts and knowledge engineers, that will
serve a crucial role in assuring the development of high
quality and accepted knowledge bases that in turn
enable the development and effective use of decision
support systems.” (Greenes, 2006)
[email protected]
2011-04-18
13 11-­‐04-­‐18 WW DD
[email protected]
2011-04-18
14