Aviate
Navigate
Communicate
Manage Systems
Improving Cockpit Task
Management Performance:
The AgendaManager
Training Pilots to Prioritize Tasks
Aviate
Observation: Cockpit Task
Management Errors
Navigate
Communicate
Manage Systems
CTM
• Cockpit (flight deck) is a multitask environment
–
–
–
–
aviate
navigate
communicate
manage systems
• Results of distraction, preoccupation
– Everglades L-1011 accident
– many incidents
• Hypotheses:
– flightcrew must manage as well as perform tasks:
Cockpit Task Management (CTM)
– CTM is a significant factor in flight safety
2
Aviate
Preliminary Normative
Theory of CTM
•
•
•
•
Navigate
Communicate
Manage Systems
CTM
initiate tasks to achieve goals
assess status of all tasks
terminate completed and ‘obsolete’ tasks
prioritize remaining tasks based on
– importance:
•
•
•
•
aviate
navigate
communicate
manage systems
– urgency
– other factors (?)
• allocate resources (attend) to tasks in order of priority
3
Aviate
Cockpit Task Management
Research
Navigate
Communicate
Manage Systems
CTM
• CTM Errors in Aircraft Accidents (1991)
– 80 CTM errors in 76 (23%) of 324 accidents
• CTM Errors in Critical, In-Flight Incidents (1993)
– 349 CTM errors in 231 (49%) of 470 incident reports
• Part-Task Flight Simulator Study (1996)
– CTM error rate increases with workload
• ASRS Study of CTM and Automation (1998)
– Task prioritization error rate higher in advanced technology reports
• Findings:
– CTM is a significant factor in flight safety
– CTM can potentially be improved
4
Aviate
Navigate
Communicate
Manage Systems
CTM
Improving CTM Through Technology:
The
AgendaManager
5
Aviate
Statement of Needs and
Requirements Definition
Navigate
Communicate
Manage Systems
CTM
• CTM aid shall
– maintain a current model of aircraft state and current
cockpit tasks,
– monitor task state and status,
– compute task priority,
– remind the flightcrew of all tasks that should be in
progress, and
– suggest that the flightcrew attend to tasks that do not
show satisfactory progress.
– leave the pilot in control
6
Aviate
Navigate
System Analysis
Communicate
Manage Systems
CTM
• Generic, twin-engine transport aircraft
– major subsystems
•
•
•
•
•
•
•
power plant
fuel system
electrical system
hydraulic system
adverse weather system
autoflight system
flight management system.
– state variables of importance to pilot
•  specifications for simulator
7
Aviate
Basic and Detailed Design of
The AgendaManager
Navigate
Communicate
Manage Systems
CTM
• Object-Oriented Design
– things & activities from IDEF0 models  objects
• Multi-Agent Approach
– AMgt functions are complex, cognitive functions  AI
– AMgt is complex interplay of many entities  DAI
•
•
•
•
•
•
System Agents
Actor Agents
Goal Agents
Function Agents
Agenda Agent
Agenda Manager Interface
• Display Design
– general display design guidelines  alternative display designs
– consistency with EICAS  final display design
8
AMgr Architecture and
Function
AMgr display
Pilot
Verbex ASR
information flow
satisfactory functions
unsatisfactory functions
conflicting goals
reduce to 240 kt Goal & Function Agents
maintain 070 deg G & F Agents
Flightcrew Agent
Aircraft
Aircraft Agent
Autoflight
Autoflight Agent
descend to 9,000 ft G & F Agents
reduce to 240 kt G & F Agents
maintain 070 deg G & F Agents
descend to 8,000 ft G & F Agents
L Engine
L Engine Agent
extinguish L ENGINE FIRE G & F Agents
Hyd System
Hyd System Agent
restore C HYD PRESS G & F Agents
Fuel System
Fuel System Agent
correct FUEL BALANCE G & F Agents
System Models
System Agents
Simulator
Goal & Function Agents
AgendaManager
Aviate
Simulator
(with EICAS)
Navigate
Communicate
Manage Systems
CTM
10
Aviate
AMgr Display
(replaced EICAS)
Navigate
Communicate
Manage Systems
CTM
11
Aviate
Navigate
AMgr Operation
Communicate
Manage Systems
CTM
•
•
•
•
•
•
simulator runs
pilot declares goals via ATC acknowledgements
System & Actor Agents instantiate Goal Agents
Goal Agents watch for goal conflicts
Function Agents assess function status
AgendaManager informs pilot via display
12
AgendaManager Display Design
extinguish L engine fire
not OK -> continuing
descend to 7,000 ft
maintain 070 deg
slow to 240 kt
A/F alt goal conflict
correct fuel balance
L heavy -> increasing
fast -> accelerating
extremely important,
urgent goals
(highest priority)
trend info
aviate goals
(high priority)
extinguish L engine fire
not OK -> continuing
descend to 7,000 ft
maintain 070 deg
slow to 240 kt
A/F alt goal conflict
correct fuel balance
L heavy -> increasing
gray = OK
amber = not OK
red = important/urgent not OK
fast -> accelerating
system goals
(lower priority)
Initial Conditions:
altitude = 15,000 ft
heading = 120 deg
speed = 280 kt
all systems normal
maintain15,000 ft
maintain 120 deg
maintain 280 kt
ATC: “... descend and maintain 11,000 ft”
pilot: “Roger, “... descend and maintain 11,000 ft”
sets A/F altitude to 11,000 ft
descent begins
descend to 11,000 ft
maintain 120 deg
maintain 280 kt
high -> descending
ATC: “... turn left heading 070”
pilot: “Roger, “... turn left heading 070”
begins turn
levels off at 11,000 ft
maintain 11,000 ft
turn L to 070 deg
maintain 280 kt
right of -> turning L
pilot: rolls out on 070 deg
AMgr:detects fuel imbalance & displays it
maintain 11,000 ft
maintain 070 deg
maintain 280 kt
correct fuel balance
L heavy -> unbalancing
pilot: begins fuel crossfeed
ATC: “... descend and maintain 9,000 ft; reduce
speed to 240 kt”
pilot: “Roger ... descend and maintain 9,000 ft;
reduce speed to 240 kt”
sets altitude to 9,000 ft, descent begins
reduces throttles, aircraft slows
descend to 9,000 ft
maintain 070 deg
slow to 240 kt
high -> descending
correct fuel balance
L heavy -> balancing
fast -> slowing
AMgr: detects left engine fire
pilot: “... we have a problem ...”
ATC: “... descend and maintain 7,000 ft”
pilot: “Roger ... descend and maintain 7,000 ft”
mis-sets altitude to 6,000 ft
speed increases
extinguish L engine fire
not OK -> continuing
descend to 7,000 ft
maintain 070 deg
slow to 240 kt
A/F alt goal conflict
correct fuel balance
L heavy -> balancing
fast -> accelerating
fire out
speed controlled
pilot: sets A/F to 7,000 ft
forgets to secure crossfeed when fuel balanced
maintain 7,000 ft
maintain 070 deg
maintain 240 kt
correct fuel balance
R heavy -> unbalancing
Aviate
Navigate
Communicate
Test and Evaluation (1)
Manage Systems
CTM
• Objective: compare AMgt performance (AMgr vs EICAS)
• Apparatus
– flight simulator
– AMgr
• Subjects: 8 line pilots
• Scenarios:
– EUG to PDX
– PDX to Eugene
• Primary factor: monitoring and alerting condition
– AMgr
– EICAS
22
Aviate
Navigate
Test and Evaluation (2)
Communicate
Manage Systems
CTM
• General Procedure
–
–
–
–
–
subject introduction
automatic Speech Recognition system training
flight training (using MCP)
subsystem training (fault correction)
EICAS/AMgr training
• Trials
– Scenario 1 (EICAS/AMgr)
• experimenter/ATC controller gives clearances, induces faults, induces
goal conflicts
• subject acknowledges clearances, flies simulator, corrects faults,
detects and resolves goal conflicts
– Scenario 2 (AMgr/EICAS)
23
Aviate
Navigate
Communicate
Evaluation Results
Manage Systems
CTM
Response variable
AMgr
EICAS sig. level
within subsystem correct prioritization
100%
100%
NS
susbsystem fault correction time (sec)
19.5
19.6
NS
autoflight system programming time (sec)
7.0
5.9
NS
goal conflicts corrected percentage
100%
70%
0.10
goal conflict resolution time (sec)
34.7
53.6
0.10
subsystem/aviate correct prioritization
72%
46%
0.05
average number of unsatisfactory functions
0.64
0.85
0.05
percentage of time all functions satisfactory
65%
52%
0.05
mean subject effectiveness rating (-5 to 5)
4.8
2.5
0.05
24
Aviate
Navigate
Conclusions
Communicate
Manage Systems
CTM
• CTM is a significant factor in flight safety.
• CTM can be facilitated (e.g., AMgr).
• Future success of knowledge-based avionics depends
on a systematic approach to development:
– systematic identification of problems, needs, and
opportunities
– appropriate application of appropriate technology
– evaluation of systems based on operationally relevant
performance measures
25
Aviate
Navigate
Communicate
Manage Systems
CTM
Improving CTM Through Training:
Training Pilots to
Prioritize Tasks
26
Aviate
Research
Motivation and Objective
Navigate
Communicate
Manage Systems
CTM
• Is task prioritization trainable?
• Evidence suggests that voluntary control of
attention is a trainable skill
– e.g., Gopher (1992)
• Objective
– Develop and evaluate a CTM training program to
improve task prioritization performance.
Aviate
Navigate
Communicate
Manage Systems
Methodology
CTM
• Participants
– 12 General Aviation pilots, IFR rated, with at least 100 hrs “pilotin-command” total time.
– Recruited through flyers and word of mouth
– Oregon State (Corvallis, Albany, Salem, Eugene, Portland)
• Apparatus: Microsoft Flight Simulator 2000
– 3 monitors, Flight Yoke, Throttles, and Rudder Pedals
– IFR conditions
– Two flight scenarios
Aviate
Lab Setup
Navigate
Communicate
Manage Systems
CTM
Aviate
Participant Display
(C-182RG)
Navigate
Communicate
Manage Systems
CTM
Experimenter’s Display
Aviate
Navigate
Communicate
Manage Systems
CTM
Aviate
Experimental Groups
Navigate
Communicate
Manage Systems
CTM
• Control Group: No Training
• Descriptive Group: CTM lecture
•
•
•
•
•
•
Multi-tasking
Attention
CTM
Task Prioritization errors
Accident/Incident examples
What to be aware of.
• Prescriptive Group:
– CTM lecture
– “APE” procedure
Aviate
APE:
Assess Prioritize Execute
•
Let the APE help you
– Assess the situation:
•
•
Navigate
Communicate
Manage Systems
CTM
A P E
aircraft systems, environment, tasks, procedures
“What’s going on?” “What should I be doing?”
– Prioritize your tasks:
1. Aviate: “Is my aircraft in control?”
2. Navigate: “Do I know where I am and where I’m going?”
3. Communicate: “Have I communicated or received important
information?”
4. Manage systems: “Are my systems okay?”
– Execute the high priority tasks Now.
•
•
Invoke the APE frequently.
Think out loud.
Aviate
Experimental Procedure
Navigate
Communicate
Manage Systems
CTM
1.
2.
3.
4.
5.
6.
7.
Initial briefing, informed consent
Initial 30-minute simulator training
Pre-training flight
CTM training (break for control group)
Additional 30-minute simulator training
Post-training flight (different scenario)
Post-experiment questionnaire
Aviate
Dependent Measures
Navigate
Communicate
Manage Systems
CTM
• Task prioritization error rate
– 19 Task prioritization challenges, e.g.
• clearance near end of climb
• “bust” altitude? (+/- 200 ft)
• Prospective memory recall rate
– 5 Memory recall challenges (prospective memory), e.g.,
• “report crossing SHONE [intersection]”
• remember to report?
Aviate
Data Collection
Navigate
Communicate
Manage Systems
CTM
•
•
•
•
Flight Data Recorder
Videotape
Observation
Data reduction to:
– task prioritization error rate
– prospective memory recall rate
Aviate
Results: ANOVA
(task prioritization error rate)
Effect
df
MS
Navigate
Communicate
Manage Systems
CTM
df
SS
F-ratio
p-value
Group
2
.1914125
9 .0799194
2.395
.147
Flight
1
.2053500
9 .0088806
23.123
.001
Group x Flight
2
.0429125
9 .0088806
4.832
.038
Aviate
Interaction Plot
(task prioritization error rate)
Navigate
Communicate
Manage Systems
CTM
Interaction Plot
0.7
Error Rate
0.6
Group
Co
De
Pre
Prescriptive
0.5
0.4
Descriptive
0.3
0.2
0.1
Control
0
Pre Training
Post Training
Flight
Aviate
Results: ANOVA
(prospective memory recall rate)
Effect
Navigate
Communicate
Manage Systems
CTM
df
SS
df
MS
F-ratio
p-value
Group
2
.017
9
.028
.603
.568
Flight
1
.074
9
.034
2.181
.174
Group x Flight
2
.171
9
.034
5.055
.034
Aviate
Interaction Plot
(prospective memory recall rate)
Navigate
Communicate
Manage Systems
CTM
Interaction Plot
Gr
C
D
P
Memory Tasks
1
0.9
0.8
Control
0.7
Descriptive
0.6
Prescriptive
0.5
Pre Training
Post Training
Flight
Aviate
Paired t-tests
Navigate
Communicate
Manage Systems
CTM
• Prescriptive training group improved
• Task prioritization error rate
• Prospective memory recall rate
• Descriptive training group improved
• Task prioritization error rate
• Control group did not significantly improve
Aviate
Discussion
Navigate
Communicate
Manage Systems
CTM
• Task Prioritization Error rate
– Reduced, perhaps, due to (Prescriptive) CTM training.
– Significant interaction and post-hoc tests support that
hypothesis.
• Prospective Memory Recall rate
– Increased, perhaps, due to (Descriptive & Prescriptive) CTM
training.
– Significant interaction and post-hoc tests support that
hypothesis.
Aviate
Possible Interpretations
Navigate
Communicate
Manage Systems
CTM
•
Results may have two interpretations:
1.
2.
CTM training did improve task prioritization performance.
CTM training did not improve task prioritization.
•
•
•
•
Floor effect
MSFS experience
Age
Research favors first interpretation
•
•
•
•
ANOVA results
t-tests
Potential for better control group performance was there.
Additional tests
Aviate
Final Comments
Navigate
Communicate
Manage Systems
CTM
• CTM performance significant to flight safety
• Results are encouraging
• Evidence suggests that task prioritization is a
trainable skill
• Follow-up experiment underway to resolve
ambiguities
• If successful, would provide evidence that CTM
training can reduce risk of CTM errors and
subsequent accidents
Aviate
The Cockpit Task
Management Website
Navigate
Communicate
Manage Systems
CTM
http://flightdeck.ie.orst.edu/CTM/
The AMgr: a KBS
45