Engaging Innate Human Cognitive Capabilities
to Coordinate Human Interruption in HumanComputer Interaction: The HAIL System
Operator Console
Cognitive
System
Components
Weapon System
HAIL
Engine
Dr. Daniel C. McFarlane
Lockheed Martin Advanced Technology Laboratories
[email protected]
856-792-9708
June 30, 2003
Table of Contents
•
HAIL Overview
•
Cognitive System Research
•
Cognitive System Development
•
R&D Process
•
Summary
6/30/036/30/03-2
HAIL Overview
•
What is HAIL?
•
Technology
Overview
Operator Console
Cognitive
System
Components
Weapon System
HAIL Engine
6/30/036/30/03-3
What is HAIL?
•
•
Human Alerting and Interruption Logistics alert
mediation technology
HAIL has two parts:
(1) The HAIL Engine: A cognitive system that delivers
cognition-based resources for human alerting activities
based on proven empirical human-subjects research
(2) The HAIL System: An open architecture decision-support
system (DSS) component built around the HAIL Engine
that delivers the capabilities of the HAIL Engine for
human decision-makers within their work environments
•
The first transition of the HAIL technology is into
the US Navy’s Aegis Weapon System
The
TheHAIL
HAILtechnology
technologyhas
hastwo
twoparts:
parts:(1)
(1)AAcognitive
cognitivesystem
systemas
asaacore
coreHAIL
HAIL
Engine;
Engine;and
and(2)
(2)AAdecision-support
decision-supportsystem
system(DSS)
(DSS)that
thatrealizes
realizesthe
thecapabilities
capabilities
of
ofthe
theHAIL
HAILEngine
Enginefor
forhuman
humanusers
usersin
inreal
realworld
worldwork
workenvironments.
environments.
6/30/036/30/03-4
What is HAIL? (cont.)
Support for increased human capability:
(1) Reduce human interruption rate by decoupling
the alert rate from human interruption rate
–
The HAIL solution is generic and does not address
the platform-specific issue of reducing alert rate
overall
(2) Increase human control of information
environment with negotiation-based
coordination of interruption events
(3) Increase operator access into past information
with context recovery aids
HAIL
HAILis
isaaplatform-independent
platform-independentsoftware
softwarecomponent
componentthat
thatactualizes
actualizesthe
the
accepted
acceptedNRL
NRLNCARAI
NCARAIalerting
alertingresearch
researchfindings
findingsfor
forimproving
improving
operator
operatorperformance
performancein
inthe
thepresence
presenceof
ofalerts.
alerts.
6/30/036/30/03-5
Technology Overview (cont.)
Aegis UI
Core UI
Interface
Manager
HAIL Engine
Alert Managers
Information
Generation
The
TheHAIL
HAILEngine
Engineresides
residesat
atthe
theBusiness
BusinessTier
Tierin
inan
anN-Tier
N-TierArchitecture.
Architecture.
ItItencapsulates
encapsulatesproven
provenbusiness
businesslogic
logicon
onhow
howhumans
humanssucceed
succeedat
at
making
makingeffective
effectivedecisions
decisionsin
inalerting-dense
alerting-densework
workenvironments.
environments.
6/30/036/30/03-6
Technical Overview (cont.)
RUP-guided transformation of proven research results into a mature component-based
cognitive system component
•
Founded on
•
Process
•
HAIL Alert Engine – A Reusable Component-Based
Open System Approach
— Vision of future human workload for interrupt-driven
responsibilities
— Interdisciplinary NRL research
— Documented requirements from multiple domains
—
—
—
RUP-guided (Rational Unified Process)
Iterative spiral software development
User-centric UI prototyping and usability testing
— OO, Open Architecture, User-Centric, Modular, Extensible,
Reusable, Well Documented, Robust, Scalable
The
TheHAIL
HAILsoftware
softwaredevelopment
developmentapproach
approachis
isusing
usingthe
thesame
sametools
toolsand
and
process
processas
asAegis
AegisOpen
OpenArchitecture
Architectureto
tominimize
minimizetransition
transitioneffort.
effort.
6/30/036/30/03-7
Cognitive Systems Research
•
Vision
•
Problem
Description
•
Approaches
•
Examples
•
Research
Foundation
•
Operator Console
Cognitive
System
Components
Weapon System
HAIL Engine
Human-Subjects
Experiment
6/30/036/30/03-8
Vision of future human workload for
interrupt-driven responsibilities
•
•
Quote: “Future human-computer interface will
be rooted in delegation, not the vernacular of
direct manipulation.” (Nicholas Negroponte,
“Being Digital,” 1995, p. 101)
Increased alerting requirement identified
through experience in applied research on
intelligent command and control systems at
NRL
— ISART (Interactive Situation Assessment and
Rollup Tool)
6/30/036/30/03-9
Problem Description
•
Success of tactical operations depends on high quality
decision-making. However, this is not possible without better
alerting technology
•
The problem has four parts:
•
There is a threshold for people’s ability to handle interruptions
without support, and when this threshold is exceeded,
situational awareness fails and decision quality drops
1) People have cognitive limitations
2) No decision-support for management of context switching in
current alerting mechanisms
3) Alerting mechanism delivers a large quantity of critical
information but its current human interruption rate outpaces
operators’ ability to keep up
4) Information overload continues to worsen
People
Peoplehave
havelimited
limitedability
abilityto
tohandle
handleinterruptions
interruptionswithout
withoutsupport.
support.
6/30/036/30/03-10
Approaches
• Remove person
• Training
• Incentives
• Selection of personnel
• Construction of a cognitive system for a
DSS
6/30/036/30/03-11
Example Domain – Commercial Air
•
National Transportation Safety
Board, 1988
— Northwest Airlines
— Detroit Metropolitan Airport
— McDonnell-Douglas DC-9-61
— Preflight checklist
interrupted by Air Traffic
Control
— Change in taxi-way
instructions and weather
alert for possible windshear
— Never resumed checklist
task; Flaps in wrong
position for take-off
— Task resumption aids could
have prevented this
accident
A. Too
many interruptions
B. No operator control
C. No trend information
D. No recovery support
Many
Manyplatforms
platformscould
couldbe
beimproved
improvedwith
withInterruption
InterruptionSupport
Support
technology.
technology.
6/30/036/30/03-12
Example Domain – Aegis Alerting
6/30/036/30/03-13
Example Domain – Aegis Alerting
Menu Bar
Primary Secondary
Hooked Hooked
Track
Track
Track
Window Window
Plasma
Touch
Panel
3. Operator decides to press
“REVIEW ALERTS” button.
Tactical
Information
Windows
4.
4. These
These displays
displays are
are
refocused
refocused for
for the
the alert
alert task.
task.
A.
B.
Tactical Situation
Display Area
(TACSIT)
C.
D.
Alert
Alert Window
Window
Status Window
2.
2. Alert
Alert is
is displayed
displayed in
in the
the window.
window.
Buzzer
Too many
interruptions
No operator
control of
alert queue
No trend
information
No support
for context
recovery
after
interruption
1. Buzzer sounds
when each alert
enters the queue.
Variable
Action Buttons
“REVIEW ALERTS”
Fixed Button
Aegis
Aegisneeds
needsthe
theHAIL
HAILtechnology
technologyto
toremain
remain“Fully
“FullyMission
MissionCapable.”
Capable.”
6/30/036/30/03-14
Research Foundation
All HAIL publications are available on the HAIL-SS web site
http://www.atl.lmco.com/projects/hail/
SSC-SD
TADMUS
NASA
Research
University
ONR
Manning
Affordability
Other
MSR
NRL
HAIL
Product
Development HAIL Engine
The results of McFarlane’s NRL HAIL Research were
published as a two-article stand-alone issue of the journal
“Human-Computer Interaction,” 2002, 17(3).
•
Results
— Interruption negatively
impacts performance
— Negotiation support for
interruptions gives
biggest overall win
except when an alert
task is time-critical
— People perform better
when primed for
interruption
— People switch contexts
more easily when given
context recovery
support
HAIL
HAILis
isfounded
foundedon
onproven
provenempirical
empiricalresearch.
research.
6/30/036/30/03-15
HAIL Basic Research
• Goal: to invent a cognitive system to support a
DSS to support increased human workload
capability for alert-based activities
• Theoretical products
—A General Definition of Human Interruption
—A Taxonomy of Human Interruption
• Empirical findings
—Negotiation is the best overall interruption
coordination solution except for cases where
small differences in timelines of handling
alerts are critical
6/30/036/30/03-16
Human-Subjects Experimentation
Single-factor within-subjects Latin
squares design
• 36 subjects (18 male, 18 female), avg.
age 24.7 (min. 18, max. 47)
• 6 treatments — 4 experimental + 2
control
• 24 trials; 4.5 minute each
• 1 hour of practice followed by 1 hour
of experiment
• Pilot testing to set difficulty (59, 80)
•
6/30/036/30/03-17
Negotiated is best for primary task.
6/30/036/30/03-18
Immediate is worst for interruption task.
6/30/036/30/03-19
Small timeliness problem with Neg. and Sch.
6/30/036/30/03-20
Negotiated is efficient for interruption task.
6/30/036/30/03-21
Negotiated is best for keying effectiveness.
6/30/036/30/03-22
Negotiated is best for keying efficiency.
6/30/036/30/03-23
Cognitive System Development
•
•
•
•
•
•
•
•
•
•
Product Overview
Scope of HAIL
technology
HAIL technology
capabilities
Technical Approach
Operator Console
Cognitive
System
Components
HAIL Engine Capabilities
Work Process
Software Development
Operational Impact
Programmatic Overview
HAIL Team Overview
HAIL Engine
Weapon System
Product Overview
Domain-independent decision-support engine for alert management
HAIL-SS is producing two products
(1) A platform-independent software engine that delivers a
intelligent alert management critical future naval capability for
improved C2 decision-support in dense information
environments
Before HAIL
After HAIL
HAIL-SS
HAIL-SSis
isproducing
producingaareusable
reusablegeneral
generalcomponent-based
component-basedsoftware
software
product
productfor
foralert
alertmanagement.
management.
6/30/036/30/03-25
Product Overview (cont.)
Aegis is the first transition target
HAIL-SS is producing two
Products (cont.)
(2) A HAIL Engine integrated,
tested, and validated
within an Aegis Weapon
System
HAIL-SS
HAIL-SSis
isproducing
producingaavalidated
validatedintegration
integrationof
ofthe
theHAIL
HAIL
Engine
Enginewithin
withinthe
theAegis
Aegisplatform.
platform.
6/30/036/30/03-26
Scope of HAIL-SS Technology
Three Development Activities for a C2 Alerting Mechanism
User Interface:
• Human Factors Eng. to
dynamically balance
multimedia cueing to
match operators’
cognitive needs
(flashing, alarm
sounds, color, spatial
layout)
• Dependent on domain
• Research complete: no
Cognitive System
Component [HAIL
Engine]:
• DSS engineering to manage
alerts (negotiation-based
task management;
interactive visualization;
automatic trend inference;
context recovery support)
• Independent of domain
• Research complete: yes
Alert Generation:
• Systems engineering
to generate alerts to
support tactical action
(hard real-time
computing; operations;
tactical role modeling)
• Dependent on domain
• Research complete:
no
Application
The
TheHAIL
HAILEngine
Enginewill
willenhance
enhanceboth
bothcurrent
currentand
andfuture
future
UI
UIand
andAlert
AlertGeneration
Generationtechnologies.
technologies.
6/30/036/30/03-27
HAIL Technology Capabilities
Cognitive System
Capabilities
Broker alert
management
Alert management
Negotiation-based
support for
coordination
Visualization support
for SA
Context recovery
support
Trend inference
automation
Purpose
Warfighter
Benefits
Break design gridlock
Better alert SA
Decouple alert rate
from interruption rate
Leverage human
ability to dynamically
prioritize and schedule
Fewer interruptions
Leverage human
ability to assimilate
information
Minimize human
weaknesses for
context switching
Minimize human
weaknesses for
monitoring change
over time
Receive informational
alert content without
interruption
Switch contexts
without making errors
Optimum prioritization
and scheduling of
interruptions
Detect significant
trends
6/30/036/30/03-28
Technical Approach
•
Integrated technology definition/design/transition process
Leverage existing research
Reusable HAIL alerting engine
•
Use Navy domain application to
•
•
•
•
•
— Open architecture, components-based design and development
— Multiple domain applications driving requirements
— Define requirements
— Test and evaluate in a
realistic environment
— Support transition
Design
Develop
Transition
Link with AOA schedule
requirement
Phase 1: Test and Evaluation
of First Prototype
Phase 2: Evaluation in Operational
Setting using Baseline 7P1
Capture
Requirements
Evaluation
Integration
and Test
AA2-cycle
2-cycledevelopment
developmentapproach
approachensures
ensuresthat
that
the
theHAIL
HAILproduct
productis
isuser-centered.
user-centered.
6/30/036/30/03-29
HAIL Alerting Engine Capabilities
•
Action alert management
— Supports multiple
interruption strategies
User
Interface
HAIL-Based
Alerting Application
•
Presentation management
— Supports presentation of
information-only alerts in
multiple modalities
•
Context recovery support
— Supports domain-dependent
recovery mechanisms
HAIL Technology
•
Automated assessment
— Supports inference of higherlevel alerts and trends
HAIL
HAILtechnology
technologyprovides
providesfuture
futurenaval
navalcapability
capabilityfor
for
action
actionalert
alertmanagement,
management,presentation
presentationmanagement,
management,
context
contextrecovery
recoverysupport,
support,and
andautomated
automatedassessment.
assessment.
6/30/036/30/03-30
Work Process
Human
Cognitive
Requirements
HAIL
Research
Core HAIL
Technology
• Detailed design
• Architecture
Application
Requirements
Definition
Risk Definition
Prototype
PlatformSpecific
Application
Design
Prototype
PlatformSpecific
Application
Evaluation
Design
Integrate
and Test
Evaluation
Human
Humancognitive
cognitiverequirements
requirementsand
andapplication
application
requirements
requirementsdrive
drivethe
thedevelopment
developmentprocess.
process.
6/30/036/30/03-31
RUP-Guided S/W Development Process
HAIL process and toolset is consistent with Aegis Open Architecture
•
UML Documentation
— Use Cases
— Activity Diagrams
•
Visio and Altia Design Mockups for supporting
spiral iterations of user Interface (UI) design and
usability testing.
HAIL
HAILis
isusing
usingindustry
industrystandard
standarddevelopment
developmentprocess
processand
and
documentation
documentationartifacts.
artifacts.
6/30/036/30/03-32
Operational Impact (Estimated)
Successfully Handle Higher Information Volume
•
Better decision-making and situational awareness of
naval warfighters through support for human alerting
•
Product will have general utility potential for several
platforms; examples are for Aegis
—
—
—
—
—
—
—
—
Frequency of interruptions reduced 50-80%
Button actions required reduced 50-80%
Time and effort required for critical alerts improved 15-50%
Correlate alerts with tracks on the TACSIT improved 40-60%
Efficiency in resuming primary task after interruption
improved 10-30%
Perceived workload reduced 10-30%
Quality of situational awareness improved
Perceived stressfulness reduced 20-30%
The
TheHAIL
HAILcognitive
cognitivesystem
systemwill
willsupport
supportgreatly
greatlyimproved
improvedhuman
human
capabilities
capabilitiesfor
fordealing
dealingwith
withalerts
alertsin
incomplex
complexmultitasking
multitaskingwork
workcontexts.
contexts.
6/30/036/30/03-33
Programmatic Overview
Funded by ONR and founded on the results of interdisciplinary theory
theory--based research from NRL
NRL
NRL/ONR
HAIL ‘95-‘00
ONR KSA FNC
HAIL-SS ‘02-‘04
PEOTSC
Aegis
‘05-‘?
(existing TTA)
Integration
Research
Product Development
Technology Maturity
Measured in TRL’s
Other Naval
Systems
Other; …
•
The HAIL-SS Web Site: http://www.atl.lmco.com/projects/hail/
•
HAIL-SS is: An ONR KSA FNC project to develop and test a generic
alerting software component that actualizes the accepted NRL NCARAI
alerting research findings for improving operator performance in the
presence of alerts
•
HAIL-SS is on contract with the ONR KSA FNC Program (N00014-02-C0014, begun 4/02), LM ATL lead
The
TheHAIL
HAILtechnology
technologystarts
startswith
withNRL-proven
NRL-provensolution
solutionto
toaarecognized
recognized
future
futureNavy
Navyproblem
problemand
andis
ispaying
payingdown
downthe
therisk
riskfor
foracquisition.
acquisition.
6/30/036/30/03-34
HAIL-SS Team Overview
Full-coverage team for creation of a successful HAIL product
Surface Ship Combat
System Engineering Agent
LM NE&SS-SS, CSC
HAIL
Technology
LM ATL (lead), NRL
Receiving Command
PEO-IWS, PMS 400
PMS 461, PMS 378
HAIL Product
Transition into Fleet
Operational
SMEs
BCI, ATRC,
CSEDS
This
Thisintegrated,
integrated,diverse
diverseteam
teamprovides
providesinsight
insight
and
andexpertise
expertisecovering
coveringall
allareas
areasrelating
relatingto
toAegis.
Aegis.
6/30/036/30/03-35
R&D Process
•
Process
Operator Console
Cognitive
System
Components
Weapon System
HAIL Engine
6/30/036/30/03-36
Process
Mature COTS
Technology
Publication in
Public Domain
HAIL-SS
(ONR KSA FNC ’02-’04)
TRL 3
ISART HAILTRL 2
(NRL ‘95-‘00)
(ATL ‘00)
TRL 4
TRL 5
TRL 6
TRL 7
LM NE&SS
Aegis
Product
Development
Fielded
Surface
TRL 8
Combatant
TRL 1
TRL 9
Lab
Integration
and Test
Research
“The theorytheorybased empirical
experiments”
Future Mission
Requirements
“Fully mission
capable”
Better Human Alerting Mechanisms
(Lessons Learned ‘91-’92)
HAIL-SS
HAIL-SSis
isan
anONR-funded
ONR-fundedproduct
productmaturation
maturationeffort.
effort.
6/30/036/30/03-37
Summary
•
Summary
Operator Console
Cognitive
System
Components
Weapon System
HAIL Engine
6/30/036/30/03-38
Summary
•
HAIL is producing a platform-independent
technology composed of two things:
(1)
(2)
•
The HAIL Engine: A cognitive system that delivers
cognition-based resources for human alerting
activities
The HAIL System: An open architecture decisionsupport system (DSS) component built around the
HAIL Engine
HAIL supports increased human capability:
(1) Decouples the alerting rate from the human
interruption rate
(2) Increases user control for coordination of interruption
events and recovering context after interruption
6/30/036/30/03-39