Improving Collaboration in
Unmanned Aerial Vehicle Operations
March 27, 2007
Stacey D. Scott
Humans & Automation Lab
MIT Aeronautics and Astronautics
http://halab.mit.edu
Teamwork in Unmanned Aerial Vehicle (UAV)
Operations
UAV Operators
Intelligence Consumers
(e.g., Ground Troops)
Shadow UAV
Ground Crew
Lawyers & Politicians
Supporting Futuristic UAV Teams Engaged in TimeCritical Operations
Future UAV Teams
Current Collaboration Tools Insufficient for TimeCritical Operations
• Increasing reliance on collaboration tools
- e.g., email, instant messaging (chat)
• These tools introduce communication
overhead, hindering management of tasking &
team coordination (Scott et al., 2006)
• Trend in corporate and educational collaboration
systems towards intelligently sharing
activity information to improve planning &
coordination in networked teams
- Activity awareness design approach (Carroll et
al., 2006)
• Promoting activity awareness in UAV operator
interfaces may:
- help UAV teams coordinate their actions
- reduce the costs of obtaining activity-related
information
Research Goals: Supporting Activity Awareness in
Collaborative UAV Operations
• To develop and evaluate activity awareness tools for
future teams of UAV operators engaged in time critical
missions
• To investigate the software automation and interface
requirements of UAV operations team members
- what type of information is useful to know about one’s team
members and their task activities?
- what information do team members need to know about the
UAVs’ activities?
- how does one’s job duties / team role impact activity awareness
requirements (e.g. mission commander, UAV operator, etc.)?
Initial Research Focus
Future UAV Teams
Technological Support
for the Team Supervisor
Assisting Team Supervisors of UAV Teams
Research Approach:
1. Develop representative collaborative, time-critical C2 task scenario
2. Conduct cognitive task analysis (CTA) to generate system
requirements for supervisory decision support and activity
awareness displays
3. Develop supervisory displays in a simulated task environment
4. Conduct exploratory experiment
Representative Collaborative UAV Task Scenario
Ground Force Protection:
• Futuristic UAV Operations Team performing ISR in support of a timesensitive targeting for a ground convoy traversing a hostile region
Task Details:
• UAV Operations Team:
- 1 Mission Commander
- 3 UAV operators, each controlling
multiple semi-autonomous UAVs
• UAVs have camera sensors only
- Team must coordinate with
external strike team
• UAV team can communicate with convoy
& external intelligence sources
Generating Requirements for Assisting Supervision
Hybrid Cognitive Task Analysis (Nehme et al., 2006):
Step
Step
Step
Step
Mission
Planning
1:
2:
3:
4:
Scenario Task Overview
Event Flow Diagram
Situation Awareness (SA) Requirements
Decision Ladders & Display Requirements
Issues to be resolved in this phase:
Helpful information for resolving these issues:

 Time average target destruction will take based
on
-UAV speed :transit time
-engagement time: flagged EO imagery
sent/operator confirm/communicate with
strike team/strike team wait time)

Each UAV will have initial routes
programmed
Each operator will have a pre-defined area
under his/her responsibility
Produces Functional &
Information Requirements
 AWACS, satellite and strike team schedule
1
Phase goals
Phase Breakdown

ISR



STR
- Strike

2
Mission
Execution



BDA
- battle damage
assessment




UAV-LEx
down/
malfunction


UAV down/
malfuntion



3
UAVOP monitors his region of coverage, while the UAVs search for
IED
AWACS, Satellites, strike team and the convoy detect additional
enemies and communicate to the UAVOPs.
UAVOP receives ISR information from the UAV and requests
engagement authorization from the Mission Commander
The mCDR decides if the target should be stroke and either grants the
authorization or denies it.
When granted, the UAVOP orders the strike team to engage the target
The strike team coordinates with the UAVOP a time window for the
target engagement
When the Strike team is ready to engage, the UAVOP places his UAVLE to pinpoint the target to be destroyed, during the appropriate time
window
The strike tem engages the target of interest
UAVOP performs Battle Damage Assessment, based on information
provided by their UAVs, by the strike team and by the AWACS.
UAVOP analyzes the consistency to confirm the destruction of the
target or to order a re-strike.
All operators and mCDR are notified about the loss
When the next target is found by the UAVOPx, the team is notified that
an UAV-LE is needed.
Operators that can handover their UAV-LE send a signal (or a message)
and the mCDR decides whose UAV-LE will be handed over.
If no other UAV-LE seems to be available, the mCDR decides whether
a UAV-LE should be handed over or not, according to their priorities
and strategy.
All operators and mCDR are notified about the loss
Operators that can handover a UAV send a signal (or a message) and the
mCDR decides which UAV will be handed over.
If no other UAV seems to be available, the mCDR decides whether a
UAV should be handed over or not and if a re-zone should take place or
not, according to their priorities and strategy.
4
D3
Display strength of
convoy comm link
and availability
If holding back convoy, can release?
Interpret: Decide whether to
request convoy resumption
RECOGNIZE if the convoy is released,
threat level will remain low for
immediate future
Display measurement of
threat level based on
threats within envelope
Display
Requirements
Recognize if convoy
communications link is
active
Display convoy
acknowledgement if
release received, and
updated positioning of
convoy
IDENTIFY If threat level will remain low after
convoy resumes mission path
Regional map depicts threat locations and uncertainty
of surveyed areas. Simultaneously depict information
on known targets as well as up to date strike
schedule.
Send release request
to convoy
Perceive convoy
receives hold release
and is resuming
mission path
Perceive convoy’s current and future threat level
based on known targets’ locations/weapon’s range
with respect to convoy, current strike schedule,
and uncertainty of surveillance
Visual alert and possible
audio alert of convoy
holding; threat situation as
shown by targets within
threat envelope visual
Activation: Threat situation was passed and
convoy still holding (e.g. no known threat
envelope for immediate future)
Monitoring
Execute: Convoy resuming progress
Supervisory Displays & Simulated Task
Environment
Large-Screen
Wall Displays
Situation Map Display
TabletPC
Display
Mission Commander Display
Mission Status Display
Simulation &
Collaboration Server
(Grouplab SharedDictionary)
Designing to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy
safety
• Current & expected operator
task performance, relative to
convoy safety
1
(reviewing
ATR imagery)
4
(nominal)
8
Situation Map Display
(down)
Mission Status Display
Designing to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy
safety
• Current & expected operator
task performance, relative to
convoy safety
Target strike indicators
Potential threat
envelope
Situation Map Display
Known threat
envelope
Mission Status Display
Designing to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy
safety
• Current & expected operator
task performance, relative to
convoy safety
Situation Map Display
Mission Status Display
Exploratory Study
Subjects: 8 participants (including 6 NROTC)
Protocol:
- 1 subject + 3 “confederates” per experiment
- 2 hrs training + 2 x 15 min trials
- Retrospective interview
Situation Map Display
Mission Status Display
1 Subject
as mCdr
3 “Confederates”
as UAV Operators
Some Preliminary Results
• Convoy threat summary intuitive, useful for
timing command decisions
- Particularly, visualizing relationship between
strike team activity and convoy’s progress
• UAV status & tasking useful for deciding when
to assist UAV operators
- However, lack of time-on-task cues
required mental effort
• Operator performance displays rarely used,
since surveillance progress was easily obtained
from the Map
1
4
8
(reviewing (nominal) (down)
ATR imagery)
Ongoing Work: Test-bed for Exploring Other
Collaborative TST Issues
UAV Operations
Team
1. Interruption recovery
tools for the team
supervisor
Ongoing Work: Test-bed for Exploring Other
Collaborative TST Issues
2. Activity awareness
interfaces for the UAV
operators
3. New methods for
deriving collaborative
system requirements
UAV Operations
Team
1. Interruption recovery
tools for the team
supervisor
Expanded Simulated Task Environment
Large-Screen Wall Displays
Situation Map Display
Mission Status Display
Remote Assistance Display
Simulation &
Collaboration Server
(GroupLab SharedDictionary)
Mission Commander
Display (TabletPC)
Interruption Recovery Assistance for Team
Supervisor
Interactive event timeline,
containing event “bookmarks”
Mission Commander Display
(TabletPC)
Interruption Recovery Assistance for Team
Supervisor (cont’d)
Clicking on an event bookmark
highlights the location of the
corresponding event on the map
display, e.g., a convoy attack
Preliminary UAV Operator Display Designs
Comm Status
Op 2 – communication box
On line
0:03:57
All
[0:00:30] Conference call
Handoff Available
Op 1
[0:01:20] mCDR to OP1:
mCDR
Comm
0:03:57
Very Good
Speed: X Mb/s
MIT Network
[0:01:22] Op1 to Op3:
Op 3
[0:01:25] Op3 to strike team:
mCDR
[0:01:29] Op3 to Op2:
Strike team
Repair
Strike team
Comm
0:03:57
Very Good
Speed: X Mb/s
MIT Network
others
Mission Info:
Repair
OP1
Comm
0:01:53 - Changing shift
Very Good
Speed: X Mb/s
MIT Network
Repair
Send
private
OP3
0:00:33 - Target task
Comm
Type here…
Very Good
Speed: X Mb/s
MIT Network
Repair
mCDR
Others
Comm
0:03:57
Strike T
Op 1
Op 3
Others
ALERTS & Clarification
Very Good
Speed: X Mb/s
MIT Network
Repair
Share Files
Communications
Display
Elapsed Time:
0: 22: 45
Current Time:
1: 25: 30 PM
(0:01:50) UAV 7 (area3) has been shot down
Fuel level: 40%
4
Health Status
- Searching
(0:03:00) UAV 4 Fuel Level at 40%
View Filter:
Connected
0:03:57
Repair
Fuel level: 70%
5
Hazards
(0:03:40) UAV 6 enter in a difficult area to connect
Health Status
Connection
Blackouts
- Target ID
Connected
0:00:04
Repair
Targets Range
Fuel level: 70%
6
Health Status
0:00:18
Map
Display
Send
Type here…
Disconnected
Safety route in 0:29:59
Reconnecting in 0:00:49
Repair
Reroute/Reassignment
Target ID Task
Fuel level: 40%
3
- Searching
UAV1
Connected
UAV 3
Health Status
0:03:59
UAV 4
Fuel level: 70%
4
UAV3
- Searching
Connected
UAV 5
- Searching
UAV3
Connected
Request sent to:
Req 4
Request from: Op 3 Elapsed time: 0m30s
Detected by: UAV 8 Target type: 5 (Com Eq)
Req 5
Request from: Op 1 Elapsed time: 0m05s
Detected by: UAV 2 Target type: 3 (IED-V)
Remaining Time:
Others
(no classification
available)
Op 1
Op 2
All
ATR Information:
Possible Match:
Target Type 5:
Comm Equipment
12:56:40
Elapsed time:
Match Confidence:
Medium
0:00:20
ATR Decision:
Confirm ATR Suggested Classification:
Target Type 5 with Medium Confidence
Provide Alternate Target Classification:
Select Target Type…
Select Confidence Level…
Select Target Priority…
Veto Classification:
Threat distance
from the
15
convoy:
Request
new image
Image does not contain known target
miles
Cancel
Found by UAV2 – Location: 43:00E/24:00N
Elapsed Time:
Classification:
Operator 3
Image
Received at:
UAV
Reassignment
Target Type 5
High Confidence
mCDR
Detected by: UAV 3 Target type: 5 (Com Eq)
Operator 1
UAV 5
0: 02: 37
Req 3 Request from: Op 1 Elapsed time: 0m35s
(Complete)
UAV 4
Health Status
0:00:30
Help
Assistance Requests Overview
All
UAV 6
Fuel level: 70%
5
Health Status
0:00:05
Routes:
Submit
Classification
Tasking Display:
TargetID & rerouting,
reassigning UAVs
0: 03: 28
Improved Methodology for Deriving Collaborative
System Requirements
Project Goal:
• To develop techniques to identify dependencies in operator
decision making to understand how to assist coordination of
team member tasking
Future Work: Team Performance Prediction
• Behavioral pattern detection and performance predictions
-
Predictive algorithms (NN, HMM, Hybrid)
• Alert team supervising agent of sub-optimal cognitive
strategies
- Stress & time pressure
- Rapidly evolving situation
Is Operator
using “good”
strategies?
Operator
Excessive workload
Is the team
doing well?
Supervisor
Future Work: Team Performance Prediction
• Behavioral pattern detection and performance predictions
-
Predictive algorithms (NN, HMM, Hybrid)
• Alert team supervising agent of sub-optimal cognitive
strategies
- Stress & time pressure
- Rapidly evolving situation
Is Operator
using “good”
strategies?
Operator
Excessive workload
Is the team
doing well?
Team
Supervisor
Conclusions
• Initial activity awareness designs show potential for
facilitating supervisory-level decision making, and supporting
teamwork in general
- Particularly, designs aimed at conveying current & expected
status of interdependent activities
• Activity awareness approach (i.e. intelligent sharing of activity
data) appears to be an appropriate design technique for
large-screen displays in time-critical C2 environments
• Further work is needed to understand what activity
information should be shared (and at what level of detail)
during different UAV mission types and mission phases
Contact and Sponsor Information
Contacts:
- Dr. Stacey Scott, [email protected]
- Prof. Mary (Missy) Cummings, [email protected]
- Humans and Automation Lab (HAL) website:
http://halab.mit.edu
Thanks to project sponsor:
- Boeing Phantom Works