MODELING AND MEASURING
SITUATION AWARENESS IN
INDIVIDUALS AND TEAMS
Cleotilde Gonzalez
Dynamic Decision Making Lab
www.cmu.edu/ddmlab
Social and Decision Sciences Department
Carnegie Mellon University
In Collaboration with: Lelyn Saner, Octavio
Juarez, Mica Endsley, Cheryl Bolstad, Haydee
Cuevas, and Laura Strater
1
Agenda
• Computational Models of SA
– Individual aspects of SA
– Design aspects of SA
– Organizational aspects of SA
• Measures of SA
– Individual SA
– Shared SA
• Conclusions
2
Situation Awareness
• the Perception of the Elements in the Environment
within a Volume of Time and Space,
• the Comprehension of their Meaning, and
• the Projection of their Status in the Near Future.
• Formation of SA influenced by:
 Individual abilities
 Interactions with others
 Environment
Computational Cognitive Models
• Integrated theory of mind: ACT-R (Anderson & Lebiere,
1998)
– Shared attention (Juarez & Gonzalez, 2003, 2004)
– Learning theory (Gonzalez, Lerch & Lebiere, 2003; Gonzalez &
Lebiere, 2005)
– Representation of Recognition (Gonzalez & Quesada, 2003)
– Learning and decision making in dynamic systems (Gonzalez et
al., 2003; Martin, Gonzalez & Lebiere, 2004)
• Micro and Macro Cognition: Convergence and
Constraints Revealed in a Qualitative Model
Comparison (Lebiere, Gonzalez & Warwick, 2009)
4
Computational Cognitive Models
5
A SA meta-architecture provided a full set of cognitive models
interacting with OTB, and resulting in the “commander’s SA”
(Gonzalez et al., 2004; Juarez & Gonzalez, 2003)
6
Computational Models of Design Aspects of
SA (Juarez & Gonzalez, 2006)
7
Agenda
• Computational Models of SA
– Individual aspects of SA
– Design aspects of SA
– Organizational aspects of SA
• Measures of SA
– Individual SA
– Shared SA
• Conclusions
8
Individual Measures of SA: SAGAT
• Situation Awareness Global Assessment Technique
(SAGAT)
• Human-in-the-loop simulation exercises
• Use of SAGAT queries (from GDTA)
• Stop at random times and query the user
• Compare response with reality of the situation
– Examples: What is the aircraft altitude?
– What is the aircraft activity in this sector (en route, inbound to
airport, outbound to airport)
– Which aircraft will need a new clearance to achieve landing
requirements?
• SAGAT score: accuracy of the responses
Individual SA measures, learning and
working memory
• Can we learn to be aware? Effects of task
practice and working memory influence
situation awareness (SA) - Gonzalez & Wimisberg,
2007
• How do we measure individual SA
– Queries may be answered while the simulation
display is not visible or covered (Endsley, 1995)
or while the display is visible, uncovered (Durso
et al., 1995).
10
Methods
• The design was a 2 x 18 mixed design.
Participants were randomly assigned to one of
two conditions (covered or uncovered display)
and they were asked to run the simulation 18
times (trials).
• Individuals were asked to answer SA queries
while the simulation was paused
• Participants took the Visual Span Test (VSPAN)
(Shah & Miyake, 1996).
11
12
13
Results
Summary of results relevant for individual
measures of SA
• The correlation between SA scores and
VSPAN decreased over time
• SA scores were higher in the uncovered
condition than in the covered condition
– This is due mostly to perception
• The effect of practice was significant only
in the covered condition, but not in the
uncovered condition
18
Measures of Shared SA (Saner, Bolstad, Gonzalez & Cuevas, in
press; Saner, Bolstad, Gonzalez & Cuevas, in preparation)
Shared SA-the degree to which
team members possess the
same SA on shared SA
requirements (i.e. on the
information that they both
need to know)
(Endsley,1995, 1995b; Endsley & Jones,
2001)
Ground
TRUTH
Person 1
Person 2
X1
X2
X3
X4
X5
X6
X7
X1
X1
X2
X3
X4
X6
X7
X8
X9
X4
X6
X8
A good measure of shared SA needs to
account for the ACCURACY
Shared SA
Situation Awareness Global
Assessment Technique (SAGAT)
- Seven queries while task is stopped
- Objective knowledge of situation
Person 1
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Person 2
Q1
SimQ1
Q2
Q3
Q4
Q5
Q6
Q7
SimQ2
SimQ3
SimQ4
SimQ5
SimQ6
SimQ7
A good measure of shared SA needs to
account for the SIMILARITY
Score Similarity = 1-absolute value of
[(p1-p2)/(p1+p2)]
Range from 0 to 1
Method
• Training at Joint Personnel Recovery Agency (JPRA) JFCOM
• 16 servicemen, 3 DoD contractors; Age M=33.85
• Randomly assigned to one of four Teams:
– Navy, Army, Special Operations, or Joint Service
• Utilized Cross-Training
– Five scenarios over 3 days
– Each scenario had 3 to 12 incidents
– Scenarios randomly stopped 3 times for SAGAT,
Communication, and Workload measures
– Received training prior to the exercise
Methods and Procedure
•
•
•
Joint Personnel Recovery
Agency (JPRA) training
exercise
Four team groups (i.e. cells)
Five Predictors of Shared SA
– Experience Similarity- years
in real service
– Shared JPRA Knowledgeprior experience with
recovery operations
– Shared Cognitive Workloadsubjective ratings, five
NASA-TLX scales
– Communication Distanceinverse frequency of
communication
– Organizational Hub Distancedegree of dissociation from
Joint Service Cell
Special Operations Cell
(p13, p14, p15, p16,
p17)
Joint Service Cell
(p1, p2, p3, p4)
Army Cell
(p5, p6, p7,
Navy Cell
p8)
(p9, p10, p11, p12)
Possible Models
Classic Hierarchy
Expected
Results
Regression Models of True Shared SA
True
F
Adj.
Constant
R2
Shared SA
Exper ience
Shared
Workload
Organizational
Commu nication
Similarity
Knowledge
Similarity
Hub Distance
Distance
OVER ALL
5.11**
.21
-.03
.09
Scenario 1
2.56*
.09
.02
-.07
-.02
.18
-.26*
-.08
Scenario 2
1.55
.03
.39
.04
-.05
-.19
-.26*
.00
Scenario 3
1.66
.05
.10
.19
.09
.04
-.26*
-.02
Scenario 4
2.62*
.11
.26
.08
.31*
-.03
-.17
-.06
Scenario 5
5.79*
.24
.42
.11
*p < .0 5
**p < .0 1
.26*
-.19
.08
-.16
.50**
.45*
-.18
-.24*
Conclusions – Measures of Shared
SA
• Development of a Shared SA measure must
account for both, accuracy and similarity of SA
between members of an organization
• As shared knowledge increased, so did shared
SA.
• Organizational Hub Distance (OHD) is key
predictor
– Physical Distance and Joint Cell Membership
• Unexpected Role of OHD
– Participants processed new information directly
Possible Models
We observed that being in branch cells was associated with
higher SSA rather than being in the joint cell
Expected
Observed
Conclusions
• The success of Computational Models of SA,
depends on appropriate and robust measures of
individual and shared SA
– Although individual measures and procedures exist, there
is a huge need for defining the methods and procedures
for measuring SA at the team level
• We investigated measures of SA at both, the
individual and team levels
– We created a shared SA measure that builds on individual
SA
• Computational models of both, SA and SSA
can incorporate these measures.
27