573911
2015
EDMXXX10.1177/1555343415573911Journal of Cognitive Engineering and Decision MakingFinal Reflections
Special Issue
Final Reflections: Situation Awareness Models
and Measures
Mica R. Endsley, United States Air Force
A response is provided to the many authors who
commented on my initial paper for this special issue
on situation awareness (SA) addressing remaining questions and observations on the Endsley 1995 model of
SA. A discussion on historical perspectives on cognitive engineering and SA is included, along with future
research needs for the construct. New models on sensemaking, distributed SA, and situated SA are also discussed, with remaining comments on their differences
with the Endsley 1995 model of SA. Finally, a short
discussion of SA measurement approaches is provided,
addressing key issues raised in the commentaries.
Keywords: situation awareness, metrics, models, team
situation awareness
My initial intent in writing the introductory
paper to this special issue was to clear up some
misconceptions regarding my 1995 model of
situation awareness (SA) (Endsley, 1995c) that
have appeared in the recent literature (Endsley,
2015). Hopefully I was able to accomplish this
goal for those who are interested in what I have
intended to represent with the model.
As an unexpected benefit, the paper has
inspired a much broader discussion on SA from a
wide range of authors. Hoffman (2015) and Flach
(2015) each provided a fascinating historical perspective on much of the psychology literature
and movements going back across the last
century (and then some). This foundation provides an interesting lens with which to
Address correspondence to Mica R. Endsley, United States
Air Force, Pentagon, 4E130, Washington, DC 20330,
[email protected].
Author(s) Note: The author(s) of this article are U.S.
government employees and created the article within the
scope of their employment. As a work of the U.S. federal
government, the content of the article is in the public domain.
Journal of Cognitive Engineering and Decision Making
2015, Volume 9, Number 1, March 2015, pp. 101­–111
DOI: 10.1177/1555343415573911
consider the types of problems we are now contemplating in this modern and complex world,
and the models we are using to describe them. As
Flach points out, perhaps this history has created
a lens that has led some to misunderstand the SA
model as I intended it. In his characterization, the
psychology community has pulled away from
strictly linear “information processing models”
that were developed in laboratory environments
with artificial stimuli, towards a “triadic semiotic
frame” that considers the role of context and the
adaptive dynamics between the cognitive agent
and the environment. And because of that perhaps some people have misinterpreted the model,
although I believe my model to be quite consistent with such a situated view.
It is worth noting that in addition to the psychology history described in their articles, the
human factors community is multidisciplinary,
composed of many researchers whose training is
in engineering rather than primarily psychology.
As such, we arrive with a different lens. At the
foundation of the industrial engineering discipline is work by Frederick Taylor (from roughly
1870 to 1910) analyzing the work processes of
machinists. This approach was extended by
Frank and Lillian Gilbreth in the early 1900s,
who perfected the science of work analysis in
studying brick layers and laborers in manufacturing and clerical contexts. The experiences of
the Tavistock institute with coal miners during
World War II expanded this work into sociotechnical systems, recognizing the interaction
between people, technology, and organizational
structures and processes. (Although the Tavistock researchers were psychologists, this work
was embraced more by business and industrial
engineers involved in organizational and work
design than by experimental psychologists who
then trended towards behaviorism.)
Although the majority of this research foundation was concerned with physical tasks and
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
102March 2015 - Journal of Cognitive Engineering and Decision Making
behaviors (which characterized much of the
prevailing work environments of the day), rather
than cognitive tasks, it creates an entirely different approach to the study of human behavior
than that which Flach, Hoffman, and others rail
against. That is, we come from a perspective that
has always considered work in context and has
always studied primarily experts performing
their tasks within their real-world constraints,
mainly for the purpose of improving that work
from the standpoint of efficiency or safety. What
the field lacked was a way of incorporating cognitive work within that framework, for which we
came to the field of psychology. To most people
outside of psychology, the idea that behavior
should be studied in context is taken as a given,
leading to some perplexity at the idea that “situated cognition” or “naturalistic” studies are new.
So when I proposed a model to explain this
phenomenon of SA in my work with pilots and
the design of advanced fighter aircraft, I confess
that I took for granted that the construct occurred
within an ecologically valid environment with
expert pilots, within a dynamic, ongoing process
over long periods of time. Although others were
running around at that time complaining that “we
didn’t know what SA was,” I argued we had
much we could say about it, based on the rich
history of research on various cognitive mechanisms and constructs that made up my model.
Although I fully understand rejecting an isolated,
laboratory-based, stimulus-response model of
cognition, I chose not to throw out the baby with
the bath water, but rather to build something
more substantial from past work on cognition
and information processing—extending it to deal
with complex, cognitive work in the real world
of aviation where my research was based. The
point of the model was to provide a more detailed
understanding of SA via the interactions of these
cognitive mechanisms, the experiences, preconceptions and goals of the individuals, and the
characteristics of their tools, tasks, and environments, extending far beyond strictly data-driven
linear information processing models in many
significant ways that have already been elaborated. Although undoubtedly not complete in
representing all the details of these various factors, it is my belief that it provides a good starting
point for that discussion, subsequent research,
and our ultimate job of improving the tools and
workplaces we design for people.
Thus, the Endsley 1995 SA Model reflects significant detail on the cognitive mechanisms and
processes that are at play in creating the ongoing
mental representation of SA, and also the ways in
which characteristics of the tools, environment,
and systems affect those processes. Although I
certainly agree that our research and our models
need to reflect the ways in which people interact
with and are affected by the complex and dynamic
worlds in which they operate, I also believe we
need to realize that human cognition is also a relevant part of that ecological sociotechnical system and not one that can get easily dismissed or
ignored by only focusing on context.
In terms of historical background, Klein
(2015) points out that he discussed the importance of SA in his 1986 paper (Klein, Calderwood, & Clinton-Cirocco, 1986) on recognition
primed decision making (RPD) in fire fighters. I
referenced his RPD model in the Endsley 1995
SA model in describing the link between SA and
decision making within that context. However,
my formulation of decision making was primarily influenced by the seminal work of Stuart
Dreyfus (1981), who examined how experts
made decisions in a number of domains including business management, chess, and science.
He found strong evidence that people do not
typically go through a detailed analytic comparison of alternatives but rather recognize known
situational patterns that they match to memory
to arrive at what to do. Klein et al.’s 1986 study
on fire fighters primarily confirmed this finding
in a new domain. The more detailed model of
RPD that Klein and his colleagues later created
(Klein, 1993) goes much farther in describing
this phenomenon than the original Dreyfus
paper. However, neither author provided any
detail on what SA is or how people form SA
itself, which was the purpose of the 1995 SA
model and which I think is highly compatible
with and complementary to these works.
I endeavored to provide a comprehensive
account of a quite complex construct, one that
rests on many other cognitive mechanisms and
behaviors, and that is heavily embedded in the
challenges and messiness of complex environments. Although Klein (2015) argues that I may
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
Final Reflections
103
have changed or morphed my model over time,
the quotes provided from the original 1995
paper, in some cases augmented by quotes from
a 2000 paper that was written to further explain
parts of the model, show that these fundamental
aspects of the model have remained quite consistent, and significantly predate the articles with
which I took issue. Far from “resisting change”
as leveled by Stanton, Salmon, and Walker
(2015), I welcome research that is directed at
extending our knowledge of SA and Team SA.
Although there is plenty of room in academic
discussions to jostle over aspects of theories to
try to determine what may more accurately represent the phenomenon under discussion, it is my
belief that such discussions are most fruitful
when they discuss the real differences between
such models, rather than a (perhaps unintentional) misrepresentation of them. Thus, I can
only hope that these misunderstandings have
been rectified by this effort at clarification.
Although van Winsen and Dekker (2015)
remain strangely obtuse in some paradigm in
which boxes and arrows are somehow bad (but
circles and arrows are not?), I believe this is ultimately an artificial difference. Ecological design
and Neisser’s model actually represent many of
the same features as my model (no matter how
they are arranged on the page) (see Endsley,
2000). As Minotra and Burns (2015) and Hancock and Diaz (2002) point out, ecological
design and the SA model actually have much in
common (see also Endsley, 2004)). Van Winsen
and Dekker’s more extreme position seems to be
that SA should not be studied because it distracts
people from focusing on the task environment
(the situation). However, this is ultimately not a
terribly useful position as it provides no tools for
effectively designing that environment for the
cognitive work we are trying to aid.
Finally, as Flach (2015) points out, representations matter, so I will further address the final
questions raised on the model here. The arrows
in the Endsley 1995 model from the Task/System Factors and Individual Factors show influence on the process. Although I do show SA,
decision, and action as separate stages, they
occur in a loop acting on the environment, with
the changing state of the environment in turn
effecting those cognitive stages, often quite
rapidly. That the levels of SA were nested, and
not shown in a line with arrows, is because they
are inter-related levels and not linear stages. The
model provides the details on how the internal
and external factors all interact in affecting those
processes over time to create an ever-evolving
state of SA.
Future Research
This issue has also generated some highly
useful discussion on future research directions.
Wickens (2015) cites the need for more research
on projection (Level 3 SA) with which I heartily agree. He also discusses the interesting roles
of long-term memory (LTM), working memory (WM), and long-term working memory
(LTWM) and how they relate to SA. In that
my model (and subsequent research findings)
supports an integrated model of LTM and WM
(per Cowan, 1988) and Durso and Gronlund
(1999) also calls out an integrated model (per
Ericsson & Kintsch, 1995), it would seem that
more research would be warranted here as well,
perhaps addressing the concerns of Chiappe,
Strybel, and Vu (2015) as people move from
being novices to experts.
Vidulich and Tsang (2015) discuss the need
to consider both SA and workload when developing potential operator decision aids, which I
certainly agree with. Minotra and Burns (2015)
call out the need for more research on metacognitive issues of self-regulation and self-awareness,
SA in unstructured environments, and the differences between incorrect SA and low SA (meaning being wrong versus being uninformed).
Such meta-awareness is certainly important. For
example, not only the level of SA but also the
degree of confidence in that SA can affect how
people choose to act (e.g., conservatively or
boldly) with negative consequences if these are
misaligned (Endsley & Jones, 2012). I also echo
their support for Klein’s work on macrocognition that has sought to discover more detail on
issues such as problem detection, uncertainty
and risk management, and coordination in
teams. Although such research may not be considered as specific to SA, our research has certainly detailed how they are often integrally
related with SA, and how in many cases their
representation is a component of SA (Endsley &
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
104March 2015 - Journal of Cognitive Engineering and Decision Making
Jones, 2012). I am certain that far more can be
learned about these processes that would be of
import in understanding both SA and human
cognition in complex settings.
Equally importantly, we find that human factors work on SA has had a positive influence on
the design of systems and development of training programs (as outlined by Jones, 2015), and it
is an integral part of the way in which pilots (and
many others) think about their worlds, as discussed by Byrne (2015). As Byrne points out,
operators do not care about the theoretical arguments of academics but rather what we can do to
help them. To me the most beneficial outcome of
the past 25 years of research has been the ability
to contribute to systems designs in aviation, air
traffic control, military command and control,
intelligence operations, power grid operations,
transportation, mining, oil exploration, and myriad other complex operations. That work, by
myself and many, many others, I believe was
greatly aided by the fundamental understanding
provided by the model.
I certainly disagree with van Winsen and
Dekker (2015), who seem to believe that the
work of the human factors community to understand SA has somehow led the operational community astray with this crazy construct. On the
contrary, pilots invented SA—it was a part of
their vocabulary and conceptualization of their
world long before any of us got involved in trying to describe it, measure it, or design for it.
They would be talking about it (and lamenting
not having it), whether we were doing anything
to help them with it or not (see Byrne, 2015).
Van Winsen and Dekker’s further claim that
somehow SA researchers are to blame for the
conviction of Captain Lilgert in the sinking of a
passenger ferry in British Columbia, leading to
the death of two passengers, also does not hold
water. Capt Lilgert’s conviction was based on a
finding of gross negligence, stating that he failed
to properly operate and navigate the ferry in dereliction of duty, including failing to slow during
heavy weather, failing to have assistance on
deck, failing to follow standard practices and
regulations, failure to communicate with other
vessels, and being distracted from his duties on
deck due to a personal conversation with a shipmate with whom he had a romantic relationship
(Supreme Court of British Columbia, 2013).
Although we could probably engage in an analysis of the quality of the on-board systems and
whether there were extenuating circumstances
that contributed to the accident (although the
Judge did not find any), that the Judge stated that
“situational awareness … is key to proper navigation” within the 16-page finding is irrelevant
to the facts of his conviction. Van Winsen and
Dekker’s claim is spurious at best.
Other Models
Although the key intent of my paper was to
correct misunderstandings of the Endsley 1995
model, I also reviewed several newer models presented by their authors as a way of compensating
for the supposed weakness of the model they
criticized. Each of those authors had the appropriate opportunity to respond to my assessments
of their model, to which I’ll offer a final response.
Sensemaking
Klein (2015) continues to argue that the data/
frame model of sensemaking is different than
the Endsley 1995 SA model in that it “posits
two parallel and simultaneous activities, using
frames to define what counts as data, and using
data to select, modify and construct frames.” As
I’ve shown earlier, however, this is not different than the way in which schema and mental
models define what counts as relevant data (and
directs attention accordingly), or the way in
which data detected initially will trigger active
schema for further data gathering and interpretation (with anchoring and confirmation biases
potentially coming into play), as depicted in the
1995 SA model. The 1995 SA model also provides a detailed discussion of how these mental
models are created, modified, and selected
between. And contrary to Klein’s assertion, “the
formation of what counts as data in the first
place” is explicitly called out in the model:
•• “People are active participants in determining
which elements of the environment will become a
part of their (Level 1) SA by directing their attention based on goals and objectives and on the basis
of long-term and working memory” (Endsley,
1995c, p .41).
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
Final Reflections
105
•• “Advanced knowledge of the characteristics,
form, and location of information, for instance,
can significantly facilitate the perception of information” (Endsley, 1995c, p. 42).
•• “Working memory also plays an important role,
allowing one to modify attention deployment on
the basis of other information perceived or active
goals” (Endsley, 1995c, p. 42).
•• “A well-developed mental model provides (a)
knowledge of the relevant elements of the system
that can be used in directing attention and classifying information in the perception process …”
(Endsley, 1995c, pp. 43-44).
•• “For example, a pilot may perceive several aircraft
(considered to be important elements per the mental model) recognized as enemy fighter jets (based
on critical cues) that are approaching in a particular spatial arrangement (forming Level 1 SA)”
(Endsley, 1995c, p. 44).
however, find that the sensemaking model has
provided any new extension or explanation of
this construct. “Reframing” the same processes
that have previously been described in the SA
model in a different terminology only contributes to the potential confusion of readers that
these are somehow different processes. Klein’s
statement that I differentiate the state of SA from
the processes involved in achieving SA does not
amount to a justification for his claim that the
Endsley 1995 SA model does not address processes, which it most certainly does and quite
extensively. I certainly welcome Klein’s elaboration in this area of human cognition, as he
always has excellent insights to offer. My comments on the sensemaking model are to show
where the two models are similar and where
they are (or are not) different to improve the
clarity of discussion on this topic.
I believe this addresses what Klein refers to as
“determining what counts as data in the first
place,” or “constructing data.” Recognizing that
some cue is relevant and important occurs on
the basis of the mental model. Without one, this
would be quite difficult.
Klein also states that the data/frame model
was only intended to cover deliberate cases and
not the more automatic cases also covered by SA
theory, which is quite true. Nor is there a requirement to from his perspective. However, the fact
that it does not, while the 1995 SA model does,
leaves it at a disadvantage I believe, in that so
much of SA (or sensemaking) in real-world settings occurs with a combination of both processes in play. In fact, I find that a key problem
is whether someone realizes that the situation
they are in is novel and does not readily match to
existing or active schema. This needs to prompt
them into a more deliberate, thoughtful process
whereby they use more fundamental knowledge
from their mental models to puzzle out alternate
explanations of the information they have, or to
collect additional data to explain some discrepancy. The segue from one form of processing to
the other is critical and can be error prone, as
Klein knows from his work on RPD, and is in
need of further research attention.
I certainly respect Klein’s extensive work in
RPD and macrocognition. I do not, in this case,
Distributed SA (DSA)
I have already discussed the numerous inaccuracies that Stanton et al. (2015) have used to
bolster their claims for why a new model of SA
is needed, and will not further detail those here.
Also I have shown that DSA provides far less
explanation of SA in teams than other previous models and research on team and shared
SA (Endsley & Jones, 2001; Endsley & Robertson, 2000; Gorman, Cooke, Pederson, Connor, & DeJoode, 2005; Prince & Salas, 1993;
Prince, Salas, & Stout, 1995). In reviewing their
response, it primarily focuses on issues of SA
measurement in teams, specifically on the use
of the Situation Awareness Global Assessment
Technique (SAGAT), which I will address in
more detail below.
Before doing so, however, I would like to
address their contention that the only (and best)
approach is to examine observable actions and
communications between team members and
their tools due to the fact that “given the complexity of the socio-technical systems which
form the subject of much contemporary analysis, the study of information processing in the
minds of individuals has lost relevance” and
“one cannot ever know completely what is going
on in people’s minds” (Stanton et al., 2015).
This viewpoint strikes me as moving into a very
Skinneristic model, one that has long since been
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
106March 2015 - Journal of Cognitive Engineering and Decision Making
rejected in psychology, and against which significant progress has been made.
Although there is certainly information to be
gained from analyzing team processes, communications, and behaviors (and which has been
done for a long time by many researchers in the
SA area), there are also distinct limitations in
relying solely on such observations to understand
the SA of individuals or teams. Specifically, such
measures do not provide information on why an
individual may not have attended to certain information, and what assessments they make from
the information they do attend to in order to form
comprehensions and projections (unless they talk
about it, which is highly variable across individuals and situations). A key issue about SA is that it
deals with how people understand and form
meaning from the information they observe, and
this is highly lacking in such observational data
(Endsley, 1996; Endsley & Jones, 2012). So
although process observations and communications analysis can certainly be useful in analyzing the team processes underlying SA, by themselves they are limited for this application.
Situated SA
Chiappe et al.’s (2015) discussion of situated
SA is in many respects not as divergent from
the Endsley 1995 SA Model as it might initially
appear. Many of their arguments that reject “SA
in the head,” however, are unfortunately weak.
Change blindness, for example, can be explained
as an attentional failure that is present regardless
of which cognitive model of SA one uses. Their
argument that people do not hold a veridical visual
representation in memory is also entirely consistent with the Endsley 1995 SA Model, which only
holds that data needed by the active mental model
are retained. That is, people are only retaining
that data from the environment that is needed for
the safe operation of their system per their mental
model (e.g., to manage the traffic in their sector
as an air traffic controller or to fly the aircraft
as a pilot). The use of “external scaffolds” to aid
memory and assist in task performance is also
not a point of disagreement between us. These
“strings around the finger” are well ensconced
in many environments, including the tipping of
flight strips by air traffic controllers and the use of
speed bugs by pilots, which they mention.
Chiappe et al.’s research examples showing
strong WM dependence are unfortunately with
students, not experts. Novices have consistently
been found to have very low levels of SA (Endsley, 2006) and are believed to be WM limited in
the Endsley 1995 SA Model. They do not yet have
the mental models and schema that would allow
them to retain or process data effectively in these
complex domains and must therefore rely heavily
on external data sources, and ultimately have very
low SA (and the performance to match it). Other
examples are taken from nonoperational tasks—
for example, looking information up on a computer or retaining peripheral information such as
phone numbers or aircraft callsigns. I cannot
remember phone numbers either and am happy to
write them down. But I still need to know how fast
I am driving to avoid a ticket. These are very different things. The operational tasks where SA is
critical (flight, driving, military operations, power
grid operations, …) often involve significant time
constraints for decision making (no time to look
up the answer when you are trying to land an airplane or avoid getting shot in battle). The need for
a working mental representation of the situation is
real-time and demanding.
There is no disagreement that people will not
load up any more information in memory than
they need to. The real question is, How much do
they need to? And which information? And when?
In our work with expert air traffic controllers, for
example, I have found that they definitely do not
retain callsigns of aircraft (Endsley & Rodgers,
1996). I will call this peripheral data. Instead their
internal model is primarily a temporal-spatial one.
They know a lot about aircraft (e.g., location, type,
flight path, flight level, ascending/descending,
which aircraft are traffic for which other aircraft),
but their central mental representation anchoring
this information is a temporal-spatial one (which
is why we do not do SAGAT queries based on
aircraft callsigns, but rather referenced to a map
of the controller’s sector).
The active working situation model that controllers use to understand the flight situation and
project potential conflicts is composed of the
information that is necessary for decisionmaking for that task. Although they have certain
forms of automation (e.g., alerts that occur if
two aircraft are within 5 miles of each other),
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
Final Reflections
107
that system is only a back-up. Instead they
actively project potential conflicts and generally
move aircraft far, far in advance of the prescribed limits. This is an active mental process.
As different aircraft need clearances to ascend or
descend for various reasons, they must rapidly
assess how that change will play out over time to
make all the necessary changes across the many
aircraft under their purview. Controllers do
annotate flight progress strips with certain data
about the aircraft, which they are required to, as
discussed by Chiappe et al., however they consider this a “house-keeping task” and often do it
far after the fact in periods of low workload. And
pilots will tell you that setting the speed bugs in
no way relieves them from needing to know aircraft airspeed, which is critical to their flight. In
fact, we find pilots can report airspeed accurately in excess of 80% and generally have very
good knowledge of their flaps and slats settings.
Aids can certainly assist SA, but they do not
substitute for it.
When Chiappe et al. state that what is needed
is a “minimal internal representation,” I don’t
disagree. The key is in defining what that minimum representation is for these complex tasks.
As discussed above, it involves a lot of detailed
understanding of the changing physical dynamics of systems and environments that are needed
in real-time for operational performance. Irrelevant information (your teammate’s shoe size) is
not needed, nor is peripheral information (e.g.,
callsigns and phone numbers), which I will
define as data that are a part of the task, but not
central to decision making, and which are looked
up as needed. But the relatively poor performance of novice and student pilots, air traffic
controllers, and soldiers can be directly linked to
a poor ability to derive this minimal internal representation of critical situational information in
real-time dynamic worlds (Endsley, 2006). We
have addressed the question of what is needed
for SA (i.e., that information that is needed for
decision making) with detailed Goal Directed
Task Analyses for each domain and role. These
derive the goal-relevant, decision-needed information that is important for SA and that forms a
basis for measurement, design, and training
interventions (Endsley, 1993b; Endsley & Jones,
2012; Endsley & Rodgers, 1994).
This internal mental representation is also
important for creating comprehension and projection. It’s hard to keep those accurate if one
does not access the dynamically changing situation information and integrate it with other data
that generally must occur mentally. In early
work on SA, I wondered if the low-level data
were jettisoned once the comprehension and
projections were formed. We found that, contrary to that hypothesis, Level 1 SA data (e.g.,
airspeed, altitude, etc.) were reportable via
SAGAT (showing they were held mentally) and
are retained as far as 5 to 6 minutes after a
SAGAT freeze with no loss of accuracy (Endsley, 1990, 1995a). These data appear to be as
much of the mental situational model as any
derived assessments of comprehension and projection are. What is important to remember here
is that people in these environments are not just
looking at and memorizing information. Rather,
they are constantly updating a mental representation, setting up detailed scan patterns in many
cases, to make sure they are gathering the needed
data and keeping the model fresh to support
active decision tasks. Artificial tasks with student subjects are very unlikely to duplicate such
processes.
The other central question that underlies Chiappe et al.’s discussion is, does what is “out
there” count as SA? In most cases, we find that
people must actively manipulate their environments (e.g., displays that are in view, radar settings and directions, communications with others) to gather their SA. They are not passive
recipients of data from the displays in front of
them but are rather active participants in gathering the information they need (Endsley, 1995c).
Failures in this information-gathering process,
whether due to training, task loading, equipment
limitations, or poor display design, lead to lower
SA. And although transactive memory is certainly useful in distributed team settings, ultimately people still need to know the information
needed for their own job (Endsley, 1995c). It is
not sufficient that the copilot knows that the
plane is below the required minimum safe altitude if the pilot flying the plane does not.
For this reason, I ultimately believe we cannot assess SA just from what is “out there” but
rather at the point that people have successfully
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
108March 2015 - Journal of Cognitive Engineering and Decision Making
accessed it to update their internal representation on which their ultimate performance will
depend. Chiappe et al. finally concede that this
must occur but want to “give credit” for being
able to look up information when needed. I think
fundamentally there is no easy way to assess
how good those processes, behaviors, and system designs are for aiding SA without taking
into account whether they have completed the
process of updating their mental representation
with the needed information.
SA Measurement
Much of the disagreement on SA models that
has been presented ultimately has boiled down
to a disagreement on the best way to measure
SA, which is actually somewhat peripheral
to the theoretical model discussions. A more
detailed comparison of the substantial literature
base on SA measures must be reserved for a
separate discussion; however, I will address
some of the comments that have been made in
this special issue.
Stanton et al. (2015) report that they found
SAGAT created unexpected results in a study of
command and control decision aids. It should be
noted, however, that they referred to a study that
actually examined an artificial task based on the
game of chess in which they tried to have distributed teams of nonexperts do mission planning. However, it is widely known that chess
novices have very poor mental representations
of the game, whereas chess experts have much
more detailed mental representations (Chase &
Simon, 1973). I find their counterintuitive results
probably have far more to do with the artificial
tasks these nonexperts were asked to perform
than with the measurement technique itself. We
have used SAGAT quite successfully, on the
other hand, with expert soldiers performing
actual mission planning and command and control tasks in numerous studies (Bolstad & Endsley, 2003; Strater, Faulkner, Hyatt, & Endsley,
2006).
Other concerns were directed at the SAGAT
technique in general, which uses short freezes in
a simulation to provide queries about SA, questioning whether it relies too heavily on memory
(Chiappe, Rorie, Moran, & Vu, 2012; Durso
et al., 1998; Salmon, Stanton, & Young, 2011;
Sarter & Woods, 1991), suggesting that the Situation Present Assessment Method (SPAM),
which asks questions in real time (Durso et al.,
1998), may be a better method (Chiappe et al.,
2015; Vidulich & Tsang, 2015). Although we
can debate various aspects of these approaches,
the fact is there is quite a bit of data that help to
address these questions.
SAGAT was designed for use in simulation
settings. SPAM has the advantage of being
administrable in many real-world settings where
freezing is not possible. However, it generally
allows for far fewer queries to be provided
within the same test. During a SAGAT freeze,
many more questions can be asked and answered,
as compared to SPAM, which can only ask one
question at a time, as it must occur concurrently
with task performance. This leads to significantly lower sensitivity to SA differences for
SPAM as compared to SAGAT, as has been
demonstrated in numerous studies (Endsley,
Sollenberger, & Stein, 2000; Jones & Endsley,
2004; Pierce, 2012).
SPAM and SAGAT differ in two main
respects. SAGAT pauses the simulation and
blanks the displays while queries are administered, whereas SPAM leaves them present and
does not pause the simulation. One of the most
detailed comparisons of these two features of the
techniques was conducted by Morgan, Chiappe,
Kraut, Strybel, and Vu (2012), who did a detailed
comparison of Visible/Not Visible and Paused/
Not Paused in an ATC simulation with student
subjects. Not surprisingly, SA queries were more
accurate (by 17%) when the information
remained visible. Reaction time (RT) was slower
when the information remained visible, however,
by almost 2 s, revealing a significant speed/
accuracy tradeoff. In general, people will look to
check before answering queries if the information is available. This is consistent with Jones
and Endsley (2004), who also found a speed/
accuracy tradeoff for SPAM real-time probe
measures. This strategy works for questions of a
specific nature (e.g., Will any conflicts be present
for AAL123?), but not as well for their more general questions (e.g., Will any conflicts be present
if no action is taken?), which were answered 1
second faster, also pointing toward the necessity
of internal memory representations.
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
Final Reflections
109
The concern of whether SAGAT is too memory dependent seems to be largely unsupported.
When working with experts on realistic tasks,
they have access to mental models and schema
that support detailed internal representations of
key information needed for task performance
(Endsley, 1990, 1995a). Although people will
look up data if available (as shown with SPAM),
there is no indication that this provides a more
valid or reliable assessment of SA than does
SAGAT.
In fact, SPAM may interfere with concurrent
task performance due to the need to multitask.
Although numerous studies have not found any
interference effect of SAGAT on task performance (Bolstad & Endsley, 1990; Endsley,
1995a, 1995b; Hogg, Torralba, & Volden, 1993),
Pierce (2012) found that participants correctly
handled fewer aircraft during the SPAM trials
(approximately 6%), as compared to those without, and they interacted with aircraft significantly less frequently in the time period following a probe. Other studies have found an indication of higher workload in trials involving SPAM
(Endsley et al., 2000; Jones & Endsley, 2004;
Pierce, 2012).
In summary, claims of advantages of SPAM as
compared to SAGAT do not appear to be warranted by the available research. Due to the freeze,
SAGAT provides a much richer and more comprehensive assessment of SA by virtue of being
able to ask more queries than SPAM in a similar
length of time, thus providing greater reliability
and sensitivity. And there is no indication that
SPAM overcomes any memory reliance associated with SAGAT due to the availability of LTWM
by experts. Although people will often look up
information if available, the speed-accuracy tradeoffs associated with this do not result in any superior assessment of their SA. However, in situations
where freezing the action is not possible, SPAM
may provide a useful approach. Although it may
pose some intrusiveness to concurrent task performance, this may be warranted in order to obtain
useful data on SA in many cases.
Some measures combine aspects of workload
and SA into one metric, for example the Situation Awareness Rating Technique (SART)
(Taylor, 1990) and SPAM (Durso et al., 1998),
however, given the often independent nature of
these two constructs in less than overload situations (Endsley, 1993a; Vidulich & Tsang, 2015),
I feel it is important to keep these two separate.
It is quite easy to measure SA and workload concurrently, using specific measures for each,
which we have done in many of our studies.
Conclusions
Overall, I found this Special Issue to provide
a useful discussion of SA theory and measurement. I hope that it serves as a useful course
correction for those doing research in the field
and laid to rest any existing misunderstandings
about the Endsley 1995 SA Model. I am particularly thankful to the many researchers who
participated in this Special Issue and the many
kind things they had to say about the Endsley
1995 SA model. I am also indebted to those who
have offered their criticisms, as it has provided
the opportunity for a useful discussion.
A substantial body of research has been generated over the past 25 years on SA. Much of
this has confirmed or expanded upon the Endsley 1995 SA Model, significantly growing our
understanding of this important construct. More
importantly, that model has been used to drive a
wide body of work to develop extensive guidelines for improved training and system designs
to support SA (Endsley & Jones, 2012) that
have been used to create significantly improved
workstations in a wide range of aviation, command and control, military, intelligence, power
systems, healthcare, and the oil and gas industries. In my dealings with operators, pilots, physicians, controllers, and soldiers over this time
period, I have been overwhelmed with their
strong stated desire for systems to help support
their need for SA and their incredibly positive
response to the work we are doing in this field.
The corpus of theoretical and experimental
work on SA, and its associated metrics, provides a strong foundation for continuing to
address the real needs of people trying to
accomplish difficult jobs in complex and
demanding settings.
References
Bolstad, C. A., & Endsley, M. R. (1990). Single versus dual scale
range display investigation (NOR DOC 90-90). Hawthorne,
CA: Northrop Corporation.
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
110March 2015 - Journal of Cognitive Engineering and Decision Making
Bolstad, C. A., & Endsley, M. R. (2003). Measuring shared and
team situation awareness in the army’s future objective force.
Paper presented at the Human Factors and Ergonomics Society
47th Annual Meeting, Santa Monica, CA.
Byrne, E. (2015). Commentary on Endsley’s “situation awareness
misconceptions and misunderstandings.” Journal of Cognitive
Engineering and Decision Making.
Chase, W. G., & Simon, H. A. (1973). Perceptions in chess. Cognitive Psychology, 4, 55–81.
Chiappe, D. L., Rorie, R. C., Moran, C. A., & Vu, K. P. L. (2012). A
situated approach to the acquisition of shared SA in team contexts. Theoretical Issues in Ergonomics Science, iFirst, 1–19.
Chiappe, D. L., Strybel, T. Z., & Vu, K. P. L. (2015). A situated
approach to the understanding of dynamic situations. Journal
of Cognitive Engineering and Decision Making, 9(1), 33–43.
Cowan, N. (1988). Evolving conceptions of memory storage,
selective attention, and their mutual constraints within the
human information processing system. Psychological Bulletin,
104(2), 163–191.
Dreyfus, S. E. (1981). Formal models vs. human situational understanding: Inherent limitations on the modeling of business
expertise (ORC 81-3). Berkeley: Operations Research Center,
University of California.
Durso, F. T., & Gronlund, S. D. (1999). Situation awareness. In
F. T. Durso, R. Nickerson, R. Schvaneveldt, S. Dumais,
S. Lindsay, & M. Chi (Eds.), Handbook of applied cognition
(pp. 284–314). New York: Wiley.
Durso, F. T., Hackworth, C. A., Truitt, T. R., Crutchfield, J.,
Nikolic, D., & Manning, C. A. (1998). Situation awareness as a
predictor of performance for en route air traffic controllers. Air
Traffic Control Quarterly, 6(1), 1–20.
Endsley, M. R. (1990). A methodology for the objective measurement of situation awareness. In Situational awareness in aerospace operations (AGARD-CP-478) (pp. 1/1–1/9). Neuilly Sur
Seine, France: NATO-AGARD.
Endsley, M. R. (1993a). Situation awareness and workload: Flip
sides of the same coin. Paper presented at the Seventh International Symposium on Aviation Psychology, Columbus, OH.
Endsley, M. R. (1993b). A survey of situation awareness requirements in air-to-air combat fighters. International Journal of
Aviation Psychology, 3(2), 157–168.
Endsley, M. R. (1995a). Direct measurement of situation awareness in simulations of dynamic systems: Validity and use of
SAGAT. In D. J. Garland & M. R. Endsley (Eds.), Experimental analysis and measurement of situation awareness (pp.
107–113). Daytona Beach, FL: Embry-Riddle University.
Endsley, M. R. (1995b). Measurement of situation awareness in
dynamic systems. Human Factors, 37(1), 65–84.
Endsley, M. R. (1995c). Toward a theory of situation awareness in
dynamic systems. Human Factors, 37(1), 32–64.
Endsley, M. R. (1996). Situation awareness measurement in test
and evaluation. In T. G. O’Brien & S. G. Charlton (Eds.),
Handbook of human factors testing & evaluation (pp. 159–
180). Mahwah, NJ: Lawrence Erlbaum.
Endsley, M. R. (2000). Theoretical underpinnings of situation
awareness: A critical review. In M. R. Endsley & D. J. Garland
(Eds.), Situation awareness analysis and measurement (pp.
3–32). Mahwah, NJ: LEA.
Endsley, M. R. (2004). Situation awareness: Progress and
directions. In S. Banbury & S. Tremblay (Eds.), A cognitive approach to situation awareness: Theory, measurement
and application (pp. 317–341). Aldershot, UK: Ashgate
Publishing.
Endsley, M. R. (2006). Expertise and situation awareness. In K. A.
Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.),
The Cambridge handbook of expertise and expert performance
(pp. 633–651). New York: Cambridge University Press.
Endsley, M. R. (2015). Situation awareness misconceptions and
misunderstandings. Journal of Cognitive Engineering and
Decision Making, 9(1), 4–32.
Endsley, M. R., & Jones, D. G. (2012). Designing for situation
awareness: An approach to human-centered design (2nd ed.).
London: Taylor & Francis.
Endsley, M. R., & Jones, W. M. (2001). A model of inter- and
intrateam situation awareness: Implications for design, training and measurement. In M. McNeese, E. Salas, & M. Endsley
(Eds.), New trends in cooperative activities: Understanding
system dynamics in complex environments (pp. 46–67). Santa
Monica, CA: Human Factors and Ergonomics Society.
Endsley, M. R., & Robertson, M. M. (2000). Situation awareness in
aircraft maintenance teams. International Journal of Industrial
Ergonomics, 26, 301–325.
Endsley, M. R., & Rodgers, M. D. (1994). Situation awareness
information requirements for en route air traffic control (DOT/
FAA/AM-94/27). Washington, DC: Federal Aviation Administration Office of Aviation Medicine.
Endsley, M. R., & Rodgers, M. D. (1996, September). Attention
distribution and situation awareness in air traffic control.
Paper presented at the 40th Annual Meeting of the Human Factors and Ergonomics Society, Santa Monica, CA.
Endsley, M. R., Sollenberger, R., & Stein, E. (2000). Situation
awareness: A comparison of measures. Paper presented at the
Human Performance, Situation Awareness and Automation:
User-Centered Design for the New Millennium, Savannah,
GA.
Ericsson, K. A., & Kintsch, W. (1995). Long term working memory. Psychological Review, 102, 211–245.
Flach, J. (2015). Situation awareness: Context matters! Journal of
Cognitive Engineering and Decision Making, 9(1), 59–72.
Gorman, J. C., Cooke, N. J., Pederson, H. K., Connor, O. O., &
DeJoode, J. A. (2005). Coordinated awareness of situation by
teams (CAST): Measuring team situation awareness of a communications glitch. Paper presented at the Human Factors and
Ergonomics Society 49th Annual Meeting, Santa Monica, CA.
Hancock, P. A., & Diaz, D. D. (2002). Ergonomics as a foundation
for a science of purpose. Theoretical Issues in Ergonomics Science, 3(2), 115–123.
Hoffman, R. R. (2015). Origins of situation awareness: Cautionary
tales from the history of concepts of attention. Journal of Cognitive Engineering and Decision Making, 9(1), 73–83.
Hogg, D. N., Torralba, B., & Volden, F. S. (1993). A situation
awareness methodology for the evaluation of process control
systems: Studies of feasibility and the implication of use (199303-05). Storefjell, Norway: OECD Halden Reactor Project.
Jones, D. G. (2015) A practical perspective on the utility of situation awareness. Journal of Cognitive Engineering and Decision Making, 9(1), 98–100.
Jones, D. G., & Endsley, M. R. (2004). Using real time probes for
measuring situation awareness. International Journal of Aviation Psychology, 14(4), 343–367.
Klein, G. A. (1993). A recognition primed decision (RPD) model
of rapid decision making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action:
Models and methods (pp. 138–147). Norwood, NJ: Ablex.
Klein, G. A. (2015). Whose fallacies? Journal of Cognitive Engineering and Decision Making, 9(1), 55–58.
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015
Final Reflections
111
Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid
decision making on the fire ground. Paper presented at the
Human Factors Society 30th Annual Meeting, Santa Monica,
CA.
Minotra, D., & Burns, C. M. (2015). Finding common ground:
Situation awareness and cognitive work analysis. Journal of
Cognitive Engineering and Decision Making, 9(1), 87–89.
Morgan, C. A., Chiappe, D. L., Kraut, J., Strybel, T. Z., & Vu, K. P. L.
(2012). An investigation into the potential sources of interference in SA probe techniques. Paper presented at the Applied
Ergonomics Conference, San Francisco, CA.
Pierce, R. S. (2012). The effect of SPAM administration during a
dynamic simulation. Human Factors, 54(5), 838–848.
Prince, C., & Salas, E. (1993). Training and research for teamwork
in the military aircrew. In E. Wiener, B. Kannki, & R. Helmreich (Eds.), Cockpit resource management (pp. 337–365). San
Diego, CA: Academic Press.
Prince, C., Salas, E., & Stout, R. (1995). Situation awareness:
Team measures, training methods. In D. J. Garland & M. R.
Endsley (Eds.), Experimental analysis and measurement of
situation awareness. Daytona Beach, FL: Embry-Riddle University Press.
Salmon, P. M., Stanton, N. A., & Young, K. L. (2011). Situation
awareness on the road: Review, theoretical and methodological
issues and future directions. Theoretical Issues in Ergonomics
Science, 13(4), 472–492.
Sarter, N. B., & Woods, D. D. (1991). Situation awareness: A critical but ill-defined phenomenon. The International Journal of
Aviation Psychology, 1(1), 45–57.
Stanton, N. A., Salmon, P. M., & Walker, G. H. (2015). Let the
reader decide: A paradigm shift for situation awareness in
socio-technical systems. Journal of Cognitive Engineering and
Decision Making, 9(1), 44–50.
Strater, L. D., Faulkner, L. A., Hyatt, J. R., & Endsley, M. R.
(2006). Supporting situation awareness under data overload
in command and control visualizations. Paper presented at the
International Ergonomics Association 2006 Congress, Maastricht, the Netherlands.
Supreme Court of British Columbia. (2013). Regina v. KarlHeinz Arthur Lilgert. Vancouver, British Columbia, Canada:
Supreme Court of British Columbia.
Taylor, R. M. (1990). Situational awareness rating technique
(SART): The development of a tool for aircrew systems design.
In Situational awareness in aerospace operations (AGARDCP-478) (pp. 3/1–3/17). Neuilly Sur Seine, France: NATOAGARD.
van Winsen, R., & Dekker, S. W. A. (2015). SA Anno 1995: A commitment to the 17th century. Journal of Cognitive Engineering
and Decision Making, 9(1), 51–54.
Vidulich, M. A., & Tsang, P. S. (2015). The confluence of situation
awareness and mental workload for adaptable human-machine
systems. Journal of Cognitive Engineering and Decision Making,
9(1), 95–97.
Wickens, C. D. (2015). Situation awareness: Its applications value
and its fuzzy dichotomies. Journal of Cognitive Engineering
and Decision Making, 9(1), 90–94.
Mica R. Endsley is the Chief Scientist of the United
States Air Force. Prior to assuming this position, she
served as President and CEO of SA Technologies, and
served on the faculty at Texas Tech University and the
Massachusetts Institute of Technology. She received
her PhD from the University of Southern California in
Industrial and Systems Engineering.
Downloaded from edm.sagepub.com by Yemao Man on February 25, 2015