Gaining Insight Into
Decisionmaking Through a
Virtual World Environment
Application
May 2011
Author
Jerry W. Hedge, RTI International
Prepared for:
Department of Homeland Security
U.S. Department of Homeland Security
Washington, D.C. 20528
www.dhs.gov
Contract No. HSHQDC-08-C-00100
Prepared by:
RTI International–Institute of Homeland Security Solutions
Research Triangle Park, North Carolina
This document is in the public domain and may be used and reprinted without special
permission. Citation of the source is appreciated.
None of the investigators have any affiliations or financial involvement that conflicts with the
material presented in this report.
Suggested citation: Hedge, J. W. (2011). Gaining Insight Into Decisionmaking Through a
Virtual World Environment Application. (Prepared by RTI International–Institute for Homeland
Security Solutions under contract HSHQDC-08-C-00100.)
This report is based on research conducted under the Institute for Homeland Security
Solutions (IHSS) under contract to the Department of Homeland Security, Washington, DC.
(Contract HSHQDC-08-C-00100). The findings and conclusions in this document are those of
the author(s), who are responsible for its contents; the findings and conclusions do not
necessarily represent the views of the Department of Homeland Security. Therefore, no
statement in this article should be construed as an official position of the Department of
Homeland Security.
ii
Table of Contents
Overview of Research .............................................................................................................................. 1 Background ............................................................................................................................................... 1 Perspectives on Decisionmaking ........................................................................................................ 1 Virtual World Environment Research .................................................................................................. 4 Project Activities ....................................................................................................................................... 4 Review of Target Jobs and Available Department of Homeland Security (DHS) Technologies ......... 5 Selection of Job Activities and Development of Scenarios ................................................................. 6 Identification of Measurement Requirements and Development of Measurement Tools and
Procedures.......................................................................................................................................... 8 Development of Virtual World Environment ...................................................................................... 13 Alpha Test of Virtual World and Measurement Technologies ........................................................... 17 Summary ................................................................................................................................................ 17 Future Efforts .......................................................................................................................................... 17 References ............................................................................................................................................. 19 Appendix A: Examples of Cognitive Style Inventories ............................................................................ 23 Appendix B: Still Shots From Calexico and Otay Mesa Border Crossings ............................................. 29 Appendix C: Examples of Preliminary VWE Mock-Ups .......................................................................... 34 Appendix D: Alpha Test Version Mock-ups ............................................................................................ 40 iii
Overview of Research
The importance of sound decisionmaking to the strategic success of an enterprise is
common sense. The processes involved in decisionmaking have traditionally been
conceptualized as cognitive in nature, based on a rational and deliberate evaluation of the
situation at hand. This scientific or rational view of decisionmaking reflects a sequential model
in which decisionmakers are assumed to follow a number of stages in a prescribed order.
Although the exact number of stages and their contents vary somewhat from author to author
(see, for example, Fredrickson, 1984), the general steps may be summarized as (a) precise
formulation of the problem, (b) information search, (c) listing of alternative solutions,
(d) evaluation of alternatives according to predetermined criteria, and (e) choice of solution.
Unfortunately, few decisions in the real world are arrived at in this way. In a dynamic
environment, decision processes not only need to be well designed but they must adapt rapidly
to changes in the environment. Consequently, even on simple problems people often make
inconsistent decisions, ignore relevant information, and place greater emphasis on some
aspects of the decision than is reasonable. In sum, decisionmakers are imperfect information
processors.
The objective of the current project is to determine the feasibility of developing a virtual
world environment (VWE) for use as a test bed for studying the performance of
decisionmakers. We believe the VWE may provide an ideal platform for testing a variety of
research questions related to rational and intuitive decisionmaking because one can control
the immersive environment presented to the subjects as they begin the decisionmaking
process on a particular analytic problem. In sum, the research presented below describes the
development of a VWE and measurement processes which will allow the exposure of
participants to decisionmaking scenarios within that environment, and the measurement of
their responses as the decision process proceeds. We view this developmental effort as a first
phase in a program of research leading to a more definitive understanding of the
decisionmaking process.
Background
Perspectives on Decisionmaking
Heuer (1999), in his classic treatise on intelligence analysis, emphasized the perspective
of the fallibility of the decisionmaking process, noting the need for a clear understanding of the
inherent strengths and weaknesses of the primary analytic mechanism—the human mind—
and the way it processes information. He suggested that analysts construct their own version
of “reality” on the basis of information provided by the senses, but what they perceive, how
1
readily they perceive it, and how they process the information after receiving it are all strongly
influenced by a host of variables, which may or may not distort their view of reality.
A number of years ago, Simon (1978) distinguished between logical and judgmental
decisionmaking. In logical decisionmaking, goals and alternatives are made explicit, the
consequences of pursuing different alternatives are calculated, and these consequences are
evaluated in terms of how close they are to the goals. In judgmental decisionmaking, the
response to the need for a decision is usually rapid, too rapid to allow for an orderly sequential
analysis of the situation, and the decisionmaker cannot usually give a reliable account of either
the process by which the decision was reached or the grounds for judging it correct.
Generally, current theorizing endorses this dual-process model of decisionmaking in
human beings, and suggests that there are two distinct modes of operation of mental
processes: System 1 is autonomic, experiential, tacit, automatic, natural, and associative.
System 2 is intentional, rule-based, analytic, and explicit in nature (Gollwitzer & Bayer, 1999).
More recently, Price and Norman (2008) drew on the concept of fringe consciousness
(Mangan, 2003) to suggest an additional perspective that may help to bridge the dichotomy
between dual-process models.
Research on decisionmaking has focused extensively on System 2, and its logical or
rational components—but a broader decisionmaking perspective has begun to emerge as a
legitimate subject of scientific inquiry. Historically, research that has focused on System 1 or
judgmental decisionmaking (in Simon’s terminology) is best represented by the work of
Kahneman, Tversky, and colleagues, who have argued that heuristics, or mental shortcuts,
may lead to flawed decisionmaking (Kahneman, 2003; Kahneman, Slovic, & Tversky, 1982;
Tversky, & Kahneman, 1974). The scientific underpinnings of this perspective are based on
the earlier work of Paul Meehl (e.g., Dawes et al., 1989; Meehl, 1954), who argued that human
judges are more fallible than simple statistical models when comparing forecasting accuracy.
Others have suggested that accurate intuitive decisionmaking can be developed over time as
an individual accumulates more domain-specific knowledge and experience (Benner, 1982;
Dreyfus & Dreyfus, 1980; Phillips, Klein, & Sieck, 2004; Pyles & Stern, 1983). More recently,
Gigerenzer and Brighton (2009), in a review of the decision heuristics literature and through a
series of analytical demonstrations, suggested that in certain situations less information,
computation, and time can in fact improve the accuracy of decisionmaking.
Still other authors have suggested that both processing modes can contribute to effective
decisionmaking. For example, Shapiro and Spence (1997) proposed that problems lie on a
continuum of structuredness. Unstructured problems are less conducive to rational
decisionmaking because of the absence of well-accepted decision rules for dealing with such
situations. Sadler-Smith and Shefy (2007) added that where decisions do have to be made
speedily and with cognitive economy in the face of an overwhelming mass of information or
tight deadlines, individuals may have no choice but to rely on intelligent intuitive judgments
rather than on nonexistent or not-yet-invented cognitive routines. Indeed, rational thinking may
2
be more difficult when analysts face extreme time pressures or are confronted with novel or
unexpected situations.
Some accumulating evidence suggests that individuals are likely to rely on intuitive
thought processes when they face extreme time pressures or are confronted with novel or
unexpected situations. Consequently, intuitive decisionmaking may play a significant role, for
example, in the decisions of military commanders, firefighters, intelligence analysts, border
patrol agents, emergency room surgeons, corporate executives, and airport screeners,
operating under severe time constraints.
The reality of today’s world is that work environments are becoming increasingly complex
and unpredictable. Fast, high-quality, strategic decisionmaking in this context represents a
fundamental dynamic capability in high-performing organizations. However, in many modern
environments, a number of factors can affect the efficacy of an exclusively rational process.
Technological advancement and developments in organizational systems and processes have
contributed to an explosion in the volume of data that executives may be required to handle.
The volume and complexity of available information has the potential to be overwhelming.
When deliberative rational thought is not achievable or desirable (for example, where
unambiguous or sufficient “hard” data are not immediately at hand, might never be available at
all, or where creative solutions to problems are needed), one way of managing and coping with
uncertainty and complexity and of “thinking outside the box” is by relying on judgmental
decisionmaking. Cooper (2005) examined the intelligence analysis process and suggested that
a significant number of the most serious problems result from shortcomings in intelligence
analysis rather than from defects in collection, organization, or management.
Cooper argued that a more effective analytic paradigm must be built that incorporates the
best analytic methods from modern cognitive science—including use of more judgmental
approaches such as intuition and imagination—and employs useful and easily usable
supporting tools to overcome these impediments and prevent them from combining into
systemic pathologies.
In general, then, the literature suggests that there may be substantial promise in the
application of decisionmaking strategies beyond just the traditional rational approaches (see,
for example, Hedge & Aspinwall, 2009). However, much additional work—with both a basic
and an applied research focus—is required to critically evaluate the current state of scientific
knowledge with regard to decisionmaking. It will be important to better understand those
conditions that foster the effective use of different methods of decisionmaking. Future research
should also examine judgmental decisionmaking’s usefulness as both an alternative
decisionmaking strategy and as a strategy that supports a rational decisionmaking approach—
and differentiate under which conditions each might be most effective.
3
Virtual World Environment Research
The last decade has witnessed a proliferation of VWEs, opening new arenas of research
in social, economics, behavioral, and human-centered computer science (Bainbridge, 2007;
Fox & Bailenson, 2009; Yee et al., 2007). VWEs have been called the “ideal platform” to study
social interaction (McCall et al., 2009) and have led to the creation of valuable and innovative
research in experimentation (Castronova et al., 2009; Loomis et al., 1999), participant
observation (Antonijevic, 2008; Bardzell & Odom, 2008; Garcia et al., 2009; Williams, 2007),
quantitative analysis (Bell, 2007), and surveys (Bowman et al., 2002). A body of evidence
indicating that representative avatars, animated agents, and virtual spaces in immersive virtual
environments affect real-life social behaviors and choices is accumulating (Harris et al., 2009).
Although many researchers agree that research in virtual environments is valuable, for some
domains it is unclear whether the behavior observed in the VWE can be generalized to the real
world (Castronova et al., 2009). Yet for other domains, such as cognitive rehabilitation, the
effective use of virtual environments is well documented (Maltby et al., 2002; Wood et al.,
2010).
One of the keys to using the VWE domain as a research tool is how to apply the right
level of advanced technology in ways that focus the participant’s attention on the salient
information. Elements such as dynamic configuration of environments and objects, augmented
information, artificial visual cueing of elements, controlled interactivity, increase-decrease of
time and urgency, enhanced distracters, contextual framework, usability, and navigation within
the VWE can be used to increase the effectiveness of the scenario (Krizowsky et al., 2008).
In addition, in terms of performance assessment, VWEs offer the opportunity to capture
physiological measurements of human-computer interactions, including measures such as eye
movement, galvanic skin response, and heart rate. For example, the eye location and the
movement of the participant’s point of gaze can be tracked and associated with user actions at
that moment. Eye tracking provides information on the spatial and temporal aspects of the
participant’s attention, the user’s actions can be captured via the computer as the participant
interacts within the VWE, and stress indicators can be monitored through biometric collection
mechanisms. In combination, physiological and more traditional performance measurement
techniques may offer insights into users’ cognitive processes as they interact within the VWE.
Project Activities
The research presented below describes the activities undertaken to identify and develop
measurement tools and a VWE with controllable attributes, which can be used as a test bed to
examine the decisionmaking process more carefully.
4
Review of Target Jobs and Available Department of Homeland Security (DHS)
Technologies
Overview
The first step in the research and development process was to identify jobs (a) where a
range of decisionmaking activities would be likely to occur, and (b) that would benefit from
application of the proposed measurement and virtual world methodology. This first task
involved a review of potential jobs and associated data sources and publications that could
inform the work and serve as useful resources once a job was selected for development. Work
during this task also involved identifying currently available technologies tied to those jobs,
around which to build the VWE.
Task Activities
Within DHS there were a number of jobs where decisionmaking is a key component of
the work activity, including customs and border protection officers, intelligence analysts, border
patrol agents, immigrations and customs enforcement, citizenship and immigration service
adjudication officers, and airport screeners, to name a few. In addition, a subset of these jobs
function within a real-world environment where a VWE might be readily simulated to examine
the decisionmaking process. After considering a variety of options, the decision was made to
focus on the job of the customs and border protection (CBP) officer.
CBP officers work in an alert law enforcement environment to prevent the entry of
terrorists and terrorist weapons into the United States. This important mission extends to
inspecting travelers and their goods; detecting and seizing narcotics and other illegal,
prohibited, or dangerous articles; and interdicting the unlawful entry of undocumented or
prohibited persons seeking to enter the country.
CBP officers may be stationed at one of the United States’ international airports,
seaports, or land border crossings, performing their mission at more than 300 land, sea, and
air ports of entry, located throughout the United States, including the Canadian and Mexican
borders. In many ways, these ports of entry are the front line of defense against terrorist
intrusion and criminal activities, such as drug smuggling, money laundering, undocumented
entry of individuals, weapons trafficking, smuggling of prohibited goods, and a host of customs
violations. As U.S. federal law enforcement officers, CBP officers wear a uniform and carry
firearms. Activities include the following:
•
Playing a frontline role in CBP’s critical antiterrorism mission
•
Performing physical checks of travelers, cargo, and vehicles
•
Interacting with the traveling public arriving from overseas and inspecting luggage and
airborne cargo in international airports
•
Focusing on ships, containers, cruise passengers, luggage, and all types of seaborne
cargo in seaports
5
•
Inspecting commercial trucking and passenger vehicles at land border crossings
•
Using advanced technology tools and traditional methods such as K-9 teams
•
Working with fellow law enforcement officers from CBP and other agencies
After further consideration we chose to narrow the focus of development efforts to the job of
the CBP officer at land ports of entry.
Selection of Job Activities and Development of Scenarios
Overview
The emphasis of this task was on identification of several representative sets of job
activities, that when grouped together could depict functionally and operationally coherent sets
of tasks within the VWE yet to be built. Once identified, scenario development could begin.
Task Activities
Border Visits
To gain both a clearer understanding of the job responsibilities of the CBP officer and the
environment within which the work is performed, visits were made to several land ports of entry
along the southern border of the United States and Mexico. These visits also allowed us to
(a) visualize the physical layout of the facility and the processes by which CBP officers
engaged individuals wishing to cross the border, and (b) talk with the officers about how
decisions were made about allowing access or detaining travelers.
Visits were made to two ports of entry in California—Calexico and Otay Mesa. Calexico,
CA, is located just across the border from Mexicali, the state capital of Baja California, Mexico.
Otay Mesa, CA, is one of two border crossing within the city of San Diego. Based on the visits
and interviews with management at each facility, the following notes were generated.
1. The Calexico port of entry averages 10,000–12,000 pedestrians per day, and an
additional 12,000–15,000 vehicles (noncommercial) per day. Both entry components
operate 7 days a week.
2. The Otay Mesa port of entry averages 6,000 on foot per day, and another 12,000–
15,000 (noncommercial) vehicles per day. Both are open 7 days a week.
3. Both ports see some of the highest seizures of cocaine and methamphetamine
along the southern border.
4. Identity fraud is a growing problem (people use false IDs, which often get returned
and reused on multiple occasions).
5. The three checkpoint stages are (a) pre-primary (officers on foot checking vehicles
as they approach, with mirrors or canines), (b) primary (machine ID information
readers and officer in booth; machines include license plate reader, camera for
occupants, radiation machines, card reader for those in card program), and
6
(c) secondary (some additional tools like x-ray machines, plus canines, and
individual inspection).
6. For Calexico, the main downtown port has 10 primary lanes and a number of
secondary lanes. It is an old and crowded complex and it had funding earmarked to
begin development of a new facility next door by next year (which was recently cut
from the budget).
7. In Calexico, an East port was completed several years ago to handle primarily cargo
traffic, although foot and vehicle lanes are available but not used that much because
it is a longer drive/walk than the downtown facility. The East port has 16 vehicle
lanes, and plenty of extra space, plus adequate room before the primary area and
between primary and secondary (the downtown facility has little maneuverability).
8. The Port commander noted that the officers have approximately 47 seconds to
decide whether to pass a vehicle through or send it to secondary for individualized
inspection.
9. Primary sources of input for decisionmaking include license tag check, license ID
check, personal idiosyncrasies (nervousness, fidgeting, no eye contact), or vehicle
anomalies.
10. Both ports now have a SENTRI (primary processing) System. The SENTRI program
started in Otay Mesa in 1994, and now has more than 120,000 enrollees. CBP’s
goal is for 70% of all crossers to be certified in the SENTRI program. Individuals can
apply, pay a fee (around $125), and then once initially vetted, come in for an
interview. Once in the system, they pass through the border in less than 15 minutes,
as opposed to a 60- to 90-minute wait otherwise. However, because of security
issues, and fewer people in the program, they have a greater chance of randomly
being sent to secondary than do others not in the program.
Scenario Inputs
Based on discussions with a number of CBP officers during border crossing visits, and a
knowledge of the literature related to verbal and nonverbal deceit cues, work began on
developing detailed outlines of activities for a number of possible scenarios. To structure
preliminary VWE design in a controlled and manageable fashion (i.e., in such a way as to
facilitate both preliminary experimentation and later expansion of the environment), it was
decided that work should focus on CBP officer performance at the primary station. Recall from
the notes above that these officers have a very short time period within which to make a
decision about entry. During this time, these officers review whatever initial information is fed
to their computers based on machine readers at the check points (license tag check, license ID
check, etc.), and process visual personal idiosyncrasies (nervousness, fidgeting, no eye
contact), or vehicle anomalies. Based on these types of information, decisions are made to
approve entry or send the individual to a secondary station for further inspection.
To expand on these comments and suggestions from the CBP officers interviewed, a
review of the literature related to the “detection of deceit” was conducted. This literature (see,
for example, Buddharaju et al., 2005; Ekman & O’Sullivan, 1989; Navarro & Schafer, 2001;
7
Tsiamyrtzis et al., 2005; Vrij et al., 2000) reinforced the comments of the CBP officers, and
suggested designing a VWE and scenarios to allow incorporation of verbal and nonverbal
deceit cues. Those identified as potentially salient in the literature included facial cues (e.g.,
gaze aversion, rapid blinking of eyes, closing of eyes longer than a blink, more rapid breathing,
many short breaths, dry mouth, more frequent swallowing, more frequent yawning, and
covering mouth with hand), body posture/body movements (e.g., frequent hand/finger
movements, scratching head, scratching wrist, rubbing hands together, frequent leg/foot
movement, moving legs and feet simultaneously, no hand movement—arms drawn close to
body [frozen posture]), and latency of response (e.g., delayed response to questions).
These were then reviewed by the project team, and it was decided that such gestures
and movements could be designed into a VWE in such a way that scenarios could depict
border crossers revealing verbal and nonverbal deceit cues, in addition to the information
supplied via electronic means to officers depicted in the VWE as manning the primary
inspection stations.
Methods to Induce Intuitive Decisionmaking
Finally, because one focus of this long-term research and development effort is to
examine and identify similarities and differences between rational and intuitive decision
processes, certain manipulations within the VWE are possible to create conditions where such
questions can be more readily explored. For example, intuitive decisionmaking is more likely to
occur under certain conditions. Horstmann, Hausmann, and Ryf (2010) have defined five
conditions that can induce intuitive or deliberate decisionmaking during research experiments.
The list below is a subset of those conditions which could be implemented within the context of
a virtual environment to increase intuitive decisionmaking.
•
Provide information to view the decision within a holistic context.
•
Induce time pressure.
•
Implement distractions.
These conditions could be emphasized during the pre-session assessment and
conversation with the participants to increase intuitive decisionmaking.
•
Manipulate the mood by inducing a happy mood.
•
Induce low task relevance.
Identification of Measurement Requirements and Development of Measurement Tools
and Procedures
Overview
Work on this task required identifying the specific components of any activity within the
VWE that would supply information about participant performance/decisionmaking, and the
feasibility of gathering such measurements in these scenarios. Part of the focus of work in this
8
task, therefore, was to gain an understanding of what measurement tools exist or can be
developed to capture the behaviors of participants in ways that differentiate more successful
and less successful decisionmaking; and in fact, whether the scenarios being developed can
lead to measurable processes and outcomes. Then, once we had a better understanding of
the measurement tools that exist or could be developed to capture the behaviors of
participants within a VWE, we began work to incorporate the approaches that appeared to be
most feasible within the time and financial constraints of the current project.
Task Activities
This task focused, then, on identifying actions and activities that could be observed and
measured (thus providing the link between scenario development and VWE development) and
incorporating or developing appropriate measurement techniques and processes.
Measurement components were categorized based on sequence of administration—that is,
pre-administration, during administration, and post-administration. In the paragraphs that
follow, however, each is discussed based on conceptual content.
Performance Measures
Based on our review of the performance measurement literature and virtual world
performance measures literature, we targeted design of multiple measures of decision
performance for use with participants playing the role of CBP officers in the virtual
environment. These measures include both data captured via computer and analyzed, and
observer and participant ratings of performance. Each is described in greater detail next.
Decision Accuracy. When participants are interacting in a virtual border crossing
scenario, each will be faced with a series of decision points. The accuracy of each decision
point in the scenario will be recorded. For instance, for a particular vehicle at the most macro
level of decisionmaking, multiple options are possible (e.g., allow a vehicle to cross that should
be denied, deny crossing to one that should be allowed to cross, properly decide to send for
secondary inspection a mildly suspicious vehicle). In this way, the participant’s decision (or
decision errors) at each decision point will be recorded. Ultimately the participant will have to
decide whether the person at the checkpoint should be allowed to cross the border. In this
way, data on the number of persons correctly denied access, the number of persons correctly
allowed to cross, the number of persons incorrectly denied access, and the number of persons
incorrectly allowed to cross will be recorded. In addition, for more complex scenarios, with
multiple decisions (and more interim decisions) to be made by the participant, additional
choicepoint decisions can be recorded and analyzed.
Response Time. The length of time the participant takes to reach a decision at each
decision point will also be measured, as will the overall length of time taken to reach the
ultimate decision of allowing the person to cross or not cross the border. Response times can
then be extracted and analyzed, separately and in conjunction with decision accuracy data to
9
take a closer look at issues related to speed-accuracy tradeoffs. At some point in time, it may
also be possible to use response time as an indicator of intuitive or rational decisionmaking
activities (e.g., the length of time taken to reach decisions could be used to predict the type of
decisionmaking being used, intuitive [less time] or deliberative [more time]).
Information Gathering. Albrechtsen, Meissner, and Susa (2009) defined intuitive
processing as “an affective or experiential mode that is effortless, spontaneous, and holistic in
nature” (p. 1052). In contrast, deliberative processing “requires conscious effort and is
generally a slower, more analytic process” (p. 1052). If intuition is presumed to be automatic
and deliberative processing is slower and more analytic it follows that conscious information
searching and gathering may be used to predict which type of decisionmaking is occurring in
each scenario. More information gathering could indicate the use of deliberative processing
while less information gathering could indicate the use of intuitive processing. For example,
key strokes used to maneuver inside of the virtual environment before a decision is made
could be a measurement of information gathering. Likewise, eye tracking software can record
eye positions and movements as a measure of decisionmaking (Norman & SchulteMecklenbeck, 2009). Fixation and duration of fixation on objects in the virtual environment
could suggest which type of processing is being employed by the participant. Naturally, much
additional experimentation will be required to move beyond the point of conjecture, but the
VWE being developed will provide the research test bed for investigating such speculation.
Subjective Ratings of Performance. Beyond the collection of computer-based
performance measures, decisionmaking performance will also be assessed via several rating
scales. First, observers will evaluate participants’ overall performance on the task, using a
Likert-type rating scale to gauge effectiveness and efficiency. Similarly, participants, once they
have completed the task, will rate their own overall performance.
In addition to a global rating of performance on the entire task, observers and participants
will evaluate scenario- or decision-specific performance, which includes ratings of accuracy
and timeliness/speed of response. Finally, both observers and participants will evaluate
performance using subscales developed to allow ratings at a dimension level (i.e., situational
awareness, timing, accounting for contingencies, use of available information, ability to
visualize unfolding situation and options).
Self-Efficacy
The self-efficacy of participants will be measured prior to performing in the VWE. Selfefficacy is commonly defined as the belief in one’s capabilities to complete a task or achieve a
goal or an outcome. Individuals with a strong sense of efficacy are more likely to challenge
themselves with difficult tasks and be intrinsically motivated. According to Bandura, individuals
high on the self-efficacy scale put forth a high degree of effort to meet their commitments, and
attribute failure to things which are in their control, rather than blaming external factors. They
also recover quickly from setbacks, and ultimately are likely to achieve their personal goals.
10
Individuals with low self-efficacy, on the other hand, believe they cannot be successful and
thus are less likely to make a concerted, extended effort and may consider challenging tasks
as threats that are to be avoided (Bandura, 1994).
Cognitive and Noncognitive Measures
Prior to taking part in the VWE scenarios, all individuals will be administered a short
battery of cognitive and noncognitive inventories, geared particularly toward determining
cognitive styles and personality traits. To date, a number of brief inventories have been
developed to identify use of or preference for rational or intuitive decisionmaking, including
(a) Rational-Experiential Inventory (REI)—40 items which measure engagement in/preference
for rational and intuitive information processing; (b) Preference for Intuition & Deliberation
Scale (PID)—18 items which directly assess preference for two constructs (intuition and
deliberation —measuring affect/implicit knowledge and explicit knowledge); (c) General
Decisionmaking Style (GDMS)—15 items which measure preferences for decision styles (five
styles: rational, intuitive, dependent, spontaneous, avoidance); (d) Cognitive Style Indicator
(CoSI)—11 items assessing three styles: knowing, planning, creating; and (e) Perceived
Modes of Processing Inventory (PMPI)—10 items assessing awareness/perception of one’s
own dominant style in stressful situation. Because of time constraints it is not likely that all
scales will be administered together, but development of the VWE test bed will allow (at some
point) a closer examination of the commonalities and uniquenesses of all these instruments at
the same point in time.
In addition, a more traditional personality inventory (i.e., the NEO-five factor inventory—
NEO-FFI; 60 items) will be administered to identify personality traits in participants. This will
allow an examination of whether personality is associated with rational or intuitive decision
styles, speed-accuracy tradeoffs, etc.
Presence
The effectiveness of virtual environments (VEs) has often been linked to the sense of
presence reported by users of those VEs. Presence is defined as the subjective experience of
being in one place or environment, even when one is physically situated in another. As applied
to a VE, presence refers to experiencing the computer-generated environment rather than the
actual physical locale. Witmer and Singer (1998) developed a presence questionnaire (PQ) to
measure presence in VEs, and an immersive tendencies questionnaire (ITQ) to measure
differences in the tendencies of individuals to experience presence. They also argued that both
immersion and involvement are necessary for experiencing presence and that they interact to
determine how much presence is reported.
Experience Measures
Prior to engaging any participants in the VWE scenarios, information will be collected
pertaining to certain relevant experiences of these participants. This is important because
11
differences in experience may influence performance in the VWE scenarios. For example, prior
experience with video games, gaming, Second Life, computers, etc., may provide a clear early
advantage in scenario completion and general successful movement through the VWE. If
differences exist, these will need to be controlled for in subsequent analyses.
After-Action Review
Once participants have completed their activities in the VWE, each will be queried about
their performance in the scenarios. In particular, the interview explores issues related to
decisions, judgments, thought processes, and rationale behind actions and decisions. Each
participant will also be asked a series of questions related to their attitudes and opinions of the
VWE, the scenarios, the process, etc. Together, this after-action review is designed to gather
additional insights into the VWE and measurement system with the aim of improving the test
bed that has been developed.
Summary
To summarize, after careful review, the following measures have been identified (selected
or developed) as important components of a measurement system for use when examining
decisionmaking activities in a VWE.
1. Pretest Measurement
a. Cognitive style inventories (e.g., REI, PID, GDMS, CoSI, PMPI)
b. Personality inventory (NEO-FFI or possibly Myers-Briggs)
c. Immersive tendencies questionnaire (ITQ)
d. Self-efficacy
e. Previous experience—with video games, gaming, Second Life, computers, etc.
[A–E are all self-report measures]
2. During Testing
a. Computer-based measurement (decisionmaking accuracy, response time,
information gathering)
b. Observational
1. Overall rating of performance
2. Dimension-level ratings of performance
3. Scenario- or decision-specific ratings of accuracy and timeliness/speed of
response
3. Post-test
a. Self-report of performance (same as 2b above)
b. “Presence” questionnaire (measure of immersion in the VWE)
12
c. After-action review—questions about decisions, judgments, thought processes,
and rationale behind them
d. Attitudes/opinions (e.g., on testing, VWE, scenarios)
Development of Virtual World Environment
Overview
In this task we created an environment that synthesizes key elements of the task
environment we wish to incorporate into the VWE. The concept was to mimic the task
environment to situate the individual in a setting where real-world performance could be
expected. Consequently, our developmental efforts focused on production of a realistic VWE
that could be used for examining the decisionmaking performance of individuals. As such, the
VWE was constructed to (a) reflect the basic elements of a border crossing, and (b) allow
flexibility for design improvements and expansion of the VWE in the future.
Task Activities
Technology is used for rapid detection, collection, organization, and dissemination of
data, including those related to identity and threats. Properly employed, technology can
enhance the ability of a trained officer or analyst to make correct security judgments about
individuals and situations encountered at the border. Critics note that there can be no
substitute for the learning and intuition of an experienced field officer. DHS has recognized that
any effective border protection efforts will likely provide an appropriate balance of manpower,
technology, and infrastructure.
Background: Point of Entry (POE) Inspection Process
The process consists of several inspection stations. The goal of the primary inspection is
to quickly admit legitimate travelers into the United States and also quickly identify and refer
high-risk travelers and inadmissible aliens for a more detailed secondary inspection. If primary
inspectors have concerns about a traveler, whether based on definite information or just an
uneasiness about the traveler’s demeanor, they are to refer the traveler for a secondary
inspection. Primary inspections are not expected to be detailed and are expected to be
accomplished within an extremely short period of time, generally well less than 2 minutes.
During a secondary inspection, other CBP officers can perform as detailed an examination as
necessary, without concerns about inconveniencing United States citizens and legitimate nonU.S. citizen travelers seeking to enter the country.
Primary Inspections. Upon arrival at the POE, CBP officers conduct initial examinations
of all arriving visitors at primary inspection. The primary inspection is the only inspection
needed for those who are clearly admissible. Although the inspectors on average have less
than 1 minute per person, each person is checked through a centralized computer system. The
13
vast majority of individuals are clearly admissible and are entering the United States for
legitimate purposes with legitimate documents.
•
Upon arrival at the POE the traveler must present passport and other required
documents. The CBP officers review these to determine whether the traveler will be
allowed to enter the United States.
•
The CBP officer will ask questions to determine identity and nationality.
•
The CBP officer will determine how long the travelers will be allowed to stay in the
United States, and under what status. They will ask questions to determine the true
intent of the traveler’s trip to the United States.
•
CBP officers review passports, visas, and other supporting documents of every foreign
national arriving at a U.S. POE. CBP officers also compare fingerprint records and
name check databases for recent negative information, and ask questions about the
foreign national’s general qualifications for the visas they have.
Secondary Inspections. Any person, foreign national or person with a claim to U.S.
citizenship and presenting a U.S. passport, may be sent to secondary inspection if the CBP
officer has reservations about admitting him or her to the United States. A person may also be
sent to secondary inspection if there is a possibility he or she is smuggling contraband or
violating any other customs or immigration regulations, or federal law in general. If there
appear to be discrepancies in documents presented or answers given, or if there are any other
problems, questions, or suspicions that cannot be resolved within the exceedingly brief period
allowed for primary inspection, the traveler will be referred to a secondary inspection
procedure, where a more thorough inquiry may be conducted. Secondary inspection allows
inspectors to conduct additional research to verify information. Historically, the great majority of
these secondary inspections result in the admission of the person being inspected.
Environment. In part, because previous visits were to border crossing sites between the
United States and Mexico, it was decided that development of the VWE should depict a
notional southwestern location which includes distant mountains, and semi-arid to arid terrain;
the area consisting of the POE should be flat with highway roads feeding into the site. Based
on the visits described previously, and other available information, the following description of
a typical border crossing site guided environment development.
Primary Inspection Area. The primary vehicle inspection area includes multiple
inspection lanes, consisting of standard vehicular lanes (12 feet wide) and one bus lane (14
feet wide). The standard vehicular lanes include horizontally stacked inspection booths. A
portion of the primary vehicle inspection area is covered with canopies.
Secondary Inspection Area. The secondary inspection and operations center island
contains multiple inspection spaces and several inspection booths and is covered with
canopies. The access points to the secondary inspection area are equipped with nonintrusive
inspection facilities, such as gamma ray scanning equipment.
14
Auto Seizure and Impound Facilities. North of the secondary inspection area is an auto
seizure building and impound facility. This facility includes an impound parking lot to
accommodate multiple impounded vehicles. A portion of this area is covered with canopies.
Access is provided from an east-west connector road.
Operations Center. An operations center building is immediately east of the secondary
inspection area. The operations center building would encompass approximately 50,000
square feet on two floors.
Vehicles. A variety of vehicles are included in the environment—cars, trucks, vans—of
differing sizes, colors, makes, and models.
Locations of Contraband Concealment. A variety of options for contraband
concealment are included to reflect areas of concealment identified by CBP officers
interviewed, and other CBP publications. These include in tires, door compartments, seats,
parts under the hood (washer fluid reservoir, fire wall of engine compartment), luggage,
intermingled with other goods (e.g., clothes, mulch), on their person, and stored in other goods
(e.g., barrels, washing machine).
Types of Contraband. Types of contraband are varied, to include intellectual property
(e.g., counterfeit handbags, counterfeit tennis shoes, toys/electronic games), agriculture (e.g.,
pork products, citrus fruits, insect-infected fruits and vegetables), illegal drugs (e.g., marijuana,
cocaine, heroin, ecstasy, methamphetamine), weapons and ammunition, counterfeit
identification documents (e.g., license, birth certificate, visa, passport, green card).
Avatar Descriptions. Avatars reflect a mix of genders, ethnicities, and single-person and
multiple-passenger vehicles including children.
Electronics. There is a computer terminal in the Primary Inspections Booth. The agent
uses information from the program to aid in determining whether to further question the
individual or inspect the vehicle. The software also uses a randomized statistical sampling to
determine the level of threat at the land border ports.
VWE Development
Approach. The current environment was developed to allow for future expansion as new
research objectives and new research studies arise. Purchase of existing models and
environments focused on world building and then programming of additional logic.
Environment. The primary VWE design objective of the current project was to develop
the surrounding environment and a preliminary Primary Inspection Area. Additional
infrastructure was added to heighten the reality of the environment (e.g., operations building)
within the constraints of the budget and schedule. For experimentation, the setup of a “primary
inspection booth” will require access and space for the human-directed avatar to sit and view a
computer screen.
15
Vehicles. The vehicles for this phase consist of personal vehicles such as cars, vans,
and small trucks. Vehicles were constructed so as to allow a human-directed avatar to sit and
maneuver the vehicle. The design has built-in flexibility to allow doors and trunks to open for
inspections. Access to the contraband concealment areas listed above are also part of the
ongoing design plan.
Avatars. Rudimentary avatar development was included in the current version of the
VWE, along with plans to add automated-scripted avatars into the scenarios in roles such as
other passengers or drivers of distracter automobiles and additional scripted behavioral
indicators (e.g., facial expressions and gestures), which can be added to the human-directed
avatars.
Electronics. Software has been developed in a notional method, although the
functionality will be reduced to several still screens which provide the participant/officer with
GO-NO GO information on further investigations.
Graphical User Interface. Two timer elements are included, visible to the participants of
the VWE: one that counts down the overall time of the scenario and one that resets each time
a new automobile arrives at the primary inspections booth.
Roles. The design of the VWE includes several human-directed Avatar roles in the
scenario. These roles include the research participant who will be in the role of the CBP officer
and the drivers of automobiles who are crossing the border. There will be multiple drivers to
allow the CBP officer the opportunity to encounter both those with criminal intent and those
without.
•
CBP officer (research subject): It will be the responsibility of the CBP officer to assess
individuals and vehicles passing through the POE to determine whether to initiate a
secondary investigation.
•
Drivers: The driver will arrive at the POE checkpoint, provide the required
documentation, and answer questions. Some drivers will carry contraband in their
vehicles as they attempt to cross the POE. Scripts will dictate roles and activities for
each driver character.
Activity. Design of the system requires that each CBP officer (research participant)
consult his or her software, request documentation, and question individuals passing the
primary inspection station. During the exercise the officer will be provided with the opportunity
to assess individuals with both criminal and noncriminal intent. The officer will be required to
make a determination on all individuals requesting passage. It is anticipated that several
automobiles will be lined up at the checkpoint so the CBP officer will be required to make
several quick assessments and determinations.
Tempo. At land ports primary passenger inspections are designed to be conducted in
less than 1 minute. According to the CBP the average inspection time per vehicle is less than 1
minute, during which the officer reviews documentation for all vehicle occupants and
determines whether to detain or transfer for secondary inspection. Scenarios are designed to
16
use a similar time criterion to force the CBP officers to make quick decisions similar to real-life
job expectations.
Alpha Test of Virtual World and Measurement Technologies
Once the preliminary performance measurement and VWE development was completed,
it was important to initiate a review of the various components as a first test of feasibility,
functionality, and operability. As is commonly described in the literature, such an alpha test is
the first formal test of a newly developed hardware or software product by internal people.
Members of the RTI development team reviewed each aspect of the system, guided by
an emphasis on whether the system as currently configured could meet our feasibility criteria;
that is, functionality confirmation. Careful review of system and environment characteristics of
the VWE led to a variety of relatively minor adjustments. Similar review of the measurement
system also led to some fine-tuning of approaches. A broader, secondary review of whether
the developmental steps were sufficient to establish the system as a flexible foundation around
which to build further expansion of environment and measurement tools as required confirmed
that it was. Ultimately, based on the team review, it was determined that these thresholds had
been obtained, both for the measurement instruments and the VWE.
Summary
The objective of the current project was to establish the feasibility of developing a VWE
for use as a test bed for studying the performance of decisionmakers. In addition, it was
important, looking ahead to future research efforts using this test bed, that we research and
understand the issues associated with measuring the decisionmaking performance of
individuals working in this VWE. As a result of our project activities, we strongly believe that
the VWE can provide an ideal platform for testing a variety of research questions related to
rational and intuitive decisionmaking. In sum, this research report has described a preliminary
effort to investigate the usefulness of a VWE for studying the decisionmaking process, and
applying those findings to real-world settings. We view this feasibility study as a first phase in a
program of research leading to a more definitive understanding of that process.
Future Efforts
The reality of today’s world is that work environments are becoming increasingly complex
and unpredictable. Fast, high-quality, strategic decisionmaking in this context represents a
fundamental dynamic capability in high-performing organizations. However, in many modern
environments, a number of factors can affect the efficacy of an exclusively rational process.
The literature suggests that there may be substantial promise in the application of
decisionmaking strategies beyond the traditional rational approaches (Hedge & Aspinwall,
17
2009; Hedge & Whiteford, 2010; Hedge et al., 2010). Much additional basic and applied
research is required to critically evaluate the current state of scientific knowledge with regard to
decisionmaking.
It will be important to better understand the conditions that foster the effective use of
different methods of decisionmaking, and judgmental decisionmaking’s usefulness as both an
alternative decisionmaking strategy and as a strategy that supports a rational decisionmaking
approach. The current work is a first step leading to a more definitive understanding of the
decisionmaking process. Importantly, we intend to examine within a VWE both the more
traditionally understood, rational decisionmaking process and the less traditional and more
judgmental strategies. The objective will be to determine whether different approaches are
optimal in particular situations, may be used most effectively in combination in other situations,
or may even be influenced by the analyst’s skill set.
18
References
Albrechtsen, J. S., Meissner, C. A., & Susa, K. J. (2009). Can intuition improve detection
deception performance? Journal of Experimental Social Psychology, 45, 1052-1055.
Antonijevic, S. (2008). From text to gesture online: A microethnographic analysis of nonverbal
communication in the Second Life virtual environment. Information, Communication &
Society, 11(2), 221–238.
Bainbridge, W.S. (2007). The Scientific Research Potential of Virtual Worlds. Science,
317(5832), 472-476.
Bandura, A. (1994). Self-efficacy. In V. S. Ramachaudran (Ed.), Encyclopedia of human
behavior (Vol. 4, pp. 71–81). New York: Academic Press.
Bardzell, S. & Odom, W. (2008). The experience of embodied space in virtual worlds: An
ethnography of a Second Life community. Space and Culture, 11(3), 239–259.
Bell, M. (2007). By the numbers: Quantitative research methods In Second Life. Proceedings
of the Second Life Education Workshop at the Second Life Community Convention.
Chicago, Il.
Benner, P. (1982). From novice to expert. American Journal of Nursing, 82(3), 402–407.
Betsch, C. (2007). Chronic preferences for intuition and deliberation in decisionmaking:
Lessons learned about intuition from an individual differences approach. In H. Plessner,
C. Betsch, & T. Betsch (Eds.). Intuition in judgment and decisionmaking (pp. 231–248).
Mahwah, NJ: Lawrence Erlbaum.
Bowman, D. A., Gabbard, J. L., & Hix, D. (2002). A survey of usability evaluation in virtual
environments: Classification and comparison of methods. Presence, 11(4), 404–242.
Buddharaju, P., Dowdall, J., Tsiamyrtzis, P., Shastri, T., Pavlidis, I., & Frank, M. G. (2005).
Automatic THErmal Monitoring System (ATHEMOS) for deception detection. In IEEE
Conference on Computer Vision and Pattern Recognition, San Diego, CA.
Burns, L. R. & D’Zurilla, T. J. (1999). Individual differences in perceived information-processing
styles in stress and coping situations: Development and validation of the Perceived
Modes of Processing Inventory. Cognitive Therapy and Research, 23, 345–371.
Castronova, E., & Falk, M. (2009). Virtual worlds: Petri dishes, rat mazes, and supercollider.
Games & Culture, 4(4), 396–407.
Cools, E., & Van den Broeck, H. (2007). Development and validation of the Cognitive Style
Indicator. The Journal of Psychology, 14, 359–387.
19
Cooper, J. R. (2005). Curing analytic pathologies: Pathways to improved intelligence analysis.
Washington, DC: Center for the Study of Intelligence.
Dawes, R. M., Faust, D., & Meehl, P. (1989). Clinical versus actuarial judgment. Science, 243,
1668–1674.
Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in
directed skill acquisition (Operations Research Center #ORC 80-2). Berkeley, CA:
University of California.
Ekman, P., & O’Sullivan, M. (1989). Hazards in detecting deceit. In D. C. Raskin (Ed.),
Psychological methods in criminal investigation and evidence (pp. 297–332). New York:
Springer Publishing.
Fox, J. and Bailenson, J.N. (2009). Virtual self-modeling: The effects of vicarious reinforcement
and identification on exercise behaviors. Media Psychology, 12 (1), 1-25.
Fredrickson, J. W. (1984). The comprehensiveness of the strategic decision process:
Extension, observations, future directions. Academy of Management Journal, 27, 445–
466.
Garcia, A. C., Standlee, A. I., Bechkoff, J., & Cui, Y. ( 2009). Ethnographic approaches to the
internet and computer-mediated communication. Journal of Contemporary Ethnography,
38(1), 52–84.
Gigerenzer, G., & Brighton, H. (2009). Homo heuristicus: Why biased minds make better
inferences. Topics in Cognitive Science, 1, 107–143.
Gollwitzer, P., & Bayer, U. (1999). Deliberative versus implemental mindsets in the control of
action. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology (pp.
403–422). New York: Guilford Press.
Harris, H., Bailenson, J. N., Nielsen, A., & Yee, N. (2009). The evolution of social behavior
over time in Second Life. Presence, 18(6), 434–448.
Hedge, J. W., & Aspinwall, K. R. (2009, November). Can intuitive decisionmaking improve
homeland security? IHSS Research Brief. Research Triangle Park, NC: Institute for
Homeland Security Solutions.
Hedge, J. W., & Whiteford, B. M. (2010, September). Examining rational and intuitive
decisionmaking in a virtual world environment. Presented at the annual meeting of the
International Military Testing Association, Lucerne, Switzerland.
Hedge, J. W., Whiteford, B. M., & Ohse, D. M. (2010, November). The feasibility of studying
decisionmaking using a virtual world environment. Presented at the Institute for
Homeland Security Solutions second annual research summit, Durham, NC.
20
Heuer, R. J. (1999). Psychology of intelligence analysis. Washington, DC: Center for Study of
Intelligence.
Horstmann, N., Hausmann, D., & Ryf, S. (2010). Methods for inducing intuitive and deliberate
processing modes. In A. Glöckner & C. L. M. Witteman (Eds.), Foundations for tracing
intuition: Challenges and methods (pp. 219–237). London: Psychology Press &
Routledge.
Kahneman, D. (2003). A perspective on judgment and choice. American Psychologist, 58,
697–720.
Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and
biases. Cambridge: Cambridge University Press.
Krizowsky, P., Waters, H., Wright, M., Hubal, R., Frank, G., & Whiteford, B. (2008).
Dynamically configured scenarios for training adaptive network system operators.
Presented at Interservice/Industry Training, Simulation, and Education Conference.
Loomis, J. M., Blascovich, J. J., & Beall, A .C. (1999). Immersive virtual environment
technology as a basic research tool in psychology. Behavior Research Methods,
Instruments, & Computers, 31(4), 557–564.
Maltby, N., Kirsch, I., Mayers, M., & Allen, G. (2002). Virtual reality exposure therapy for the
treatment of fear of flying: A controlled investigation. Journal of Consulting and Clinical
Psychology, 70(5), 1112–1118.
Mangan, B. (2003). The conscious ‘fringe’: Bringing William James up to date. In B. J. Baars,
W. P. Banks, & J. B. Newman (Eds.), Essential sources in the scientific study of
consciousness (pp. 741–759). Cambridge, MA: The MIT Press.
McCall, C., Bailenson, J. N., Blascovich, J., & Beall, A. C. (2009). Leveraging collaborative
virtual environment technology for inter-population research on persuasion in a
classroom setting. PRESENCE: Teleoperators & Virtual Environments, 18(5), 361–369.
Meehl, P. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of
the evidence. Minneapolis, MN: University of Minnesota Press.
Navarro, J., & Schafer, J. R. (2001). Detecting deception. FBI Law Enforcement Bulletin, 9–12.
Norman, E. & Schulte-Mecklenbeck, M. (2009). Take a careful click at that! Mouselab and eyetracking as tools to measure intuition. In: A. Glockner & C. Witteman (Eds). Foundations
for tracing intuition: Challenges and methods. London: Psychology Press.
Pacini, R. & Epstein, S. (1999). The relation of rational and experiential information processing
styles to personality basic beliefs, and the ratio-bias phenomenon. Journal of
Personality and Social Psychology, 76, 972–987.
21
Phillips, J. K., Klein, G., & Sieck, W. R. (2004). Expertise in judgment and decisionmaking: A
case for training intuitive decision skills. In D. J. Koehler & N. Harvey (Eds.), Blackwell
handbook of judgment & decisionmaking (pp. 297–315). Victoria, Australia: Blackwell
Publishing.
Price, M. C., & Norman, E. (2008). Intuitive decisions on the fringes of consciousness: Are they
conscious and does it matter? Judgment and Decision Making 3(1), 28–41.
Pyles, S., & Stern, P. (1983). Discovery of nursing gestalt in critical care nursing: The
importance of the gray gorilla syndrome. Journal of Nursing Scholarship, 15, 51–57.
Sadler-Smith, E., & Shefy, E. (2007). Developing intuitive awareness in management
education. Academy of Management Learning and Education, 6(2), 186–205.
Scott, S. G. & Bruce, R. A. (1995). Decisionmaking style: The development and assessment of
a new measure. Educational and Psychological Measurement, 55, 818–831.
Shapiro, S., & Spence, M. T. (1997). Managerial intuition: A conceptual and operational
framework. Business Horizons, 40(1), 63–68.
Simon, H. A. (1978). Administrative behavior, 2nd ed. New York: Free Press.
Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I., Frank, M. G., & Ekman, P. (2005,
November). Imaging facial physiology for the detection of deceit (Technical Report
Number UH-CS-05-25), Houston, TX: Department of Computer Science, University of
Houston.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases.
Science, 185, 1124–1131.
Vrij, A., Edward, K., Roberts, K. P., & Bull, R. (2000). Detecting deceit via analysis of verbal
and nonverbal behavior. Journal of Nonverbal Behavior, 24(4), 239–263.
Williams, M. (2007). Avatar watching: Participant observation in graphical online environments.
Qualitative Research, 7(1), 5–24.
Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence
questionnaire. Presence, 7(3), 225–240.
Wood, D., Center, K., Murphy, J., McLay, R., Koffman, R., Johnston, S., Spira, J., Pyne, J., &
Wiederhold, B. (2010). Combat related post traumatic stress disorder: A multiple case
report using virtual reality graded exposure therapy with physiological monitoring.
American Board of Psychology Professionals, 29(1), 20–23.
Yee, N., Bailenson, J. N., Urbanek, M., Chang, F., & Merget, D. (2007). The unbearable
likeness of being digital: The persistence of nonverbal social norms in online virtual
environments. CyberPsychology & Behavior, 10, 115–121.
22
Appendix A: Examples of
Cognitive Style Inventories
PID (Preference for Intuition and Deliberation Scale, Betsch, 2007)
Scoring: recode items denoted with (-); calculate the means of the items of each scale.
PID- Deliberation
1. Before making decisions I first think them through.
2. Before making decisions I usually think about the goals I want to achieve.
3. I prefer making detailed plans rather than leaving things to chance.
4. I am a perfectionist.
5. I think about a decision particularly carefully if I have to justify it.
6. When I have a problem I first analyze the facts and details before I decide.
7. I think before I act.
8. I think more about my plans and goals than other people do.
9. I think about myself.
PID- Intuition
10. I listen carefully to my deepest feelings.
11. With most decisions it makes sense to completely rely on your feelings.
12. I don’t like situations that require me to rely on my intuition. (-)
13. I prefer drawing conclusions based on my feelings, my knowledge of human nature,
and my experience of life.
14. My feelings play an important role in my decisions.
15. When it comes to trusting people, I can usually rely on my gut feelings
16. I prefer emotional people.
17. I am a very intuitive person.
18. I like emotional situations, discussions, and movies.
23
GDMS (General Decisionmaking Style, Scott & Bruce, 1995)
Scoring: calculate the means of the items of each scale.
GDMS INTUITION
1. When making decisions, I rely upon my instincts
2. When I make decisions, I tend to rely on my intuition
3. I generally make decisions that feel right to me
4. When I make a decision, it is more important for me to feel the decision is right than
to have a rational reason for it
5. When I make a decision, I trust my inner feeling and reactions
GDMS Spontaneous
1. I generally make snap decisions.
2. I often make decisions on the spur of the moment.
3. I make quick decisions.
4. I often make impulsive decisions.
5. When making decisions, I do what seems natural at the moment.
GDMS Deliberation
1. I plan my important decisions carefully
2. I double-check my information sources to be sure I have the right facts before making
decisions
3. I make decisions in a logical and systematic way
4. My decisionmaking requires careful thought
5. When making a decision, I consider various options in terms of a specific goal
24
REI (Rational – Experiential Inventory, Pacini & Epstein, 1999)
Scoring: recode items denoted with (-); calculate the means of all items per scale. To
reach overall values for experientiality and rationality calculate the means of the engagement
and ability scales, respectively.
REI Experiential ability
1. I don’t have a very good sense of intuition (-)
2. Using my gut feelings usually works well for me in figuring out problems in my life.
3. I believe in trusting my hunches.
4. I trust my initial feelings about people.
5. When it comes to trusting people, I can usually rely on my gut feelings.
6. If I were to rely on my gut feelings, I would often make mistakes. (-)
7. I hardly ever go wrong when I listen to my deepest gut feelings to find an answer.
8. My snap judgments are probably not as good as most people’s. (-)
9. I can usually feel when a person is right or wrong, even if I can’t explain how I know.
10. I suspect my hunches are inaccurate as often as they are accurate. (-)
REI Experiential engagement
11. I like to rely on my intuitive impressions.
12. Intuition can be a very useful way to solve problems.
13. I often go by my instincts when deciding on a course of action.
14. I don’t like situations in which I have to rely on intuition. (-)
15. I think there are times when one should rely on one’s intuition.
16. I think it is foolish to make important decisions based on feelings. (-)
17. I don’t think it is a good idea to rely on one’s intuition for important decisions. (-)
18. I generally don’t depend on my feelings to help me make decisions. (-)
19. I would not want to depend on anyone who described himself or herself as intuitive.
(-)
20. I tend to use my heart as a guide for my actions.
25
REI rational ability
1. I’m not that good at figuring out complicated problems. (-)
2. I am not very good at solving problems that require careful logical analysis. (-)
3. I am not a very analytical thinker. (-)
4. Reasoning things out carefully is not one of my strong points. (-)
5. I don’t reason well under pressure. (-)
6. I am much better at figuring things out logically than most people.
7. I have a logical mind.
8. I have no problem thinking things through carefully.
9. Using logic usually works well for me in figuring out problems in my life.
10. I usually have clear, explainable reasons for my decisions.
REI rational engagement
11. I try to avoid situations that require thinking in depth about something. (-)
12. I enjoy intellectual challenges.
13. I don’t like to have to do a lot of thinking. (-)
14. I enjoy solving problems that require hard thinking.
15. Thinking is not my idea of an enjoyable activity. (-)
16. I prefer complex problems to simple problems.
17. Thinking hard and for a long time about something gives me little satisfaction. (-)
18. I enjoy thinking in abstract terms.
19. Knowing the answer without having to understand the reasoning behind it is good
enough for me. (-)
20. Learning new ways to think would be very appealing to me.
26
PMPI (Perceived Modes of Processing Inventory, Burns & D’Zurilla, 1999)
Scoring: calculate the mean of all items.
PMPI Automatic Processing
1. I am often aware of how to cope with a stressful situation even be fore I review all its
aspects.
2. If an approach works I use it again and again so I don’t have to come up with a new
one for each stressful situation I face.
3. I’ve had enough experience to just know what I need to do to cope most of the time
without trying to figure it out every time.
4. The right way to cope usually comes to mind almost immediately.
5. I typically figure out the way to cope swiftly.
6. I quickly do the right thing when coping because I’ve often faced almost the same
thing before.
7. Most of the time, I use the same method to cope.
8. I rely mostly on my past experience to find a way to cope.
9. I rarely need to mull things over; how to cope usually becomes quickly apparent.
10. When a stressful situation occurs I know right away what I need to do to cope with it.
27
CoSI (Cognitive Style Indicator, Cools & Van den Broeck, 2007)
Scoring: calculate the means of the items of each scale.
Knowing Style
1. I want to have a full understanding of all problems
2. I like to analyze problems
3. I make detailed analysis
4. I study each problem until I understand the underlying logic
Planning Style
1. Developing a clear plan is very important to me
2. I always want to know what should be done when
3. I like detailed action plans
4. I prefer clear structures to do my job
5. I prefer well-prepared meetings with a clear agenda and strict time management
6. I make definite engagements, and I follow up meticulously
7. A good task is a well-prepared task
28
Appendix B: Still Shots From
Calexico and Otay Mesa Border
Crossings
29
30
31
32
33
Appendix C: Examples of
Preliminary VWE Mock-Ups
34
35
36
37
38
39
Appendix D: Alpha Test Version
Mock-ups
40
41
42
43
44