Crisis managers making decisions in the
near real-life training environment of
PANDORA: evaluation framework for
human performance and user experience
Michele CORNACCHIA1, Filomena PAPA1, Stefano LIVI2,
Bartolomeo SAPIO1, Enrico NICOLO’1, Keith STRICKLAND3
1
Fondazione Ugo Bordoni, Viale del Policlinico 147, Rome, 00161, Italy
Tel: +39 06 54801, Fax: + 39 06 54804406,
Email: [email protected], [email protected], [email protected], [email protected]
2
Sapienza Università di Roma – Dipartimento di Psicologia dei processi di sviluppo e
socializzazione, Via dei Marsi 78, Rome, 00185, Italy
Tel: +39 06 49917967, Fax: + 39 06 49917652, Email: [email protected]
3
Emergency Planning College (Cabinet Office), The Hawkhills, York YO61 3EG, UK
Tel: +44 (0)1347 825027 Fax: + 44 (0)1347 822575
Email: [email protected]
Abstract: This paper describes the general evaluation framework to be adopted in
the FP7 Project Pandora, a near real-life training simulated based environment for
learning activities suitable for crisis scenarios. Pandora provides its end-users with
much technological support to promote both single organisation and multiple
organisation collaborative strategic decision making in response to a crisis. It takes
into consideration emotional responses and provides significant stimuli to exert the
decision makers capability to recognise, analyse and correctly assess a developing
crisis simulation. User performance and acceptance of technology will be modelled
within the evaluation for different levels of end-users; namely, single trainees,
groups of trainees and indeed the trainer. In particular, the operating environment
within the simulation will exert certain conditions upon the participants including
critical time pressure and/or difficulties in information processing, group and
collaborative decision making which is approached in terms of cognitive closure and
the somewhat irrational mechanisms in play to eliminate ambiguity in order to reach
definite conclusions.
1. Introduction
Pandora is the FP7-ICT-2007-1-225387 project winner for the particular mixed call on ICT
and Security, co-funded by the European Union under the Security Programme and
monitored by REA (Research Executive Agency). It started in January 2010 and will end
on 31 December 2011 [1]. Pandora is a simulated based training system for crisis managers
that takes full account of human behaviour. It is a matter of fact that, when a catastrophic
event occurs (e.g. extreme weather including hurricanes, floods drought etc., fire,
explosions, earthquake, disease, deliberate acts, etc.), human reactions and the exactness of
key decisions are often the most relevant factors in determining the efficacy of the response
in order to prevent an emergency from becoming a disaster. Crisis situations occur
frequently and they invariably require a multi-agency response [2], as impacts and
consequences are usually wide ranging and are a risk to human health and life. Effective
management of a crisis is therefore a major issue and the training of strategic decision
makers is therefore an important preparation to ensure a well managed response [3].
Currently there are two main methods for delivering strategic training [4]: table-top
exercises and real world field exercises exercise. A table-top exercises is low cost and is
relatively easily organised, but it cannot recreate the real atmosphere (e.g. stress, anxiety,
confusion) and the pressure of making critical decisions in short timescales. On the other
hand, simulation exercises in the field can be very effective, but such activity is hugely
expensive in both time and resources and is not easily or quickly organised.
Pandora is designed to bridge the gap between table-top exercises and real-world
simulation exercises, providing a near-real training environment at affordable cost in a
controlled environment. The new training environment is provided in three deployment
modes of training to provide single site, deployable and remote distribution for strategic
level crisis managers (not tactical or operational), to exercise under simulated stressful
conditions [5]. This paper focuses only on the first mode as that gives a clear picture of the
capabilities that are invested within the Pandora product. It also provides a comprehensive
ICT solution, based significantly upon the analysis of both the end-users needs (trainees and
trainer) and the learning/didactic objectives for strategic decision making training. In this
respect, the evaluation framework proposed below aims to verify whether or not human
performance and individual/group decision making in a simulated scenario of crisis is
improved and to what extent. Moreover, in respect of the pressurised training environment,
where information may not be complete and decisions are still required and quickly, group
decision making is approached in terms of cognitive closure, and the mechanisms
(sometimes irrational or subjective) used to eliminate ambiguity and reach definite
conclusions.
1. User Training Needs
The user requirements have been determined from research conducted by the UK
Government with over 300 senior emergency response practitioners [4]. These user
requirements have subsequently informed a system requirement document that details the
functional capabilities that Pandora needs to encompass, with accompanying justification
and prioritisation of individual attributes, in order to meet the user training requirements.
1.1 – Multi-Agency Working
Major Incidents/Emergencies are multi-agency by definition as there is unlikely to be
any single agency that has all of the skills, knowledge and resources necessary to provide
an effective response. Consequently, there is a need to promote collaborative working to
deliver a combined and co-ordinated response that will unify the expertise and resources
available across a wide partnership of organisations.
1.2 – User Training Needs



The main requirements that emerged from the user needs analysis are:
to collectively train, practice and exercise strategic emergency management decision
making across a range of different conditions to optimise both individual and group
decision making processes;
to undertake individual and collective training across all agencies that have a stake in
emergency management (Public and Private Sectors);
to undertake training to develop an in depth understanding of individual and group
decision making behaviours;






to conduct training that presents possible courses of action for consideration and
selection by the decision maker(s);
to conduct training that stimulates decision makers to develop options for consideration
and subsequently select a preferred course of action with supportable justification;
to provide training that will take account of scenarios contained in risk assessments at
the local, regional and national levels;
to undertake training in conditions that reflect as far as possible the real operating
environment and the stress and dynamic tensions induced by the realism of the scenario;
to develop confidence among individuals to adopt different decision making models
under emergency response conditions;
to collect objective training event data to support debriefing and wider learning
processes.
2. Pandora Solution
Pandora builds its solution by developing a system capable of providing the following
crucial functionalities:
 Selection, modelling and monitoring the relevant human factors or psychological
variables (e.g. personality traits, leadership style, background experience) that have
been shown to have an influence in decision making under stress.
 Modelling the trainees’ actual behaviour/status with respect to the selected variables.
 Personalisation of the training stimuli according to the trainees model.
 Provision of appropriate metrics on the performance of an individual actively engaged
in the strategic management of a crisis.
2.1 – The Pandora crisis training environment
The environment makes available a realistic and complete scenario with near real-time
actions, coherent with those expected in a real-world situation. Realistic emotional statuses,
through affective inputs (emotional statuses: fear, irritation, joy) and stress factors (pressure
to make a decision, fragmentary information, short timescales for decisions to be made) are
represented to different crisis managers representing different organisations but working
towards a common purpose [6].
The main innovative aspects are in the realism and complexity of the simulation model.
Pandora can analyse behavioural aspects and report them during the training, it can model
the crisis scenario and its evolution; create the user experience up to the reproduction of
emotional status; create multiple interconnected training procedures; represent
consequences within the simulation of trainees decisions.
2.2 – The architectural system
The architectural system is composed of 4 main modules. The Behaviour Simulation
and Modelling [7] module aims to create and update a profile for each trainee capturing
their features and actual behaviour as well as providing indications on how to personalize
the training for individual trainees.
The Crisis Simulation and Modelling [8] module combines the information coming
from the previous sub-system and the knowledge about the critical infrastructure domain in
order to create the crisis scenario evolution that is to be simulated. The Environment and
Emotion Simulation [9] module aims to represent effectively the Non–Player Characters
and environments within the simulation. Lastly, the Crisis Room [5] module is intended as
the environment where the training event takes place.
This deployable mode in a fixed site is due to be set up at the UK Emergency Planning
College outside York, England, in a standard training room, in accordance with the basic
end-users requirement and utilising “severe weather” crisis simulation scenario.
2.3 – Training Task
A group of trainees representing from public authorities and private agencies (e.g. Civil
Protection, Transport, Telecommunication, Police, Hospitals, etc.) will participate in
training sessions. In the event that certain organizations are unable to be represented such
entities will be simulated by the system (Non Players Characters). Each trainee, before
participating in the training session, will be required to provide personal information and
details in order to create personal profiles for each trainee (Pandora Kernel) covering their
competences, attitudes and behavioral style (Behavioral Model). With these primary
acquisitions the system will subsequently be capable of addressing the need to provide
training that is personalized (Behavioral Planner). The real training process is contrarily
created by a second module (Crisis Planner) which exploits either individual information
retrieved from the trainee profile and or the knowledge base (Crisis Knowledge) within
which the component of the working scenario reside [10].
In this way, Pandora will plan the training session dynamically by choosing, in relation
with both the crisis progression and the reactions of each trainee, the most proper sequence
of goals and required tasks to submit to the group or the general information that they will
be given to share appropriately between themselves. A further important module (Emotion
and Environment Synthesizer) contributes in parallel to the training plan by taking control
of the primary stimuli modulation function and stress level generator. An additional module
(Trainer Support Framework) provides the trainer with the opportunity to monitor the
temporal evolution of the session, i.e. the behavioral analysis of each trainee. The trainer
can control in real time the didactic activities of both group and single individuals, for
example, by intervening on their decision path by inserting new or conflicting information
about the situation under discussion by the trainees. This capability allows the trainer to use
judgment and experience to extract the best learning outcomes from the trainees.
3. Relevant psychological factors in decision making for complex
scenarios
As described in the previous sections, Pandora simulates and reproduces crisis scenarios for
learning activities where individuals and individuals within a group are highly involved.
Thus, in such stressful and complex environments, psycho-social factors are highlighted
requiring individuals to be trained to quickly and accurately accomplish their own tasks,
and to efficiently integrate and coordinate their actions in stressful situations with other
trainees [11]. In particular, the deleterious effects of stress on human performance are well
documented and have been a focus of research in the psychological sciences although, even
in simulation-based training, converging more on individual than group performance in
decision making [12, 13]. In this case we use the term stress to indicate a process by which
environmental demands (in our case time pressure during a threatening event) evoke a
process in which perceived demand exceeds resources and that, in turn, results in
undesirable physiological, psychological, behavioural, or social outcomes [14]. As a result,
both individual and group level characteristics, processes and outcomes are involved,
although some environmental characteristics will also be taken into account particularly in
extreme training environments such as Pandora. In fact, Pandora, as a high-stress
environment, includes specific task conditions (such as time pressure, uncertainty, high
cognitive load and so on) that require specific responses such as the flexibility to adapt to
novel and changing environmental contingencies [15].
At an individual level, stress influences both the psychological ability (e.g. memory,
attention) and emotional and motivational ability, in that it reduces motivation and
consequently the need for cognitive closure. This motivation indicates a tendency or an
inclination, rather than a cognitive deficit, that is function of an individual weight of costs
and benefits (or a lack) of a closure of the process by which people acquire and thus grasp
information from the environment [16]. The need for closure has relevant implications both
at individual and also group level [17, 18, 19] were decision making processes are involved.
The antecedents of the motivation towards cognitive closure can be found in those
circumstances that highlight the perceived benefits and reduce the perceived costs of
closure (i.e., situational uncertainty, time pressure, environmental noise, mental fatigue,
boredom of a cognitive task). By contrast, the need to avoid closure may be instilled in
those conditions that stress the costs of closure and the benefits of a lack of closure (i.e.
accountability, fear of invalidity, evaluation apprehension). Moreover, the need for closure
can also represent a stable individual disposition where at one polar extremity individuals
may display a systematic proclivity to value closure positively, while, at the other pole,
individuals may be predisposed to avoid closure in order to keep the situation open [17].
The need for closure [16] has widely ramifying consequences for social-cognitive
phenomena at the intrapersonal, interpersonal and group levels of analysis [20]. Specialists
in the matter identify two general tendencies: urgency and permanence. The urgency
tendency represents an attitude to attain closure as soon as possible, in this case any other
postponement of closure is experienced as troublesome. On the other side, the permanence
tendency is the individual inclination to maintain the decision making process as long as
possible, it is prevalent in the desire to perpetuate closure, giving rise to “freezing” past
knowledge and safeguarding future knowledge.
These basic considerations lead us to identify a first view centred on the “need for
closure” as depicted in Figure 1, where circumstances are the physical, environmental and
psychological states, that increase the difficulties of information processing.
Figure 1: Antecedents and consequences of need for cognitive closure.
The motivation of cognitive closure may affect the way that individuals process
information towards the formation, alteration or dissolution of knowledge. Such processes
are typically embedded in a social-interaction context and they may affect significantly the
way a person thinks about, feels about, acts towards, and even talks about others.
At a group level, decision making under stress becomes more centralized so that control
and decision making is concentrated at higher levels of the group [12]. This centralization
of authority in groups operating under stress, and the increased reliance on specific leaders
to make decisions, is seen as an adaptive response to threat and uncertainty as it places
responsibility in the hands of those that appear to better represent the group as a whole.
Moreover, on time pressure effects, some behavioural data [21, 22] shows that
individuals are less likely to take action as risk levels increase, and that time pressure
doesn’t have a uniform effect on choice probability. Under time pressure individuals are
more conservative at the lower risk levels but are more prone to take risks at the higher
levels. It means in effect that they take more time to make specific decisions at moderate
levels of risk and reduce their response time in general across all risk levels.
4. The evaluation framework
The performance measurement at individual and group levels, is currently driven by
many theories and they all generally give many perspectives on how to approach the
different playing situations. There are important implications within the Pandora
environment concerning the simulation technique applied in the project [9]. This leads us to
consider those frameworks that concentrate upon learning outcomes, defined as “persistent
states making possible a variety of human performance”. From the training point of view,
these are the steps that have to be negotiated in performing tasks explicitly designed for
learning. Literature provides a further distinction about this in learning categories [13] by
indicating the different implications for the nature of performance as well as the methods
for measurement.
At the level of the individual trainee, the performance theory appropriate for Pandora
measurements in SBT (Simulated Based Training) is the Learning Outcomes Framework.
This is where the outcomes of learning drive the effective performance, and different types
of tasks require different types of learning outcome (cognitive, such as need for cognitive
closure, skill and affective). This requisite in particular works well within Pandora, with a
system conceived to monitor performance measures in an SBT environment (i.e. decision
making under stress and trainee behavioural responses), by addressing a real time analysis
of the learning outcomes involved in the task being performed and including the generation
of corrective feedbacks.
At the level of the group and their processes (effectiveness and performance), the
library of available literature is vast but the performance theory most appropriate [23] for
Pandora, seems to best fit the topics of the IPO (Input, Process, Output) models. The IPO
model seeks to characterize the input variables (i.e. individual and team characteristics), the
process variables (i.e. shared cognition, leadership, communication, coordination, decision
making, and back-up behaviour), and outcome variables (i.e. performance outcomes,
productivity, and satisfaction). By studying the relationships between the IPO variables it is
possible to characterise the group work as dynamic and multidimensional as well as
capturing the group performance [24] during practice activities even in SBT.
The two theories outlined are valuable tools for understanding performance in a
complex training scenario like the one provided by the Pandora system. In order to enhance
these outcomes, the critical aspects of performance must be observed and captured during
end-user practice activities or, in the case of Pandora, specifically during the execution of
those tasks designed to enable the demonstration/evaluation sessions. There are several
methods available to achieve this goal including qualitative to quantitative, subjective (e.g.,
self-reports) to objective (e.g., task outcomes). It is nevertheless important to take into
consideration the individual and group level that categorize performance measures in order
to correlate them with end-user experience and capabilities (trainees level: basic, medium,
advanced) [3].
In general, qualitative methods (e.g., the protocol analysis) seem to be particularly
suited for the Pandora task (e.g., drives performance in near real-life world emulation and
preliminary understanding) as these are easier to arrange and manage in a fixed site
configuration. Conversely, quantitative methods are mostly suited to deep investigation
approach when the drive outcomes have already been identified at both individual and
group levels of performance. Considering that Pandora constitutes a novel approach to
training for complex situations and that it is actually aiming, at least initially, at a
preliminary understanding of “what’s happened?”, it can be concluded that many
methodologies from literature (e.g., Behavioural Observation Scales, Communication
Analysis, Event-based measurement, Structural Knowledge assessment, Automated
Performance Recording) are likewise applicable at individual and group levels of analysis.
The Critical Incident Technique in particular, generally used to elicit behaviours tied to
competencies targeted for training, can be applied without fear of damaging either people or
critical structures.
4.1 – User experience and technology acceptance issues.
As the fixed site deployment of Pandora is hugely ICT equipped and the end-users,
namely trainees and trainer, are required to interact in some manner with a personal console
specifically designed to monitor the scenario’s events/data flow and communicate each
other, further user experience and user acceptance evaluations need to be carried out.
The user experience issues in the literature at least include:
• Usability of Graphical User Interface (GUI), about that the literature defines specific
metrics and techniques in [25, 26, 27].
• Perception of realism, namely the feel of being in the real crisis situation. As already
shown in the previous sections, different features of the system are responsible of the
realism perception and as well as they may influence the psychological processes in act.
• Perceived quality of the support provided for the system in use. The use of a new
technology even though tacitly implies the acquisition of new skills, for the trainer to
redesign and deliver the lesson, for the trainees to correctly use the system [28].
As for technology acceptance, the literature points out the UTAUT (Unified Theory of
Acceptance and Use of Technology) model [29], the four main constructs of which are
recognised to play a significant role as direct determinants of the use behaviour, especially
in the evaluation of contexts highly integrated by emerging ICT [30]:
• Performance Expectancy, to which extent an individual believes that using the system
will help him or her to attain gains in job performance.
• Effort Expectancy, easiness associated with use of the system; this is strongly related to
the extent to which a person believes that using the system would be free of effort.
• Social Influence, the impact other people may have on individual perception of utility.
• Facilitating Conditions, about technical infrastructures to support the use of the system.
5. Conclusions
The assessment aims to point out the key features of the Pandora complex environment
when viewed through the lens used by the main actors involved, i.e. the trainees and the
trainer. The dimension of both individuals and groups within the process of making
decisions under crisis or emergency scenario is foremost and central in the adopted human
factors approach. The evaluation aims to verify if the underlying hypothesis of the project
that Pandora effectively bridges the gap between table-top exercises and real world
simulation exercises, providing a near-real training environment at an affordable cost, is
validated. If the evaluation proves the concept, then Pandora will serve to increase the
effectiveness of the strategic decision making processes used in response to a crisis or
emergency situation.
References
[1] Pandora project website, http://www.pandoraproject.eu.
[2] Marecki, J., Schurr, N., Tambe, M., Scerri, P., (2009), Analyzing Dangers in Multiagent Rescue Using
DEFACTO, Safety and Security in Multiagent Systems, M. Barley, H. Mouratidis, A. Unruh, D. Spears, P.
Scerri, F. Massacci (Eds), Lecture Notes in Artificial Intelligence, vol. 4324, pp 241 -257.
[3] Cesta, A., Cortellessa, G., De Benedictis, R. and Strickland, K. Training Crisis Managers in Strategic
Decision-Making. In ERCIM News, N. 85:47-48, 2011.
[4] Deliverable D1.1 Part II: User Requirements Analysis and State of the Art, Chap 1.1 Civil Protection
Definitions, http://www.pandoraproject.eu.
[5] Deliverable D5.1: The emulated crisis room blueprint, http://www.pandoraproject.eu.
[6] Sayegh, L., Anthony, W. P., & Perrewé, P. (2004). Managerial decision-making under crisis: The role of
emotion in an intuitive decision process. Human Resource Management Review, 14, 179-199.
[7] Cortellessa, G., D’Amico, R., Pagani,M., Tiberio, L., De Benedictis, R., Bernardi, G., Cesta, A.: Modeling
Users of Crisis Training Environments by Integrating Psychological and Physiological Data. In: IEA/AIE-1.
Proceedings of the Twenty-fourth International Conference on Industrial, Engineering and Other Applications
of Applied Intelligent Systems, Siracuse, NY. Lecture Notes on Computer Science N.6704. 2011.
[8] Deliverable D3.2: Crisis module framework alpha release, http://www.pandoraproject.eu.
[9] Deliverable D4.1: Definition and modelling of the emotion simulation engine including the environment
framework design, http://www.pandoraproject.eu.
[10] Celino, I., Dell'Aglio, D., De Benedictis, R., Grilli, S., Cesta, A. Ontologies, rules and linked data to
support Crisis Managers Training. In IEEE Learning Technology Newsletter,13(1): 3-6, 2011.
[11] Brannick, M. T., Salas, E., & Prince, C. (1997). Team performance assessment and measurement:
Theory, methods, and applications. Hillsdale, NJ: Erlbaum.
[12] Driskell, J. E., & Salas, E. (1991). Group decision making under stress. Journal of Applied Psychology,
76(3), 473-478.
[13] Salas, E., Rosen, Michael A., Held, Janet D. and Weissmuller, Johnny J., Performance Measurement in
Simulation-Based Training: A Review and Best Practices, Simulation Gaming 2009 40: 328 originally
published online 22 December 2008; http://sag.sagepub.com/content/40/3/328.
[14] Driskell, J. E., & Salas, E. (Eds.). (1996). Stress and human performance. Hillsdale, NJ: Erlbaum
[15] Driskell, J. E., Salas, E., & Johnston, J. (1999). Does stress lead to a loss of team perspective? Group
Dynamics: Theory, Research and Practice, 3(4), 291–302
[16] Kruglanski, A. W., Webster, D. M. (1996). Motivated closing of the mind: “Seizing” and “Freezing”.
Psychological Review, Vol.103, No.2, 263-283.
[17] Pierro, A., Mannetti, L., De Grada, E., Livi, S., Kruglanski, A.W. (2003). Autocracy Bias in Informal
Groups under Need for Closure, Personality and Social Psychology Bulletin, 29 (3), 405-17
[18] Livi, S., Mannetti, L., Pierro, A., Kruglanski, A.W., Kenny, D.A.(2005). Epistemic Motivation and
Groups’ Reaction to Change: Effects of Need for Cognitive Closure on Norms Transmissions in a Laboratory
Micro-Culture. CogSci 2005. http://csjarchive.cogsci.rpi.edu/Proceedings/2005/docs/p1308.pdf
[19] Livi, S., De Grada, E., Pierro, A., Mannetti, L., Kruglanski, A.W., Kenny, D.A. Bisogno di Chiusura e
trasmissione di norme sperimentalmente indotte in piccoli gruppi. Giornale Italiano di Psicologia. 34, 3, 581-601.
[20] Webster, D. M., & Kruglanski, A. W., (1998). Cognitive and social consequences of the needs for
cognitive closure. European review for social psychology, 8, 133-161.
[21] Dror, I., E., Busemeyer, J., R., Basola, B. (1999). Decision making under time pressure : An independent
test of sequential sampling models. Memory & Cognition. 27 (4), 713-725.
[22] Finucane, M.L. (2008). Emotion, affect, and risk communication with older adults: challenges and
opportunities. Journal of Risk Research. Vol.11, No.8, 983-997.
[23] Salas, E., Stagl, K. C., & Burke, C. S. (2004). 25 years of team effectiveness in organizations: Research
themes and emerging needs. International Review of Industrial and Organizational Psychology, 19, 47-91.
[24] Guzzo, R. A., & Dickson, M. W. (1996). Teams in Organizations: Recent Research on Performance and
Effectiveness. Annual Review of Psychology, 47, 307-338.
[25] ISO/DIS 9241-11 (1998), Guidance on Usability. International Organization for Standardization.
[26] ISO 9126-1 (2000), Product quality Part1: Quality model. International Organization for Standardization.
[27] Bevan, N. and Macleod, (1994) Usability measurement in context, Behaviour & Information technology,
13,132—145.
[28] Gnisci A., Papa F., Spedaletti S. (1999), Usability Aspects, Socio-Relational Context and Learning
Performance in the Virtual Classroom: A Laboratory Experiment, Behaviour & Information Technology,
Taylor & Francis 1999,Vol. 18, N.6,pp. 431-443.
[29] Venkatesh, V., Morris, M. G., Davis, G. B., Davis, F. D., (2003). User Acceptance of Information
Technology: Toward a Unified View, MIS Quarterly vol. 27 No. 3, pp. 425-478, September.
[30] Cornacchia, M., Baroncini, V., Livi, S., (2008). Predicting the Influence of Emerging Information and
Communication Technologies on Home Life; J. Filipe and J. Cordeiro (Eds.): WEBIST 2007, Revised
Selected Papers, LNBIP 8, pp. 184-200, Springer-Verlag Berlin Heidelberg, 2008.