Assessing and Managing ICT Risk
with Partial Information
Fabrizio Baiardi∗ , Fabio Corò∗ , Federico Tonelli∗ , Alessandro Bertolini∗ , Roberto Bertolotti∗ , Daniela Pestonesi†
∗ Dipartimento
di Informatica, Università di Pisa, Italia
Ingegneria e Ricerca SpA, Pisa, Italy
Email: [baiardi,fcoro,tonelli]@di.unipi.it, [email protected]
† ENEL
Abstract—Haruspex is a suite of tools to assess and manage
in a probabilistic way the risk posed by an ICT system. The
tools apply a Monte Carlo method to a scenario where some
intelligent threat agents sequentially select and compose attacks
to reach their goals. Some tools build a description of a scenario
with the various agent, the target system, its vulnerabilities and
the attacks they enable. Starting from this description, another
tool applies a Monte Carlo method to simulate step by step the
attacks of each agent. This tool returns a database with samples
on the attacks the agents have implemented, the goal they have
reached and the required time. Further tools use this database to
produce statistics to assess the risk and to select countermeasures
to be deployed. Since several tools require information on both the
agents and the target system, we discuss how to conduct a robust
assessment even when only a partial information is available. To
exemplify the proposed approach, we describe the assessment of
an industrial control system with some SCADA components.
I.
I NTRODUCTION
We consider a quantitative approach to the probabilistic risk assessment and management of an ICT system where intelligent,
goal oriented threat agents compose the elementary attacks
enabled by system vulnerabilities into sequences of attacks or
complex attacks. A complex attack results in an agent privilege
escalation [1] that may involve distinct nodes and where the
agent exploits the privileges granted by an attack to execute the
following ones in the sequence till it collects all the privileges
in a goal.
To evaluate the resulting risk, we consider a scenario
based assessment where each scenario describes the target
system and some agents. By considering how agents select and
implement attacks in each scenario, this approach increases the
accuracy of the assessment but the complexity of estimating
the probability that agents reach their goals is usually unmanageable. As an example, the complex attacks each agent selects
and implements strongly depends not only on its goals but also
on the information on the system it can access and it considers.
If an agent has access to any information on the target system
and fully exploits it, then it will implement a plan, e.g. one of
the shortest complex attacks to reach a goal. Otherwise, some
of the attacks the agent implement may be useless to reach a
goal.
To overcome the previous problems, we adopt a Monte
Carlo method that runs, for each scenario, multiple simulations
of how the agents select and implement attacks in the same
time interval. Each simulation collects samples of the events
of interest to compute statistics to support the assessment.
To partially automate the assessment, we have developed the
Haruspex suite, an integrated set of tools that:
•
builds the description of the target system and of the
agents in a scenario,
•
estimates some parameters of the Monte Carlo method
such as the time interval to be simulated,
•
applies the Monte Carlo method to a description,
•
uses the method outputs to select countermeasures.
After presenting the suite, we discuss how to conduct a
robust assessment and select countermeasures in spite of partial
information on the target system or on the agents.
This paper is structured as follows. Sect. II briefly reviews
related works on vulnerabilities, attacks, agents and their
simulation. Sect. III presents the tools to support the Monte
Carlo method. Sect. IV discusses the handling of uncertainty
on the tool inputs. Sect. V introduces the suite tools that select
countermeasures to minimize the agent success probability.
Sect. VI applies this suite to assess an ICS that manages a
power generation plant. Lastly, we draw some conclusions and
outline future work.
Some tools of the suite have been described in [2], [3],
[4] that only consider the building of a description and the
application of the Monte Carlo method. These papers do not
describe the full suite, the tools to manage the risk and the
handling of partial information on some inputs.
II.
R ELATED W ORKS
We briefly review the main related works on complex
attacks, plans, their description, the evaluation and the management of the corresponding risk.
[5], [6], [7], [8], [9], [10] analyze the simulation of attacks
against ICT or critical infrastructures. In particular, [10] discusses the modeling of agents with partial information and [11]
introduces an alternative definition of look-ahead. Intelligent,
goal oriented agents have been analyzed with reference to
terrorism [12], [13]. [14] presents a formal model of plans
similar to the one we propose. [15] describes the prerequisites
and the effects of attacks and pairs each attack with the proper
countermeasures.
[16], [17] analyze the simulation of agent attacks in the
framework of game theory. With respect to this approach,
the Haruspex suite assumes an assessment aims to reduce the
probability that an agent reaches a goal rather than to divert it
Table I.
S
c
op
ag
res(ag)
g
at
v
v(at)
pre(at)
res(at)
post(at)
succ(at)
λ(ag)
sq(i)
hisu
L IST OF A BBREVIATIONS
A. Describing a scenario
the target of the assessment
a component of S
an operation of a component
a threat agent
the resources ag can access
a goal of an agent
an elementary attack
a vulnerability
the vulnerabilities enabling at
the rights to execute at
the resources to execute at
the rights granted if at succeeds
the success probability of at
the look-ahead of ag
the i-th attack ag executes
acceptable success probability
An Haruspex scenario describes S, the system to be
assessed, and some intelligent threat agents, or agents. The
description is modular as it does not require information on
the overall behavior of S or of the agents.
1) System Components: S is seen as a set of interconnected
components, e.g. hardware/software modules. Each component
is described in terms of the operations the other components
or the user of S can invoke, the component vulnerabilities and
the elementary attacks, or attacks, they enable. To describe
social engineering attacks such as spear phishing, users of S
are modeled as components as well [40]. The security policy
of S defines the rights of both users and components. Each
right enables the owner to invoke one operation.
to a distinct target [18], [19]. As a consequence, an assessment
may consider several scenarios to compute the probability that
some agents reach their goals in each of them.
2) Vulnerabilities: Each component is affected by some
vulnerabilities. A vulnerability is a defect in a component or
an erroneous or malicious behavior of some users [41], [42],
[43] that enable some elementary attacks that grant some rights
to the agent that successfully implement them. The Haruspex
suite supports both effective and potential vulnerabilities. The
former has already been discovered, while the latter is suspected only and it is paired with the probability distribution of
being discovered at a given time.
[20], [21] survey agent-based simulation, whereas [22],
[23] use agent based simulation to discover dependencies
among components of a critical infrastructure. [24] discusses
the evaluation of ICT risk, while [25] reviews the problems
posed by a Delphi approach. [26] describes alternative approaches to evaluate the risk due to software systems. [27],
[28], [29] review multi objective optimization that underlies
agent selection strategies.
3) Elementary Attacks: An elementary attack, or attack, at,
consists of some actions and it is paired with some attributes
[2]:
Besides [4], other works have already discussed agent plans
and complex attacks but most of them do not support their
automated discovery. As an example, the taxonomy in [30]
is focused on a series of use case events. [31] introduces a
classification of vulnerabilities that maps each vulnerability
into just one class. [32] analyzes attack plans involving distinct
nodes and it considers the compromised level of a node. The
resulting solution is rather efficient but it cannot discover all
the plans. [33] defines a language to model scenarios. [34]
introduces an algorithm to detect stepping stones. [35] specifies
M2D2, a formal data model to correlate IDS alerts. [30]
computes the success probability of a complex attack without
considering its relation with the agent selection strategy. With
reference to attack graphs, [36], [37], [38] discuss several
approaches to discover plans and select the vulnerabilities to
patch.
III.
•
v(at), the component vulnerabilites that enable at,
•
succ(at), the probability that at is successful,
•
pre(at), the rights to execute the actions of at,
•
res(at), the resources to execute the actions of at,
•
post(at), the rights acquired if at is successful,
•
time(at), the time to execute the actions of at.
Haruspex does not define the actions of at and introduces
succ(at) and time(at) to model their complexity.
An agent can implement at if it can access the resources
in res(at) and owns the rights in pre(at) that it may acquire through previous attacks. If any vulnerability in v(at)
is effective, at is enabled and succeeds with a probability
succ(at), otherwise at fails. In the following, we focus on offthe-shelf components. Their effective vulnerabilities and the
corresponding descriptions are deduced from a vulnerability
database such as the Common Vulnerability Enumeration [42].
[2] describes in more details attack attributes.
T HE H ARUSPEX SUITE
This section briefly describes the kernel of the suite,
namely the tools that, respectively, build the description of a
scenario and apply the Monte Carlo method to this description
and simulate the agent attacks. Other tools will be described
in the following. Here and in the following, for the sake
of brevity, assessment is a synonymous for probabilistic risk
assessment, right is a shorthand for access right and Haruspex
denotes the unifying framework underlying the suite. We adopt
a rather general definition of risk [39]:
4) Threat Agents: Haruspex assumes that a threat agent ag
is a user of S. No generality is lost by this assumption because
the security policy of S can entitle ag even to an empty set
of rights. The description of ag includes the resources it has
available for its attacks, its initial rights, e.g. the operations
ag can legally invoke, and the goals it aims to reach. Any
goal g is a set of rights that ag reaches after acquiring all the
corresponding rights. g is paired with an impact, i.e. a loss that
the owner of S pays per each unit of time ag owns the rights
in g. While the security policy forbids ag to acquire each right
in g, there is an impact only when, and if, ag reaches g.
"Risk is the result of a threat with adverse
effects to a vulnerable system. It can be quantified
in terms of the loss for the system owner and of the
probability that the loss occurs."
Table I defines the abbreviations used in the following.
2
example, the builder description cannot consider the vulnerabilities of a word processor if the scanning cannot discover
such a processor. Furthermore, the builder constrains an agent
privilege escalations by assuming that only an agent that fully
controls, e.g. is an administrator of, a node can launch an
attack from it. A further limitation concerns social engineering
attacks because while users of S can be modeled as further
components, the corresponding vulnerabilities and attacks have
to be manually inserted into a builder description.
5) Selection Strategies: To reach its goals, ag selects and
implements complex attacks according the rights it currently
owns and the known vulnerabilities in the components of S.
After any successful attack, ag applies its selection strategy to
discover the best complex attack to reach its goals and then
it implements the first attack of the one this strategy returns.
To minimize its effort to reach a goal, ag aims to implement
a plan, e.g. one of the shortest complex attacks reach a goal.
However, the complex attack, e.g. the sequence of elementary
attacks, ag implements is a plan only if the selection strategy
of ag can access complete information on S and properly
exploits it. Otherwise, ag may execute an attack that increases
its current rights but it is useless because ag does not exploit
the rights it grants to reach a goal.
C. Describing Threat Agents
By interacting with the descriptor, the user describes the
agents in the various scenarios. The user defines the initial
rights and the goal of any agent ag, the resources it has
available, its selection strategy and λ(ag).
The lookahead of ag, λ(ag) is the parameter that determines how the selection strategy of ag exploits the information
on S it has available. In fact, to select a complex attack, this
strategy ranks those with, at most, λ(ag) attacks and such that
ag owns the rights in the pre condition of their first attack.
If no such attack exists, ag is idle till the discovery of some
potential vulnerability. If any ranked attack grants the rights
in a goal, then it is returned. Instead, if, due to a low λ(ag),
the strategy cannot determine a complex attacks that leads to
a goal, then it selects the one to return according to some of
its attributes. In this case, ag may implement a useless attack.
As a counterpart, we show in the following that larger values
of λ(ag) may slow down the selection.
Currently, the suite supports the following strategies:
1)
2)
3)
4)
random: selects, with the same probability, each
elementary attack ag may execute. It may even select
an attack that is not enabled,
maxP rob: selects the complex attack with the highest success probability,
maxIncr: selects the complex attack granting the
largest set of rights,
maxEf f : selects the complex attack with the best
ratio between success probability and execution time.
If λ(ag) = 0 then only the random strategy can be
selected. Otherwise, any of the other strategy may be selected.
B. Describing the Target System
D. Simulation Engine
The builder maps the output of a vulnerability scanning
of S into the description for the tool that applies the Monte
Carlo method.
By applying the Monte Carlo method to a scenario description, the engine executes an experiment that consists of
several independent runs. Each run simulates, for the same
time interval, the agents attacks and the discovery of potential
vulnerabilities. To guarantee run independence, at the end of
a run, the engine re-initializes the state of S and of any agent
and starts a new run. An experiment returns a database with
the samples collected in the various runs.
Initially, the builder classifies the vulnerabilities discovered by scanning the nodes of S to determine the attributes
of each attack at they enable. In particular, the tool computes
pre(at) and post(at) and checks whether at may be remotely
exploited. [4] details how the builder computer the attributes
of each attack.
At each time step of a run, after determining whether some
potential vulnerabilities are discovered, the engine considers
each agents ag, selects a complex attack and executes its
first attack at. If at is enabled, ag will be busy for the next
time(at) steps plus the time of the selection. If at succeeds,
the engine checks if ag has reached a goal and updates the
impact due to ag. ag repeats a failed attack for, at most, nr(ag)
times before selecting another complex attack. nr(ag) is a
further parameter the user defines when describing ag.
To simplify the implementation of the selection strategy,
the builder considers how global attacks can compose elementary ones. To this purpose, it assumes that each complex
attack results in a global escalation, e.g. in a sequence of
privilege escalations each involving a distinct node of S. For
any agent ag, this sequence starts in node ni such that ag
owns some legal rights on the components in ni and it ends in
a node nf with some components ag aims to control, i.e. some
rights in a goal of ag are related to components running on
nf . ag attacks the intermediate nodes in the sequence because
ni cannot directly interact with, e.g. attack, nf due to some
constrains in the connection topology. The builder description
includes the privilege escalations for all nodes and how they
can be interleaved into a global one.
To model the time to discover information about S and
to select an attack, the engine assumes that ag runs a
vulnerability scanning of a node n, the first time its selection
strategy ranks a complex attack where an attack is enabled
by a vulnerability of a component c running on n. The time
of scanning depends upon n. This shows that larger values of
λ(ag) result both in a better visibility and in a slower selection
due to a larger number of scannings. To model insiders, ag may
be paired with the nodes its already knows.
It is worth noticing that, with respect to a description
the other tools can handle, the builder only considers the
information returned by a vulnerability scanning of the nodes
of S. This implies that the operations, the rights in the goals
and in attack pre and post conditions and the vulnerabilities
are those in the components this scanning can discover. As an
The engine inserts into the database samples with information on the attacks the agents have implemented in a run, the
3
favor of the agents in the scout run. If the bound the scout
returns is larger than the one of interest for an assessment, the
scout run may suffice to deduce that ag cannot reach a goal
in the time interval of interest.
goals they have reached and the time this has taken. Since each
run contributes with exactly one sample to the database, the
confidence level of a statistics computed through these samples
depends upon the number of runs in an experiment. The user
can specify either this number or the confidence level for some
predefined statistics. In the latter case, the engine starts a new
run until reaching the confidence level.
IV.
V.
M ANAGING THE R ISK
This section describes the planner and the manager, the
tools that drive the selection of countermeasure to minimize
the risk in a scenario. In the following, we assume that for
each attack there is a countermeasure to decrease its success
probability. As an example, the patching of a vulnerability
results in the failure of each elementary attack it enables, while
a longer password or a longer encryption key decrease the
probability the agent guess them.
I NCREASING THE ROBUSTNESS OF AN A SSESSMENTS
The consistency of the samples the engine returns may be
strongly reduced if partial information is available on some
agents, on the component vulnerabilities or on the attacks
they enable. In this case, the user can only supply a set of
values for a parameter of the agent or of the target system.
We discuss how to handle this critical issue to prevent that it
reduces the accuracy of the assessment. At first, we consider
partial information about agents and then about the target
system. Lastly, we introduce the scout, a tool that simplifies
the selection of the interval of time to be simulated.
Here and in the following, for the sake of simplicity,
we assume that a scenario includes one agent ag with one
goal g. Extensions to several agents and/or several goals are
straightforward.
Even if only a set of values is known for some parameters
of ag, an assessment can analyze the behavior of ag through
an experiment that considers several agents, the derived agents
of ag. There is a distinct derived agent for each element
in the Cartesian product of the sets for the partially defined
parameters of ag. Suppose, as an example, that the assessment
is in doubt about three selection strategies and two values of
λ(ag). To analyze ag, six derived agents are simulated, one for
each of the six combinations of a strategy and a value of λ(ag).
As an example, the assessment can deduce that ag cannot
achieve g only if no derived agents achieves g. Otherwise,
the assessment can evaluate the risk in a scenario in terms of
the number of derived agents that achieve g and of the time
they take.
A. Discovering Agent Plans
The planner analyzes the database returned by the engine
to discover the plans ag has executed to achieve g and
their success probabilities. In general, these plans are not
known before an engine experiment. Since the engine only
returns the complex attack, e.g. the sequence of attacks, ag
executes in a run, the planner removes from this sequence any
useless attack. By focusing the selection of countermeasures
on the plans rather than on the complex attacks, we avoid
the deployment of countermeasure for useless attacks. This
increases the cost effectiveness of the countermeasures that
are selected.
The planner maps a sequence sq with n attacks into the
plan p(sq, g) with the attacks of sq useful to achieve g. To
describe the computation of p(sq, g), let sq(i), i ∈ [1..n] be
the i-th attack of sq and tp(sq, g) a set that is initialized with
sq(n), the last attack of sq. usef ul(i) is the set with all and
only the rights ag should own before executing sq(i) to achieve
g at the end of s. As an example,
To minimize the computational overhead, we simulate all
the derived agents in a single experiment by exploiting the lack
of conflicts among these agents, i.e. an agent selection does
not constrain those of the other ones.
The handling of uncertainty about S is more computationally expensive because, when in doubt about some parameters,
the assessment runs an experiment for each value of the
parameters and it considers the worst case.
usef ul(n) = pre(sq(n)) ∪ (g − post(sq(n)))
e.g. before executing the last attack of sq ag should own the
rights in the pre condition of this attack and those in g that the
attack does not grant. The algorithm scans sq backward from
n − 1 to 1 and does not insert sq(j) into tp(sq, g) if
A last important parameter is the length t of the time
interval that each run simulates. The scout is a tool to estimate
a lower bound on t and it is inspired by those simulation
techniques, e.g. importance sampling, that favor the occurrence
of rare events [44]. The scout simulates the behavior of the
various agents in one run where it offers to each agent the
best opportunities to reach a goal because all the attacks are
successful and all the suspected vulnerability have been discovered. Furthermore, the scout selection strategy favors those
complex attacks that leave the largest number of opportunities
open, e.g. they result in the largest number of attacks that may
be selected in the next steps. Notice that, even in the scout
run, ag may select some useless attacks. However, ag may
select the same attacks in an engine run. The scout returns a
lower bound on the time it takes some agents to reach a goal.
From this bound, the assessment can deduce the value of t.
1)
2)
no right in post(sq(j)) belongs to usef ul(j),
ag owns the rights in the intersection between
post(sq(j)) and usef ul(j) even before executing
sq(j), i.e. these rights belong to the union of the
initial rights of ag and of the post conditions of
sq(1), ..., sq(j − 1).
If sq(j) is useful, we add it to tp(sq, g) and compute
usef ul(j − 1) by removing from usef ul(j) the rights in
post(sq(j)) and by adding those in pre(sq(j))
usef ul(j − 1) = (usef ul(j) − post(sq(j))) ∪ pre(sq(j))
Since we do not compute the statistics to assess S through
the output of the scout, we do not need to remove the bias in
If sq(j) is useless, usef ul(j − 1) = usef ul(j).
4
The manager selects plans of Epi to be inserted into
Cpi according to their success probability, starting from those
with the largest one. If several plans have the same success
probability, it selects at random those to insert. It is worth
noticing that in a correct solution Cpi only includes the plans
with a success probability larger than hisu. In fact, given a
set of plans where each plan has a success probability lower
than hisu but where the sum of these probabilities is larger
than hisu, the selection of the plans to be affected by countermeasures is rather arbitrarily because some success probability
may strongly increase after deploying some countermeasures.
An alternative solution considers all the plan subsets with an
overall success probability larger than hisu but this requires
more than one experiment in each iteration. As a compromise,
we bound the size of Cpi through a threshold defined as a
percentage of successful plans. This guarantees that the success
probability of ag decreases at each iteration and can handle
the case where ag implements a large number of plan, each
with a very low success probability.
At the end of the backward scanning, p(sq, g) = tp(sq, g).
p(sq, g) is the correct plan provided that ag never implements an attack that does not increase its current rights.
In some cases, the planner cannot discover the correct plan
because sq interleaves several alternative plans. In this case,
it shoulds map into a plan any permutation of sq that is a
complex attack, e.g. any permutation where each attack is
enabled by the rights granted by those executed before it.
After discovering the plans ag has executed, the planner
computes the success probability of each plan p as the ratio
between the number of runs where p has been implemented
and the number of runs in the experiment. This considers that
distinct complex attacks may corresponds to the same plan.
B. Selecting Plans to be Affected
Starting from the plans returned by the planner, the
manager determines the countermeasures to reduce the risk
posed by S. To this purpose, an input parameter, hisu, defines
the highest success probability of ag that the assessment
accepts.
C. Selecting Countermeasures for a Set of Plans
Since a countermeasure for at affects all the plans that
share at, the selection of countermeasures for plans in Sp is
focused on the attacks these plans share to reduce both the
number of countermeasures and their overall cost.
The manager has an iterative behavior. After selecting, as
described in the following, some countermeasures, it updates
the description of S to model their deployment and runs
another experiment with the new version of S to discover
how ag reacts to countermeasure. In other words, the new
experiment discovers whether, when attacking S, ag neglects
some plans it can successfully implement against the new
version of S because they have not been affected by the
deployed countermeasures. Any of these plans is denoted as
a dependent one because ag selects it only when other ones
are affected by countermeasures. Only a new experiment can
discover dependent plans because we cannot deduce them
from the output of an experiment where ag neglects them
in favor of some other plans. If the new experiment shows
that some dependent plans still have a success probability
larger than hisu, the manager selects and deploys further
countermeasures, runs another experiment and so on.
To formally define this focus, we pair any plan in Sp with
an index in 1..n where n is the size of Sp and any attack at
with both c(at), the cost of its countermeasure, and the set
Sp(at) = {i1 , ..., ik } of the indexes of the plans that share at.
A set of countermeasures stops all the plans in Sp if it is a
coverage [45] of {1, ..., n} through the elements of
S
Sa = (Sp(at))∀at ∈ Sp
Since we are interested in a coverage with the minimal cost, we
can safely neglect an attack at1 if all the plans that share at1
also share a distinct attack at2 with a cheaper countermeasure.
After removing any attack that can be neglected, the manager
considers any possible coverage and selects the cheapest one.
Given the number of plans and of the attacks they share,
the time to compute a coverage acceptable execution time even
if the problem is NP complete. As a matter of fact, if both the
previous numbers are very large, then there is a large number
of alternative plans that ag can select to attack S. This requires
a redesign of S rather than a few countermeasures.
Two features that strongly influence the computational
overhead and the effectiveness of the countermeasures the
manager returns are how it determines, at the i − th step,
the set Si with the plans to affected by the countermeasures
and the selection of the countermeasures for plans in Si .
In a global approach, Si depends upon the plans ag has
executed in any previous iteration. Then, the manager selects
some attacks from these plans and deploys the corresponding
countermeasures. Since the countermeasures the i − th step
selects depend upon Si only, they may completely differ from
those selected in a previous step. Instead, in the incremental
approach, the manager only considers the dependent plans
discovered in an iteration and it extends the previous set of
the countermeasures with those for these plans.
To optimize the execution time, the manager records the
current optimal cost and it aborts the building of a coverage
as soon its cost is larger than the current optimum.
By considering, at each iteration, all the plans to be
stopped, an iterative approach exploits at best the shared
attacks to minimize the number of countermeasures. Instead,
an incremental approach cannot fully exploit shared attacks
because it does not known the dependent plans selected in the
next iterations and the attacks they share with those actually
considered.
Currently, the manager adopts a global approach where Si
is the union of S1 , ..., Si−1 , the set of plans the manager has
considered in the previous iterations, and of Cpi . Given the set
Epi with all the plans ag executes in the i−th experiment, Cpi
is a subset of Epi such that the sum of the success probability
of the plans in Epi − Cpi is lower than hisu.
To examplify our solution, suppose that ag can implement
any of {at1 , ..., at7 } and Sp includes these plans:
p1 ≡ at1 ; at2 ; at3
5
p2 ≡ at2 ; at4 ; at5
p3 ≡ at5 ; at4 ; at6
p4 ≡ at1 ; at6 ; at7
Each attack is paired with a set of indexes as follows:
at1 → {1, 4}
at2 → {1, 2}
at3 → {1}
at4 → {2, 3}
at5 → {2, 3}
at6 → {3, 4}
at7 → {4}
If c(ati ) < c(at(i+1) ), ∀i ∈ 1..6, we can neglect a5 because
its countermeasure has a cost larger than the one of the
countermeasure for a4 . The following sets of attacks cover
all the plans:
{at1 , at4 }
{at2 , at6 }
{at1 , at2 , at6 }
{at2 , at4 , at7 }
{at3 , at4 , at7 }
Figure 1.
control a distinct device. The agents have no information on the
infrastructure and scan each node to discover its vulnerabilities.
To show the handling of partial information, we assume no
further information on the agents is available. Hence, for each
agent, we consider any selection strategy and, when appropriate, two lookahead values, 1 and 2. The latter value strongly
reduces the time it takes the agent to discover a sequence
of attacks to reach its goal. Hence, for each agent, seven
derived agents are simulated. One adopts the random selection
strategy, two the maxIncr, two the maxP rob and two the
maxEf f one. The confidence level of all the experiments is
95% on the components an agent attacks. This has required
from 150.000 to 500.000 runs in an experiment. For the sake
of brevity in the following, st/la agent is a shorthand for the
agent that adopts the st selection strategy with a lookahead
equal to la.
Given the previous cost assumptions, the manager selects one
of the first two sets.
D. Iterations and Plans
Given the iterative behavior of the manager, an assessment does not need to apply the planner. In this case,
the manager discovers in a following iteration that it has
deployed a countermeasure for a useless attack and it can
select another countermeasure. This saves the time to run the
planner at the expense of further manager iterations. The
optimal compromise depends upon both the architecture of S
and λ(ag).
VI.
Network architecture of the experimental IT infrastructure
The Haruspex tools for this assessment run on an highly
parallel IBM system with 96 cores in 16 clusters and have
been carefully optimized to exploit at best the available cores.
As an example, several threads of the builder classify distinct
vulnerabilities, the engine maps distinct runs of an experiments
onto distinct cores and the manager computes alternative
coverages of a set of plans through threads mapped onto
distinct cores. In the following, the execution times refer
to this version of the tools. Furthermore, to analyze how
an agent features influence the time to simulate its attacks,
each experiment simulates just one derived agent. Under these
conditions, the time of an experiment ranges from 30 seconds
to 10 minutes. The worst case is the simulation of the random
agent due to the very long complex attacks it implements.
C ASE S TUDY
We have applied the Haruspex suite to assess an ICT
infrastructure with SCADA components that manages a power
generation plant. It consists of 49 nodes segmented into six
subnetworks as in Fig. 1. The Central Network is used for
some business processes and it includes 24 nodes. The Power
Context is the one with the workstations of the plant operators
and it includes 7 nodes. The Process Network1 and Process
Network2 consist of, respectively, 9 and 7 nodes, the SCADA
servers to control and manage the plant. Each Control Network
includes a PLC device. Three nodes in the Central Network are
connected to the Power Context one. A pair of nodes connects
each Process Network to the Power one. To properly stress the
suite tools, we have manually inserted further vulnerabilities
in the database that the builder returns.
The vulnerability scanning of all the infrastructure nodes
has taken less than one day. The whole infrastructure is affected by 1253 vulnerabilities and the number of vulnerabilities
per host ranges from 2 to 57 with an average slightly larger
than 25. The builder produces the infrastructure description in
less than ten minutes. Most of this time is devoted to store the
description in the output database. We have used the descriptor
to model the various agents to be simulated.
A. Running the Assessment
The assessment considers four agents that initially own
some rights on the Central Network and aim to control the PLC
devices. In particular, one agent aims to control both devices,
one to control at least one device and the last two aim to
6
Even if the target infrastructure is not very complex, we
have applied the scout to estimate the time it takes an agent to
reach a goal. In a scout run an agent reaches a goal through a
complex attack that composes five elementary attacks and this
takes less than 12 hours. Then, a first set of 20 experiments
simulate the various agents with a timelimit of 72 hours for
each run. In this time interval, all the agents reach their goal.
The average time to reach a goal ranges from 11 hours of
the maxEf f /2 agent that aim to control any PLC to the 22
hours of the random agent that aims to control both PLCs.
The number of attacks in a complex one ranges from 15 of
the random agent to 5 for the maxEf f /2 agent. The longest
complex attack of the random agent includes 28 elementary
attacks. Most of these attacks are useless.
C. Selecting Countermeasures to Deploy
We consider now the manager output for two agents,
maxIncr/2 and maxP rob/2, aiming to control at least one
PLC. This case is rather complex because the ability of
selecting the PLC to control largely increases the number of
dependent plans. In this example, each countermeasure is a
patch that removes the corresponding vulnerability and results
in the failure of the corresponding attack.
The countermeasures to stop the maxP rob/2 agent are
determined in four manager iterations, i.e. in the fifth experiment the agent never reaches its goal. Initially, this agent
has 370 available plans to reach its goal. The countermeasures
selected at the first iteration patches 7 vulnerabilities in the
nodes that connect one of the PLC to the corresponding Power
network and in the one that connects the Process network
to the other Power network. The agent reacts by selecting
dependent plans that the manager stops in the next iteration.
In particular, the following iterations discover, respectively,
40, 129 and 40 dependent plans. In particular, in the third
iteration, the manager selects a set with the same number of
countermeasures as in the previous iteration but that concerns
distinct attacks. The final set applies 9 patches.
The following experiments have considered a timelimit of
24 hours. Now only the agents with a lookahead equal to 2 or
that aims to control any of the PLC always reach their goal.
The probability that a random agent controls any of the PLCs
is 0.85 while the one that it controls both PLCs is 0.71.
A similar behavior is observed with a timelimit of 18 hours.
In these experiments, as in most of the previous ones, the
random agent and the maxIncr/1 one always reach their
goal with the same probability. This is a strong indication that
only one of the agents may be simulated if an assessment is
interested in minimizing the computational overhead.
The manager determines the countermeasures for the
maxIncr/2 agent in two iterations that return, respectively, 5
and 9 countermeasures. The same countermeasures are selected
for the maxP rob/2 agent. A lower number of iterations
suffices because the behavior of the maxIncr/2 agent is less
deterministic. Hence, in an experiment, the agent implements
a larger number of plans than the maxP rob/2 agent. This
confirms that, in general, the number of manager iterations
is inversely related to the agent determinism, but the exact
relation depends upon the target system.
In the last experiments, where the timelimit is 12 hours,
the engine cannot reach the required confidence level for the
agents that want to control both PLCs due the large number
of components they can attack. Among other agents, only the
maxEf f /2 agent aiming to control at least one PLC always
reaches its goal. The success probability of other agents ranges
from 0.97 for the maxEf f /2 aiming to control one PLC to
0.1 for the randomagent aiming to control one PLC.
To stop both agents, less than 1% of the system vulnerabilities have to be patched. This is mostly due to the segmentation
of the target infrastructure.
In all these experiments, the best strategy is maxEf f , the
worst ones are maxP rob/1 and random. Lastly, maxIncr is
the most sensible strategy to the lookahead value that strongly
influences the time to reach a goal.
VII.
B. Complex Attacks and Plans
C ONCLUSION
We have discussed the Haruspex suite, a set of tools to
assess and manage in an automatic way the risk due to an
ICT system under attack by a set of agents that escalate
their privileges through sequences of attacks. Then, we have
introduced and discuss a methodology to overcome uncertainty
about some parameters of the target system and of the agents.
Lastly, we have presented the automatic assessment of an
infrastructure with 49 nodes that is attacked by four agent. An
interesting output of this assessment is the very low number
of vulnerabilities to be patched to stop all the agents. Future
developments concern the definition of metrics to evaluate the
robustness of a system with respect to a set of agents and to
compare alternative versions of a system.
The considered infrastructure is characterized by a large
number of plans the agents can select. This is due both to the
vulnerabilities we have manually inserted and to several sets
of equivalent nodes. We say that a node n is equivalent to
a distinct node m if any complex attack where an elementary
attack at(n) involves n can be mapped into a distinct complex
attack by replacing at(n) with at(m), an elementary attack that
involves m. An elementary attack involves n if n is the source
or the target of the attack. As an example, any complex attack
where at targets one of the nodes that connect the Central
and the Power networks, can be transformed into a distinct
complex attack by replacing at with at0 that targets another
node that connect the two networks. For the same reasons, the
two nodes that connect the Power network to each Control
network are equivalent too. Since, in general, an agent may
implement distinct plans in distinct runs, a large number of
runs is required to achieve a high confidence level. Consider,
as an example, that the ratio between the number of runs and
the one of distinct plans is less than 5 for some random agent,
while may be larger than 1000 for a maxEf f /2 agent.
R EFERENCES
[1]
C. Tankard, “Advanced persistent threats and how to monitor and deter
them,” Network Security, vol. 2011, no. 8, pp. 16 – 19, 2011.
[2] F. Baiardi and D. Sgandurra, “Assessing ict risk through a monte
carlo method,” Environment Systems and Decisions, pp. 1–14, 2013.
[Online]. Available: http://dx.doi.org/10.1007/s10669-013-9463-4
7
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
F. Baiardi, F. Corò, F. Tonelli, and L. Guidi, “Qsec: Supporting security
decisions on an it infrastructure,” in Eighth CRITIS Conference on Critical Information Infrastructures Security, Amsterdam, The Netherlands,
September 2013.
——, “Gvscan: Scanning networks for global vulnerabilities,” in First
International Workshop on Emerging Cyberthreats and Countermeasures, Regensburg, Germany, September 2013.
V. Gorodetski and I. Kotenko, “Attacks against Computer Network:
Formal Grammar-Based Framework and Simulation Tool,” in Recent
Advances in Intrusion Detection, ser. LCNS, vol. 2516. Springer,
2002, pp. 219–238.
I. Kotenko, “Active vulnerability assessment of computer networks
by simulation of complex remote attacks,” Int. Conf. on Computer
Networks and Mobile Computing, p. 40, 2003.
D. Helbing and S. Balietti, “How to do Agent Based Simulations in the
Future,” june 2011.
S. H. Conrad, R. J. LeClaire, G. P. O’Reilly, and H. Uzunalioglu,
“Critical national infrastructure reliability modeling and analysis,” Bell
Labs Technical Journal, vol. 11, no. 3, pp. 57–71, 2006.
T. Brown, W. Beyeler, and D. Barton, “Assessing infrastructure interdependencies: the challenge of risk analysis for complex adaptive
systems,” Int. Journal of Critical Infrastructures, vol. 1, no. 1, pp. 108–
117, 2004.
E. LeMay, W. Unkenholz, D. Parks, C. Muehrcke, K. Keefe, and
W. Sanders, “Adversary-driven state-based system security evaluation,”
in Proc. of the 6th Int. Workshop on Security Measurements and Metrics,
ser. MetriSec ’10. New York, NY, USA: ACM, 2010, pp. 5:1–5:9.
——, “Model-based Security Metrics using ADversary VIew Security
Evaluation (ADVISE),” in Proc. of the 8th Int. Conf. on Quantitative
Evaluation of SysTems (QEST 2011), 2011.
D. Rios Insua, J. Rios, and D. Banks, “Adversarial risk analysis,”
Journal of the American Statistical Association, vol. 104, no. 486, pp.
841–854, 2009.
D. M. Buede, S. Mahoney, B. Ezell, and J. Lathrop, “Using plural modeling for predicting decisions made by adaptive adversaries,” Reliability
Engineering and System Safety, vol. 108, no. 0, pp. 77 – 89, 2012.
S. Cheung, U. Lindqvist, and M. Fong, “Modeling multistep cyber
attacks for scenario recognition,” in DARPA Information Survivability
Conference and Exposition, 2003. Proceedings, vol. 1, april 2003, pp.
284 – 292 vol.1.
S. Barnum, “Common attack pattern enumeration and classification (capec) schema description,” Cigital Inc, http://capec.
mitre. org/documents/documentation/CAPEC_Schema_Descr iption_v1,
vol. 3, 2008.
V. M. Bier, S. Oliveros, and L. Samuelson, “Choosing what to protect:
Strategic defensive allocation against an unknown attacker,” Journal of
Public Economic Theory, vol. 9, pp. 563–587, 2007.
K. Hausken and V. M. Bier, “Defending against multiple different
attackers,” European Journal of Operational Research, vol. 211, pp.
370–384, 2011.
D. Florencio and C. Herley, “Sex, Lies and Cyber-crime Survey,” in
The Tenth Workshop on Economics of Information Security, 2011.
——, “Where Do All the Attacks Go?” in The Tenth Workshop on
Economics of Information Security, 2011.
C. M. Macal and M. North, “Tutorial on agent-based modelling and
simulation,” Journal of Simulation, vol. 4, no. 3, pp. 151–162, 2010.
A. Rob, “A Survey of Agent Based Modelling and Simulation Tools,”
Technical Report DL-TR-2010-07, Science and Technology Facilities
Council, 2010.
A. Ghorbani, E. Bagheri, Onut, R. Zafarani, H. Baghi, and G. Noye,
“Agent-based Interdependencies Modeling and Simulation (AIMS),”
Technical Rep. No. IAS-TR01-06, Intelligent and Adaptive Systems
Research Group, Faculty of Computer Science, UNB, Tech. Rep., Sep.
2006.
E. Casalicchio, E. Galli, and S. Tucci, “Federated Agent-based Modeling and Simulation Approach to Study Interdependencies in IT Critical
Infrastructures,” in Proc. of the 11th IEEE Int. Symp. on Distributed
Simulation and Real-Time Applications, ser. DS-RT ’07. Washington,
DC, USA: IEEE Computer Society, 2007, pp. 182–189.
[25]
[26]
[27]
[28]
[29]
[30]
[31]
[32]
[33]
[34]
[35]
[36]
[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]
[45]
8
A. Arora, D. Hall, C. Piato, D. Ramsey, and R. Telang, “Measuring the
risk-based value of it security solutions,” IT Professional, vol. 6, no. 6,
pp. 35 – 42, nov.-dec. 2004.
A. Herrmann, “The quantitative estimation of it-related risk probabilities,” Risk Analysis, pp. no–no, 2012.
C. Alberts, J. Allen, and R. Stoddard, “Risk-based measurement and
analysis: Application to software security,” Software Engineering Inst.,
CMU, Tech. Rep., 2012.
A. Konak, D. W. Coit, and A. E. Smith, “Multi-objective optimization using genetic algorithms: A tutorial,” Reliability Engineering and
System Safety, vol. 91, no. 9, pp. 992 – 1007, 2006.
K. Deb, “Multi-objective optimization,” in Search Methodologies,
E. Burke and G. Kendall, Eds. Springer US, 2005, pp. 273–316.
R. Marler and J. Arora, “Survey of multi-objective optimization methods for engineering,” Structural and Multidisciplinary Optimization,
vol. 26, pp. 369–395, 2004.
J. D. Howard, “An analysis of security incidents on the internet 1989 1995,” Ph.D Thesis, 1998.
S. Engle, S. Whalen, D. Howard, and M. Bishop, “Tree approach to
vulnerability classification,” May 2005.
P. Ammann, J. Pamula, R. Ritchey, and J. Street, “A host-based approach
to network attack chaining analysis,” Technical Report SERC-TR-165-P,
2005.
S. Cheung, U. Lindqvist, and M. Fong, “Modeling multistep cyber
attacks for scenario recognition,” in DARPA Inf. Survivability Conf. and
Exposition, 2003, vol. 1, april 2003, pp. 284 – 292 vol.1.
Y. Zhang and V. Paxson, “Detecting stepping stones,” 2000.
B. Morin, L. Mé, H. Debar, and M. Ducassé, “M2d2: A formal data
model for ids alert correlation.”
S. Noel, M. Elder, and S. Jajodia, “Advances in topological vulnerability
analysis,” Proceedings of the Cybersecurity Applications and Technology Conference for Homeland Security, pp. 139–154, 2009.
S. Jajodia and S. Noel, “Topological Vulnerability Analysis,” in Cyber
Situational Awareness, ser. Advances in Information Security, S. Jajodia,
P. Liu, V. Swarup, and C. Wang, Eds. Springer US, 2010, vol. 46, pp.
139–154.
L. Wang, M. Albanese, and S. Jajodia, Network Hardening - An
Automated Approach to Improving Network Security, ser. Springer
Briefs in Computer Science. Springer, 2014.
Y. Y. Haimes, “On the definition of vulnerabilities in measuring risks
to infrastructures,” Risk Analysis, vol. 26, no. 2, pp. 293–296, 2006.
T. N. Jagatic, N. A. Johnson, M. Jakobsson, and F. Menczer, “Social
phishing,” Commun. ACM, vol. 50, no. 10, pp. 94–100, Oct. 2007.
NIST, “National vulnerability database,” Tech. Rep. [Online]. Available:
http://nvd.nist.gov/
MITRE, “CWE - common weakness enumeration,” Tech. Rep.
[Online]. Available: http://cww.mitre.org/
K. Scarfone and P. Mell, “An analysis of cvss version 2 vulnerability
scoring,” in Empirical Software Eng. and Measurement, 2009, oct. 2009,
pp. 516 –525.
P. Heidelberger, “Fast simulation of rare events in queueing and
reliability models,” ACM Trans. Model. Comput. Simul., vol. 5, no. 1,
pp. 43–85, Jan. 1995.
R. Karp, “Reducibility among combinatorial problems,” in Complexity
of Computer Computations, ser. The IBM Research Symposia Series,
R. Miller, J. Thatcher, and J. Bohlinger, Eds. Springer US, 1972, pp.
85–103.