A Robust Process Model for Calculating Security ROI
SE 690/698 Project Proposal
Ghazy M. Mahjub
DePaul University
1 Introduction
There are only three things certain in life: death, taxes, and risk. Every moment of our lives
involves each and every one of us taking some kind of risk, thereby hoping that the option we
choose amounts to more utility than the option forgone. We trust ourselves in these every day
decisions that we make, knowing that the decisions we make today often end up deciding the
path of our future. Of course, with every investment there is risk and a justification for choosing
to either take on the risk or avoid it all together. For example, part of our tax money goes to
funding the local metropolitan bomb squad. The bomb squad is and has been a part of every
major metropolitan police and fire department since the early 20th century. Bomb squads are
funded hundreds of millions of dollars a year to constantly practice and research and train and
practice some more in a never ending process to stay ahead of the knowledge curve of explosives
technology. Despite this fact, not once this year have explosives caused any significant human or
infrastructure damage in any one of the major metropolitan areas of the United States. So the
question beckons, how do we justify the expenditure of taxpayer money on maintaining and
building such an elite team to handle such rare cases? The answer is simple: preventing the cause
is easier than undoing the effect.
2 Understanding Risk and Securing Uncertainty
Depending on what line of business one is involved in, the word “risk” has different implied
meanings. We take a risk in one area to receive gratification in another. Keynesian economics,
the basis of capitalism, states that in choosing one opportunity, we forgo another. This concept is
well known as the opportunity cost. This is where risk becomes a playing factor because if we
knew for sure the outcome of two choices, the decision would be easy, but we don’t. Uncertainty
rules, as it does in just about every risk management situation, and so the game becomes to
minimize the areas where we have little to no control over the outcome, where the link between
cause and effect is hidden, and maximize the areas where we have some control over the
outcome.
Consider the following quote by Louis Bachelier, a famous 18th century philosopher and
economist: “The information you have is not the information you want. The information you
want is not the information you need. The information you need is not the information you can
obtain. The information you can obtain costs more than you want to pay.” Information is the key
to effective decision-making, and yet it is the most elusive asset known to man. If, for example,
we had the necessary information two years ago on a fateful day in September, the world would
be a different place today. That day in September clearly demonstrated that the lack of critical
information can have devastating effects.
In 1882, George Parmalee, an English inventor from Bolton, England told the cotton
spinning factory owners of the English community that, “you, my fellow Englishmen, need fire
sprinklers”. Obvious advice considering the flammability of cotton and yet nobody listened;
moreover, most laughed in his face and considered the investment nothing more than a waste of
money. In fact, sprinklers in 1882 were considered to be as dubious an investment as information
security is today. Even after setting fire to a cotton spinning factory and then extinguishing it
with his 32 automatic sprinklers, sales remained extremely slow. It seemed that not even the
most convicting of evidence could budge the mill owners in their attitudes towards the idea. The
reason for this lack of open-mindedness is the same reason people fail to invest in information
security expenditures today: “Parmalee realized that he could never succeed in obtaining
contracts from the mill owners… unless he could ensure for them a reasonable return on their
investment,” brilliantly stated by Sir John Wormald, a witness to the incineration spectacle. Of
course times change, but things remain the same and in the high-priced, high-stakes game of
information security, proof of return is ink on the dotted line!
2.2 Annual Loss Expectancy
Many methodologies have been proposed for calculating information security risk values and
managing these risks. Peter S. Trippett, vice-chairman and chief technologist at TruSecure
Corporation, a security consulting firm, recently argued in the Wall Street Journal that the
following simple formula can be used to quantify a particular risk at hand: Risk = Threat x
Vulnerability x Cost.
The beauty of this method is that if any of the variables can be reduced to zero, then the total
risk calculates to zero and is consequently eliminated. Contrary to what many security
consultants would like us to believe, no two identifiable risks are created equal. Risk is not
just a technical issue; rather, it is also a business issue, which carries business impact. Security is
not a binary issue; it is not a question of being secure or not being secure; it is more concerned
with the level of security that is required, and the level of risk that can be tolerated. If the
hallmark of the modern age is the mastery of risk, then modern times demand that computer
security risk be managed with an open and rational process that depends more on quantified
costs and benefits than on the pronouncements of a guru and the effects of fear!
One of the earliest used estimators of loss due to security breach in the computer industry
was a quantitative method for performing risk analysis known as the Annual Loss Expectancy
(ALE). It was described in a 1979 FIPS publication by the National Institute of Standards and
Technologies as appropriate for use by large data centers. Calculation of a risk estimate was
produced by multiplying the estimated frequency of occurrence of attacks by the possible loss
amount for each data file, and then summing these results. The method was criticized and proved
lacking because of the lack of empirical data on frequency of occurrence of impacts and the
related consequences, resulting in an interpretation of results as having more precision than they
actually had.
Another major deficiency with ALE is that its validity rests on a scenario analysis procedure,
involving the analysis of all assets, security concerns, threats, and vulnerabilities, in order to
generate all possible scenarios of attack/breach. Each asset translated into multiple security
concerns, which multiplied into multiple threats, which multiplied into multiple vulnerabilities.
As the growth of scenarios was exponential, developing acceptability tests for each one and
running this through an automated application proved unrealistic for even the most high
performing computer hardware. The final downfall of this method was its unrealistic treatment of
security as a binary issue, a point we touched on earlier. Probabilities do exist in the calculation
of risk and therefore any evaluation of calculated security investments must take this into
account when making decisions on security investments. Models such as ALE have generated
false confidence in such numbers. When risks are not weighed and probability of risk
materialization is not considered, security investments often wind up wasted and providing
absolutely no mitigation from low impact, high probability risks which can add up to immense
losses.
2.2.2 Other models
Other attempts at information security risk management include R. Campbell’s [1] proposed
modular approach. In his research, Campbell designed a generalized framework for modeling a
broad spectrum of risks [15]. The model includes several sub-models, including value-analysis,
threat analysis and identification, vulnerability analysis, risk analysis, risk assessment,
management decision, control implementation, and effectiveness review [16]. Summers [17] also
proposed a similar four-step procedure in risk-analysis: 1) Identify the assets and assign
monetary values to them; 2) Identify the threats and the vulnerabilities; 3) Calculate the annual
loss expectancy exposure (ALE) of each threat; 4) Identify potential safeguards and estimate
how much they reduce the exposure.
S. Boran [18] designed top-down and bottom-up risk analysis methodologies to improve
security. The bottom-up approach determines how much protection the business system requires
and then follows that up with deciding the potential risk to the system. This method lacks the
precision required to achieve accurate cost-benefit results. The top-down approach is much more
meticulous and precise than the bottom-up, but it is slower and more time consuming due to the
large number of scenarios that must be analyzed. This approach involves five steps including
asset analysis, analysis of current security rules, policies, and practices, definition of basic
security objectives, threat analysis and impact analysis. [16] A similar approach by Pfleeger [19]
concentrates on the calculation of expected annual loss, the cost of control, and annual savings of
control.
3 Information Security and Quality
Risk management in corporate investing categorizes risk strictly according to business impact.
As IT is fundamentally a business facilitator, the risks should therefore be viewed in the same
manner. The purpose of information technology is to provide free flow of information with
minimal latency to customers who make business decisions based on that information. Security
hampers this cause, demanding tight control over the flow of information and demanding that
checks and balances be integrated into an information system in order to ensure the primary
attributes of information: availability, integrity, confidentiality, and authenticity.
Security has been defined as “the protection of information, systems and services against
disasters, mistakes, and manipulation so that the likelihood and impact of security incidents is
minimized”. Security and information can only be correctly balanced through the services of an
Arbitrator. That arbitrator is Quality. Consider the situation in which a corporation issues
passwords for every single imaginable access point and access resource in its IT armada. This
may provide a minimal increase in security, but it will have a much greater adverse impact on the
transfer of critical information, causing a plummet in productivity, and often forcing employees
to bypass the security controls (e.g. sharing existing passwords), thereby deeming the system
useless anyway. Producing Quality security infrastructure and Quality software applications that
integrate security, instead of bolting it on as an afterthought, can and will do amazing things for
the security ROI curve.
Old methods for quality assurance relied heavily upon inspecting products as they “rolled
off the line”. These legacy methodologies have somehow managed to find themselves engrained
in the creation of software. Quality issues in software are in more and more cases being
addressed as they come up, and fixes being bolted on in response. Nowhere is this more evident
that in the case of Microsoft and their Windows operating system. No doubt we have all come to
know our Windows Update Manager very well, and quite simply, enterprise level solutions
demand a much higher level of Quality upon delivery. These types of security patches are a
perfect example of “bolt-on security” and too often are either to late to trickle down to all
customers, or simply are not effective in such a dynamic industry. With this in mind, I would like
to introduce the Robust Design Method as a prime candidate for simplifying the software quality
control procedure.
In the face of massive complexity of today’s software systems, it is growing extremely
difficult to guarantee the quality of these products using current methodologies. Integration
issues have risen to the top of the priority list in the software community as corporations chose to
upgrade current systems rather than replace them altogether. Customers will continue to demand
increased functionality to enable increased profit from the software systems they have chosen,
thereby continuing to raise the bar on software complexity. As complexity rises, the risk of
failure itself is elevated. In order to maintain risk at a level at which it can be mitigated and if
necessary insured, quality must take a continually higher precedence in the SDLC. It can be said
with confidence that the risk management decisions that are made should rest squarely on the
confidence that risk managers have in the quality of the system that their counterparts in software
development are attempting to build. A manger who has the utmost confidence in the quality of
his software system will not be forced to insure beyond the natural security factors (those factors
that are risks in every project and are thus accepted). Security now becomes a quality issue and
to effectively analyze its components, it is broken down in components that reflect the quality of
the system. QUALITY BREEDS CONFIDENCE!
3.2 Security as a quality issue
Quality of Service (QoS) is a measure of what is expected from a system versus what the
system is delivering, in terms of its service responsibility. There are undeniable factors that
prohibit a system from performing at its ideal capacity, and such is the case with software
systems. The most widely known application of the Quality of Service attribute is in applications
to network technology and resource management.
Aurrecoechea and Campbell systematically studied the architecture of QoS [20]. Irvin
proposed the concept of quality of Security Service (QoSS), and QoSS’s cost method and
taxonomy [21]. QoS involves users’ requests for different types and levels of service from some
system. The level of service that they actually receive is influenced by performance variables
which are inherent within the system [20]. Irvin’s research defines security as a dimension of
quality of service [21], as has been done in this discussion. Irvin justifies his view of security as a
dimension of quality by defining security services, to include integrity, authenticity, availability,
and confidentiality. Of course, it is NOT a coincidence that these qualities are also the ones used
to previously define Information. Irvin’s research falls short in defining the cost-benefit analysis
The Standard Taguchi Model
X
M
Signal Factors
Noise Factors
PRODUCT PROCESS
SYSTEM
Z
Y
Response
Control Factors
Figure 2: The standard Taguchi Model.
for security cost versus its benefits and the subsequent trade-off criteria used to support security
related decision making. This research draws upon a similar concept as Irvin’s QoSS [21] to
create a model which formulates a separate set of requirements (we will call them malicious-user
requirements for now) to identify the necessary security infrastructure so that it effectively
offsets possible losses due to attack while minimizing the negative effect on the free flow of
information that the system provides.
3.3 Application of Robust Design Method
The Robust Design Method was pioneered by Dr. Genichi Taguchi after the end of the
Second World War, and has greatly evolved in the five decades since. In software, the term
“Robust” is widely used to describe a software system and yet, its meaning is not always clearly
defined. The Robust Design Method’s prime call of duty is to improve engineering productivity
by consciously considering the noise factors and the cost of failure of a system to ensure that a
defective product does not cause damage in real time situations. Robust design focuses on
improving the fundamental function of the product or process, thus facilitating flexible designs
and concurrent engineering. It is a very powerful method available to reduce product cost,
improve quality, and simultaneously reduce the development interval. The key is to understand
how the Robust Design Method can apply to software security, quality, and risk management in
a field that has some commonalities but many differences from traditional engineering.
The first phase of the Robust Design Method involves developing a robustness strategy.
Essentially, we must eliminate deviations from established standards in order to provide a quality
end result product. From the software perspective, eliminating deviation rests squarely upon the
shoulders of the requirements specifier. And yet the fundamental fact is that project requirements
do and will change due to changing customer demands, fluctuations in budget, and other such
unpredictable yet undeniable surprises. Uncertainty rules, and therefore risk is involved here. For
the sake of this discussion, let’s call these types of risks Natural Risks. Natural risks in the
general sense of the term include tornados, earthquakes, floods, and other such things that we
really have no control over, and therefore must accept and essentially insure. And yet, although
traditionally our response to managing these risks has been just that, insuring, the concept of
robust design argues that in order to improve the quality, we must not only mitigate these risks
but minimize the effect of these risks by making our product as insensitive as possible to the
environment it will operate in and the variation in the components it employs.
3.5 Quality loss function analysis
The quadratic nature of the Quality Loss Function, a function developed by Gnechi Taguchi
that measures loss due to quality, or the lack thereof, tells us that the higher the deviation from
the target value of a factor, the higher the loss and the more rapid the loss function grows.
Therefore, the validity of this function as it applies to software development lies squarely on
mathematics involved in identifying noise factors, which have the most effect on loss to our
system through nominal range analysis, identifying acceptable ranges of risk for these noise
factors through parametric sensitivity analysis, and finally achieving a system response that
yields control factor costs which effectively offset any probable loss from these noise factors.
Quality Loss Function
L(x)
y = 6.6667
R2 = 0
x
Figure 3: The Quality Loss Function [2]
3.6 The bridge to anti-requirements
The vision of the anti-requirements community is one in which security requirements are
engineered by a process of organizational theory. Essentially, this process acts as a bridge
between the well-ordered world of the software project informed by conventional requirements
and the unexpected world of anti-requirements associated with the malicious users. An antirequirement is a requirement of a malicious user that subverts an existing requirement. As part of
the traditional security requirements process, we establish security requirements that intend to
enforce and maintain confidentiality, integrity, and availability of our software system and the
information that it holds. Anti-requirements are generated either when requirements are
insufficiently detailed, thereby leaving holes in the implementation, or when the framework for
developing security requirements is inadequate and does not address the areas that are
consequently being formalized into anti-requirements by the malicious user.
An anti-requirement specifies what the software system should not do. Essentially, the
operational process of a software system is only wrong if someone says it is wrong. Therefore,
these things must be specified in an anti-requirements document. Consider the scenario in which
a security requirement has called for the implementation of role-based access control to restrict
the access of information assets. An anti-requirements document would document what each role
is not supposed to do, so that roles can be further refined so as unusable roles do not get assigned
to employees. Roles which are vaguely specified or assigned so that they do not fit the
requirements of users never fail to reduce productivity and increase risk. Anti-requirements
allow us to view our system through the eyes of the malicious user to avoid these situations.
3.7 Security ROI Calculator
It can now be concluded that security demands a robust software system that is not overly
sensitive to natural factors, such as the inevitable virus, breach, or denial of service attacks, and
is formalized by anti-requirements. In order to ensure this, the nominal values of critical system
parameters must be changed such that the system’s function becomes insensitive to these antirequirements. We do this by compensating with control factors. Security, being a non-functional
requirement, requires that to ensure insensitivity to bad environmental conditions, we must
integrate this specification into our requirements at a very early stage in development.
The model in Figure 4 illustrates the process. First, anti-requirements are formulated as noise
factors that are beyond the control of the designer. In order to obtain the most robust design, antirequirements are used to perform risk assessment in order to minimize stray from the
requirements of our security system, thereby producing safeguards from actual malicious user
requirements, and consequently maintaining a high quality, robust design. The risk assessment
phase involves taking each anti-requirement and mapping it to system assets to obtain an impact
value and using Pfleeger’s likelihood prediction process to estimate probability. The quality of
this model is dependent upon the quality of the Risk Assessment step, and ensuring that the
control factors offset the noise factors sufficiently to ensure an acceptable system response. The
development of the anti-requirements produces a set of requirements that the proposed security
system should meet in order to adequately defend the functional layer from attack.
The risk assessment phase is a critical part of this model. Earlier we noted that Risk is
essentially a product of probability and impact. Impact is the easier of the two to estimate since
values can be placed on assets using traditional accounting methodologies. Probability on the
other hand, is a different issue; accurate probability estimation requires historical data on the
event that probability is being estimated for, and yet historical data is rare and that which is
available has been dubbed, “crap data” by many in industry. The Robust Design Method
proposes a Brainstorming step prior to all other steps in the Robust Design Method to formulate
probability and impact values for the Risk Assessment phase. The brainstorming session
Noise Factors
Risk Assessment
Assessment
Risk
X
Control Factors
Z
Controlled
Risk Adjusted, Xr
Noise Factors
X
Robust Design Method
Response
COST-BENEFIT ANALYSIS
PROCESS
Y
Figure 4: Security ROI calculator with Robust Design Simulation of possible
risk adjusted impact.
involves gathering all people that will be involved in the project, both technical and business-
oriented, to assess available data, past experience, and individual expertise to come up with a set
of risk values that can then be run through the Robust Design Model to produce ROI numbers
that can then be further analyzed.
5 Conclusion and Future Work
This research has introduced a new way of viewing security. We have introduced a
methodology which involves viewing security as a Quality issue. The degree of security
provided by our software in insuring the confidentiality, integrity, and availability of information
assets and systems relies completely on the quality of the software system being built. The most
efficient model of security is one in which security is built directly into the system not as a bolton mechanism but as an integrated part of the software system. Therefore, the security of our
information assets, both tangible and intangible, depends completely on the quality of the
security system being built as a module of the software system as a whole. In this paper, the
issues of software security and software quality are handled synergistically, so that security
becomes a quality issue. The Robust Design Method was introduced to create a model for robust
determining the correct security investment. By formulating anti-requirements, the model
formalizes threats and provides traceability to those components which could be affected by
these threats.
This research has been ongoing now for several quarters, beginning since the summer.
Since then, a substantial amount of revisions have gone into the research and plans have changed
several times. Currently, the research is in a stage of testing of the Robust Design Method as it
applies to calculating ROI of security infrastructure. Data has been obtained from computer
crime surveys which have been done by the FBI in the past and efficacy data of different types of
security infrastructure also has been gathered from past research into the subject. Using this data,
code has been developed in combination with Excel models to analyze the data. The current
struggle is to determine the correct amount of data necessary to produce statistically significant
results that can be trusted within a tight confidence interval. Although the main scope of this
research is designing the ROI calculator, further research providing traceability between antirequirements and assets susceptible to attack will be done but primarily in order to demonstrate
the model as it applies to a real-world situation. I have provided a schedule for the research
attached to this proposal outlining expected milestones and completion dates for the various
phases.