Estimating the Cost and Benefits
of Software Assurance
Investments
Thomas P. Frazier
November 9, 2006
1
Background
• Enemies of our country may try subverting
government systems by intentionally embedding
malicious code or by remotely exploiting
unintentionally vulnerable software.
• This risk is rising with the broadened use of
commercial off-the-shelf products and the
increasing globalization of software
development.
• Estimate the costs and benefits associated with
specific policies and procedures to mitigate
these risks.
2
Business Case
• Business case requires that we turn the benefits
into dollar values.
• A software assurance investment should be
undertaken if the cost of this protection is less
than the expected loss of not protecting
• This is done by computing the expected value of
costs and benefits
• The expected value is defined as E = p*L, where
p is the probability of a possible outcome and L
is the payoff or dollar value of such an outcome.
3
Cost-Benefit Criterion
• An investment in assurance should be
undertaken if
C < p*L
Software assurance investment should be
undertaken if the [present value of the] cost of this
protection is less than the [present value of the]
expected loss of not protecting
where:
C =the cost associated with the software
assurance investment
p = probability of a successful attack by an intruder
L =dollar loss resulting from malevolent intrusion
4
Concerns
 Many of the policies and procedures are likely to
be so general that merely enumerating all the
cost and benefit elements is hard, much less
assigning values to them.
 Policies, procedures, and investment ideas will
be correlated, meaning that implementing A
could affect the costs and benefits associated
with B.
 Quantifying C, and especially p and L
5
Literature Search is Not Encouraging
• Cashell, Brian, William D. Jackson, Mark Jickling, and
Baird Webel, The Economic Impact of Cyber-Attacks,
Congressional Research Service, Library of Congress,
April 1, 2004
“No one in the field is satisfied with our present ability to
measure the costs and probabilities of cyber attacks. There are
no standard methodologies for cost measurement, and study of
the frequency of attacks is hindered by the reluctance of
organizations to make public their experiences with security
breaches.”
• Howard E. Glavin, “A Risk Modeling Methodology,”
Computer Security Journal, Vol. 19, No. 3 (Summer
2003).
• U.S. House of Representatives, Select Committee on
Homeland Security, Subcommittee on Cybersecurity,
Science, and Research & Development, “Cybersecurity
for the Homeland,” December 2004.
6
Proposed Approach
• Use some combination of data taken from
the U.S. liability insurance industry,
combined with estimates from experts to
estimate p and L
• Underwriters essentially estimate p and L
every time they write a policy.
• Employ liability insurance data from firms
that write and implement software
7
Liability Insurance Data
• Data from the product liability industry can
shed some light on the following:
– How much liability insurance is purchased by
firms of various industries and sizes,
– How much the firms pay for the insurance,
and
– How much the firms set aside for selfinsurance, etc. (i.e., exclusions and other
coverage information)
8
An Example
• Assume that one of the recommended
investments is a “red team” review of key
software.
• C - cost of performing the red team reviews,
measured by the fully burdened cost of the red
team members plus some estimate of the other
direct charges typically incurred in these types of
operations (e.g., 15 percent of the direct labor
charges). COSECMO model may also prove to
be useful in estimating C.
9
Estimating Risk
• Assume that the cost of insurance (C) is the expected loss coverage
(L) times the probability of loss (p)
– That is, C = p  L or p = C ÷ L
– p may overstate the probability since the insurance company makes a
profit
– However, the policyholder retains some risk (deductible) which means
the actual loss is usually greater than the expected insured loss in the
above relationship
– For simplicity, we will assume these effects offset each other
• Insurance data on premiums paid for typical coverage indicate that
software companies pay about $5 million in annual premiums (C) for
$1 billion of coverage (L).
• p = C ÷ L = $5M ÷ $1B = 0.005
• This leads us to estimate that the probability of code failing (p) in
any one year is 0.005 ($5 million/$1 billion)
10
Estimating SwA Effectiveness
• Gartner results indicate that removing all defects
would reduce the malicious attack rate by 35%*
• Now assume software assurance (SwA) in the
form of a red team review can cut defects by
50%
• If failure p = 0.005 without SwA, then SwA can
reduce the malicious attack failure by 0.005 x
.35 x .5 in this example
• Reduction in p due to SwA is 0.000875
*Gartner reports that 35% of attacks exploit unintended vulnerabilities in code
11
Estimating Loss
• Next, we need an estimate of the loss suffered if
a code failure occurs
– Loss could be estimated by time systems are out of
service
• Ship fleet costs $23M/day (development & operation)
• An attack might cause a 1-day loss of service ($23M loss)
– Examples across industry show higher collective
losses:
•
•
•
•
•
Love Bug (2000) $15B damage to 3.9M systems
Code Red (2001) $1.2B damage, $740M recovery
Slammer (2002) $1B damage
Blaster (2003) $50B damage
My Doom (2004) $38B damage
12
Comparing Risk and Benefit
• Avoided annual loss probability
– p(reduced) = 0.000875
• Cost of loss for one day of a ship system
– $25M (2006 dollars)
• Annual expected loss avoided
– 0.000875 x $25M = $22K
• System lifetime loss avoided
– 30 years x $22K = $660K (2006 dollars)
• Cost of review
– Can the SwA measure be done for less than $660K?
13
Nomograph of C, L, p = 0.005
Assuming 30 year system life, 35% of defects lead to attacks, 50% defect reduction
Area that satisfies our costbenefit criterion
14
Private Industry Insurance Data are Not a
Perfect Proxy
• Firms that make up the sample drawn from the
insurance database may not face the same
threat and losses that the government faces
• Private firms probably have different tolerance for
risk than the government, and this difference
influences how much coverage they buy and the
price they are willing to pay
• But, isn’t such a proxy better than merely
guessing at p and L in those instances where the
experts cannot provide estimates of the
parameters?
15