An Empirical Analysis of Software
Vendors’ Patching Behavior
Rahul Telang
With
Ashish Arora, Ramayya Krishnan and Yubao Yang
Carnegie Mellon University
1
“Developing and deploying patches is an
increasingly important part of the
software development process” –
Joseph Dadzie, MICROSOFT
2
Motivation
• Software vendors’ incentives to invest in product quality, product
maintenance has been a keen area of interest (Krishnan et al 2000,
Slaughter et al 2000)
• Many believe that vendors provide suboptimal levels of quality and
security and applying product liability rules to software is being
debated.
• One key aspect of better and more secure software is timely and
reliable patching by vendors.
• The quality of a software product critically depends on the ex-post
support a vendor provides (Arora, Caulkins and Telang 2005).
• In this paper, we derive empirical estimates for the drivers of vendor’s
patching behavior.
3
Motivation
• In particular, our focus in on the patching of security
related vulnerabilities by vendors
– The number of such vulnerabilities found and reported has
exploded in last few years (CERT/CC published more than
3000 vulnerabilities in 2003 alone)
– Many such vulnerabilities are particularly costly to
customers. Thus patching is of paramount interest to
customers.
– Unlike many other measures of security, patching speed can
be measured more reliably.
4
Motivation
• Unique feature of software vulnerabilities is that anyone can find
them (randomly or by exerting effort) and can make the
information regarding this flaw public. This can exert significant
negative externalities on all users using that software.
• Many such discoverers do not trust the vendors to provide timely
and reliable patches. In many instances, they resort to disclosing
vulnerability information in public forums in the hope of
pressuring vendors.
• There is no consensus on “what is the optimal time”. Vendors
want more time to be given and have taken the users to court for
disclosing the vulnerabilities (Cisco, 2005)
• There is little empirical work to quantify vendors’ incentives to
patch.
5
Motivation
• "The truth is that unpatched Windows flaws
have a value to the underground community,
and it is not at all uncommon to see these
things sold or traded among certain groups
who use them by quietly attacking just a few
key targets. So, the longer Microsoft takes to
patch vulnerabilities the longer they are
leaving customers exposed” –Marc Maiffret,
eEye
6
Literature
• How process improvements lead to better quality and lower
maintenance costs (Krishnan et al 1999; Slaughter et al 1998).
• Arora, Caulkins, Telang (2003) highlight the incentives of vendors to
“Sell First and Fix Later”.
• Telang and Wattal (2004) show that disclosure is costly to vendors and
hence provides incentives to vendors to improve the quality of their
software
• Disclosure of vulnerabilities and how it affects patching behaviors
– Arora, Telang, and Xu (2004) outline a model for how long one
should wait before disclosing. (Cavusoglu et al 2004)
• Nizovtsev and Thursby (2005) model the incentives of individuals to
disclose software vulnerabilities through an open public forum.
• August and Tunca (2005) analyzes the patching behavior of users and
how it affects vendors’ incentives to improve network security.
7

Vendors’ Incentives to Patch
•
Security vulnerabilities are costly to vendors and customers.
(i) Vendors incur cost of patching. One would expect that more time
they have for patching less it costs. In short, they would like to
delay the patch as much as possible.
(ii) Vendors’ customer incur loss if the vulnerabilities are exploited
when they are breached. Vendor cares about customer loss
(reputation loss, sometimes contractual obligations). How much
they care depends on the market structure.
•
Vendors trade-off these costs.
•
Vendor characteristics, vulnerability characteristics, and disclosure
affects these costs and hence patching behavior.
•
The factors that increase “customer loss” should force vendor to
accelerate patching. Factors that increase “patching” cost should slow
patching.
8

Predictions of Analytical Model
• Larger vendors patch faster.
– Larger customer base and worry of reputation loss that may
spillover to other products. They may have less average patching
cost.
• Severe vulnerabilities are patched faster.
–
Severe vulnerabilities increase customer loss .
• Publicly traded firms patch faster.
–
Public firms may experience the consequences of a fall in reputation
more directly through reduction in share price (see Telang and
Wattal 2004).
9

Predictions of Analytical Model
• Open Source Vendors – Large debate in academic and trade publications
on comparisons between open source and closed source vendors.
Patching is also part of the “quality”.
• Finally, disclosing the vulnerabilities would increases customer loss and
should force vendor to patch faster.
• The policy makers have struggled with “how long should vendors be
given to patch”. Our paper can provide key empirical estimate on how
changes in disclosure affect patching time. This “quantification” is
necessary to understand both vendors response to disclosure and hence
the efficacy of disclosure as a tool.
10
Data source
•
Vulnerabilities are published by many sources. We focus on CERT/CC
and Securityfocus (SF). In particular, we focus on the vulnerabilities that
are published by both sources to reduce any inherent bias.
•
CERT/CC (Center for Emergency Response/ Coordination Center) is a
federally funded organization which disseminate vulnerability
information. It researches the vulnerability when a user notifies it, and
then contacts the vendor and provide it with “protected period” before
making it public (A typical protected period is 45 days).
•
Securityfocus (an online open forum) has less quality control over the
vulnerabilities published on its forum. It also does not provide any
“protected period” unless individuals choose to do so. In short,
individuals can post the vulnerability information even without patches.
•
Fortunately, there are exception to these policies on both CERT and SF
which allows us the variation in our independent variables.
11
Data
•
Vulnerabilities published by SF and/or CERT/CC from 9/26/2000 to
8/11/2003.
– Vendor Notification date (CERT provided us this data and from SF
website)
– Patching date (from vendor websites and from CERT and SF).
•
Vendor information
– from Hoover’s online and vendor’s website
•
Vulnerability characteristics
– from the national vulnerability database (NVD, previously ICAT
database)
•
After cleaning up the data we have 1280 observations, related to 255
unique vendors and 303 unique vulnerabilities.
12
Example of vulnerability publication by CERT
13
Example of vulnerability publication by SF
14
Disclosure
• Once the vendor is notified of vulnerability, it is possible
that information get disclosed before the patch is out.
• Many times notification and disclosure happens at same
time.
• First vendor to patch may disclose the information, CERT
may disclose it (sometimes before protected period), SF
always discloses it, any third party may disclose it.
• In short, disclosure is “time-varying”.
• In fact, for the same vulnerability different vendors may
face different disclosure time.
notification
disclosure
patch
15
Descriptive statistics
Patching Time (in days)
Mean
56.5 (114.8)
Median
19
% patched
90%
Disclosure Time (in days) *
*
Mean
15.6 ( 43.0)
% instant disclosure
67.1%
Number of observation facing disclosure
985
Elapsed time between Vendor Notification and Disclosure (in days)
16
Descriptive statistics
Vendor characteristics (N=142)*
Mean
Std. Dev.
Vendor employee size (in 000’s)
17.60
66.12
Public firm
0.34
0.47
Open source
0.23
0.42
* Include only the vendors for which the reliable information is available
Vulnerability severity metric*
Mean
Std. Dev.
Min
Max
Vulnerability severity metric
14.82
16.48
0
108.16
Number of affected
vendors/vulnerability
8.23
21.53
1
242
* Vulnerability severity measurement by CERT/CC
17
Compare between CERT and SF
Published by
CERT/CC and/or SF
Published by
SecurityFocus alone
For All Observations
% patched
93.21%
62.66%
Disclosure Time
15.49 (38.06)
16.15 (72.20)
Severity Metric
25.43 (22.06)
8.65 (4.40)
Obs / vuls
1311 / 349
158 /89
For Patched Observations
Patching time
55.38 (114.09)
70.88 (122.52)
Standard errors are in parenthesis
18
Kaplan-Meier survival estimates
0.00
0.25
0.50
0.75
1.00
Kaplan-Meier survival estimates, by disclosure
0
100
200
analysis time
disclosure = 0
300
400
disclosure = 1
20
Empirical Strategy
• OLS is inappropriate.
• Proportional hazard model (PHM), assuming the baseline
hazard has the form of Weibull distribution (Cox leads to similar
estimates)
• In PHM, covariates shift the hazard function proportionally
(needs to test for it).
• Key covariates – vendor size, severity, Open source/closed
source, publicly traded firm, CERT dummy and disclosure.
• Disclosure is a time varying covariate
– For a vulnerability i and vendor j, if disclosure happens at some
time t1 such that t0 < t1 < t then, disclosure = 0 before t1 and
disclosure = 1 after t1
21
Clustering and Heterogeneity
• We control for the unobserved heterogeneity
across vendors.
• Same vulnerabilities affect multiple vendors.
We control for such clustering.
• We run what is known as “frailty” models
(more or less these are like “random-effect”
models in regression)
22
Estimates (Hazard Ratio) Weibull Hazard Model
(N=1469)
(1) With frailty
(2) Without frailty
Haz. ratio
Std. Error
Haz. ratio
Std. Error
CERT_effect
3.04***
0.35
3.10***
0.33
Disclosure
2.37***
0.20
2.19***
0.17
Small
0.84
0.51
0.80**
0.09
Vendor size
1.01
0.03
1.00
0.01
Public firm
1.32
0.26
1.28***
0.11
Open source software
1.71***
0.30
1.61***
0.14
Severity metric (log)
1.24***
0.03
1.20***
0.03
Post 9_11
2.08***
0.14
2.05***
0.13
ln_p (Weibull shape parameter)
-0.58***
0.02
-0.65***
0.02
σ (frailty parameter)
0.32
0.10
Log Likelihood
•
•
-2978
-3014
10% level, ** 5% level and *** 1% level.
The constant for the proportional hazard model is not identified when hazard ratio is estimated.
23
Interpretation
• Disclosure increases vendors’ patching speed by 137%.
• Open source vendors patch 70% faster than closed source
vendors.
• Severe vulnerabilities are patched faster.
• Vendors respond more favorably to CERT than SF. (almost
200% faster). In short vendors do not care as much if an
individual or SF informs them about a vulnerability. Credibility of
“who” is informing them matters.
• Small vendors do not patch as fast.
24
Estimated hazard ratio with and without disclosure
25
The Estimated Effect of Disclosure
Disclosure time T
Expected patching time (in days)
Effect of disclosure
(in days)
disclosed at time T
without disclosure
0 (Instant disclosure)
33.03
61.77
28.74
1
36.90
61.77
24.87
2
38.60
61.77
23.17
3
39.98
61.77
21.79
4
41.16
61.77
20.61
5
42.21
61.77
19.56
6
43.18
61.77
18.59
7
44.07
61.77
17.70
8
44.90
61.77
16.87
9
45.69
61.77
16.08
10
46.43
61.77
15.34
These calculations are done setting the covariates at their average sample values namely that the vulnerable
vendor is a public firm and has the average employee size of our sample; the vulnerability is published after
9/11 event, handled by CERT/CC and has the average severity metric of our sample.
26
Does it matter who is disclosing?
• Apparently it does. Vendors care about
the “source” of the disclosure just like
they care about “who” informs them of
vulnerabilities.
27
External Validity
• Last month, we found another data source which is similar to
ours.
• Brian Krebs of Washington Post is was also interested in finding
out whether disclosure affects the patching time. He collected
data from Microsoft between 2003-2005. He collected
information on the time when Microsoft was informed and the
time when Microsoft released patch.
• For some of the vulnerabilities, instant disclosure happened.
• Almost all of these vulnerabilities were then reported by
Microsoft to CERT and SF and posted on their pages along with
a patch.
• We can use this data to verify what the impact of disclosure is.
29
Summary statistics
• N = 99
• No disclosure N = 83, mean patching time = 130 (80)
days, average severity score = 7.06
• Disclosure N = 16, mean patching time = 62 (58)
days, average severity score = 6.4.
parameter
estimate
disclosure
severity
3.71** (1.17)
1.69* (.50)
Log Likelihood
-352.
30
Results
• Very strong effect of disclosure. In fact,
more pronounced than in our data set.
• Suggests that disclosure is a very
important determinant of vendor
patching behavior and confirmed by two
independent data sources.
31
Robustness
• Our results are not sensitive to the assumption of the Weibull
distribution since the Cox non parametric specification yields
similar results
• Controlling for the source of disclosure, and for unobserved
heterogeneity in vulnerabilities also does not qualitatively affect
the results.
• Controlling for vulnerability specific random effect (rather than
vendor specific random effects) doesn’t change the results.
• The impact of disclosure and CERT/CC remains qualitatively
unchanged with including only patched observations or
dropping patching time of more than 1 year.
32
Proportionality assumption
• Schoenfeld residuals Proportionality assumption cannot reject the
proportionality hypothesis for the disclosure effect
• But test results show that with time the effect of CERT/CC on patching
time changes
• We interact the CERT dummy with time to correct for lack of
proportionality
• Proportionality test after including the interaction term between the
CERT dummy and time corrects for proportionality assumption.
33
Conclusion
• One of the first papers to examine vendors patching behavior.
• Disclosure forces the vendors to patch faster
– increases vendors’ patch delivery speed by almost 137%.
– instant disclosure force a vendor to release the patch 29 days
earlier. This does not mean this is the right thing to do. But
we have some quantifiable measure of vendor response.
• Our model is still reduced form. A structural model may identify
the effects separately (i.e. customer loss and patching cost).
• Do not control for the quality of the patch. Disclosure could be
endogenous? [Though we control for severity and vendor size].
Might need to find an instrument.
34