Metrics Revisited
Kim L. Jones CISM, CISSP, CRISC, MSIA
Kim L. Jones CISM, CISSP, CRISC, MSIA
Sources and Inspirations
• Paul Glen, How to Speak to the Business
– www.leadinggeeks.com
• Lance Hayden, IT Security Metrics: A Practical
Framework for Measuring Security &
Protecting Data
• Andrew Jaquith, Security Metrics: Replacing
Fear, Uncertainty, and Doubt
2
Kim L. Jones CISM, CISSP, CRISC, MSIA
The Mantra:
“Infosec is Terrible at Metrics”
• The metrics we can measure has little to do with
security
– Ex: Success of Antivirus System
• The stuff we really need to convey is the hardest to
collect/quantify
– “What is the sound of one hand clapping?”
• When we quantify numbers, they question our
calculations
• They really don’t care about security…only
compliance
– “What needs fixing in security, and when will it be fixed?”
3
Kim L. Jones CISM, CISSP, CRISC, MSIA
Defining the Problem
• Good vs. Bad Metrics
• Contraxioms
• Asking the Right Question
4
Kim L. Jones CISM, CISSP, CRISC, MSIA
Good vs. Bad Metrics
Good
Bad
• Consistently Measured
• Inconsistent Results
• Cheap to Gather
– Technologically driven,
where possible
• Expensive to Gather
– Extremely Manual
• Expressed as a cardinal
number or a percentage
• Highly Subjective
– High/Medium/Low
• Expressed using at least
one unit of measure
– Hours, defects, dollars,
etc.
5
Kim L. Jones CISM, CISSP, CRISC, MSIA
“Contraxioms”
6
Kim L. Jones CISM, CISSP, CRISC, MSIA
Contraxiom #1 -- Work
Geeks
7
Non-Geeks
• For Geeks, Work is
about solving
problems
• For Non-Geeks, Work is
about achieving a
vision
• Problems organize our
thinking and provide a
specific structure and
approach
• Visions are an
imagined experience
that get us out of bed
in the morning.
• Problem solving starts
in the present.
• Vision realization starts
in the future.
Kim L. Jones CISM, CISSP, CRISC, MSIA
Contraxiom #1 -- Work
• Impact on Metrics
– Do we truly understand the vision?
• And what the business must do/is trying to
do to achieve that vision?
– Are we relating our metrics TO the vision?
• This gives our metrics appropriate business
context (the “So What?” factor)
8
Kim L. Jones CISM, CISSP, CRISC, MSIA
Contraxiom #6 -- Lying
Geeks
9
Non-Geeks
• For Geeks, Lying is
evil. Truth is sacred.
• For Non-Geeks, Lying is
not good. Lying is bad
manners
• If you don’t know that
it’s true, and you say
it’s true, you’re lying.
• If you know that’s it’s
false and say it’s true,
you’re lying
• Exaggerations and
opinions stated as fact
are lies.
• Exaggerations and
opinions are part of
normal speech.
Kim L. Jones CISM, CISSP, CRISC, MSIA
Contraxiom #6 -- Lying
• Impact on Metrics
– If exaggeration is normal speech, are our
“metrics” accurate or exaggerated?
• Business can/will ask this…after all, “spin” is
natural
– When asked for specifics re: what will happen,
are our qualifications of answers view as lack of
commitment to our metrics/statements?
10
Kim L. Jones CISM, CISSP, CRISC, MSIA
Asking the Right Question
• How close is the nearest
rebel encampment?
• Are there mines on the
road?
• What is the current state
of rebel supplies?
• Is the destination still
neutral
Is The Road Open?
11
Kim L. Jones CISM, CISSP, CRISC, MSIA
Asking the Right Question
Are We Secure?
Are We Compliant?
What Is The Current Level of Risk?
Are Our Controls Sufficient?
Is The Risk Balanced Sufficiently To
Achieve Our Vision?
12
Kim L. Jones CISM, CISSP, CRISC, MSIA
Random Thoughts…
• Compliance Isn’t Always Bad
• Testing the Hypothesis
• Making the Subjective Objective
• Data Visualization Principles
13
Kim L. Jones CISM, CISSP, CRISC, MSIA
Compliance Isn’t Always Bad
• Executives latch on to compliance because it meets
the requirements of a good metric.
• The problem (as we all know) is that compliance
doesn’t equal security
– Worse, compliance does not equal
appropriately balanced risk
• Even if you win the metrics battle, compliance will
remain an issue if you are a regulated entity
• Possible (useful) workaround: measuring
compliance with your policy framework
14
– Meets compliance standards
– Sets the risk floor!
– Is in line with the vision!
Kim L. Jones CISM, CISSP, CRISC, MSIA
Testing the Hypothesis…
• Gathering metrics to test hypothesis can be very
useful when looking to ascertain and solve
problems in your network.
• All previous rules re: metrics, context, etc. apply
• Remember: don’t prove the positive…disprove the
negative.
15
Kim L. Jones CISM, CISSP, CRISC, MSIA
Testing the Hypothesis…
• Corporate Mission: “Enable a Better Way for
Trusted Commerce
• Infosec Mission: “We ensure the Trust in Trusted
Commerce”
– Trust defined as: your transactions will process as
expected, when expected, how expected (i.e., without
alteration).
• Hypothesis: Our Transactions Can be Trusted
– Sub-Hypotheses:
• There are limited points of entry through which an outsider can get
into our information systems
• Once inside, attackers cannot obtain access to internal systems
because of strong passwords
• An intruder finding a hole somewhere cannot jump to core
transactional systems
• Administrative credentials are difficult to obtain
16
Kim L. Jones CISM, CISSP, CRISC, MSIA
Testing the Hypothesis:
Disproving the Negative
• The network is porous,
permitting easy access to any
outsider
• There are limited points of
entry through which an
outsider can get into our
information systems
• Attackers can obtain access to
internal systems because of
password policies are weak
• Attackers cannot obtain
access to internal systems
because of strong
passwords
• An intruder finding a hole
somewhere can easily jump
straight to core transactional
systems
• Once on the network,
attackers can easily obtain
administrative credentials
17
• An intruder finding a hole
somewhere cannot jump
to core transactional
systems
• Administrative credentials
are difficult for attackers to
obtain
Kim L. Jones CISM, CISSP, CRISC, MSIA
Testing the Hypothesis:
Diagnostic Questions
• The network is porous,
permitting easy access to any
outsider
•
•
•
• Attackers can easily obtain
access to internal systems
because of password policies
are weak
• An intruder finding a hole
somewhere can easily jump
straight to core transactional
systems
• Once on the network,
attackers can easily obtain
administrative credentials.
18
How many sites are connected directly to the
core network without intermediate
firewalls?
How many sites have deployed unsecured
wireless networks?
Starting with zero knowledge, how many
minutes are required to gain full access to
network domain controllers?
•
What percentage of accounts could be
compromised in <15 minutes?
•
How many internal zones/subnets exist to
compartmentalize the environment?
•
How many administrative-level passwords
could be compromized in the same time
frame?
How many universal administrator accounts
exist in the environment?
•
Kim L. Jones CISM, CISSP, CRISC, MSIA
Making the Subjective Objective…
• One of the complaints re: security metrics is an
inconsistency in measurement
– This undermines even the strongest/most significant
metric as being opinion versus fact.
• Semi-qualitative metrics are a good starting
point…but consider going a step further and
implementing a standard evaluation checklist with
relative values.
• Plotting the results of multiple assessments over a
specific population may create a contextually
relevant metric
19
Kim L. Jones CISM, CISSP, CRISC, MSIA
Making the Subjective Objective
20
Kim L. Jones CISM, CISSP, CRISC, MSIA
Data Visualization Principles
1. It’s All About The Data, Not the Design
– Pretty designs and backgrounds are fun, but
they exist to enhance the data, not overwhelm
it
2. Simple Is Better
– Erase what you don’t need
– Avoid 3-D
– Hint: Wizards aren’t necessarily helpful
3. Simplify the Color Palette
– Muted, Primary Colors
21
Kim L. Jones CISM, CISSP, CRISC, MSIA
Data Visualization Principles
4. Label Honestly and Accurately
– Titles should be meaningful
– Labels should enhance understanding
– Always identify units of measure
– Avoid clutter
5. Consider the Best Depiction of Data
– Pie Chart? Stacked Bar? Pareto?
6. Test the Data!
– Grant’s Captain
22
Kim L. Jones CISM, CISSP, CRISC, MSIA
Wrapping it Up…
• Security is, at a fundamental level, a state of mind
– Ditto for balanced risk
• It stands to reason, then, that measuring security
and/or risk can be like catching a moonbeam
– “What is the sound of one hand clapping?”
• Metrics and measurement are both art and
science…you need to study both
• Make your metrics contextually relevant
– What’s the vision?
Be sure you’re answering the right question!!
23
Kim L. Jones CISM, CISSP, CRISC, MSIA
Questions?
Kim L. Jones CISM, CISSP, CRISC, MSIA
Contact Data…
Kim L. Jones CISM, CISSP, CRISC, MSIA
(480) 253-9120
[email protected]
25
Kim L. Jones CISM, CISSP, CRISC, MSIA