What Would You Say You Do Here?
Explaining Security to Management – and Vice Versa
Presented at the Rochester Security Summit 2016
Presented By:
Matt DeMatteo
Principal Security Architect, SecureWorks
Connect with me on LinkedIn
Note: This slide deck has been modified from the presented version to retain only the self-learning
content.
What Does Security Look Like?
What would
you say...
you do here?
How Can We Explain Security to Management?
400 BCE
800s
1846
1942
1953
2015
The field of medicine has evolved over thousands of years. During
that time, advancements have come through building a universal
mission statement, building schools, experimenting with simple
solutions to durable problems, and leveraging innovations to
achieve better and more efficient outcomes.
1760
1778
1790
1960
1995
2010
Similarly, manufacturing has evolved through incremental
advancements but over a much shorter timeline. Recent
innovations revolve largely around measuring process steps and
driving changes to improve those measurements.
1494
1602
1642
1938
1960s
2000s
Modern business has evolved too. Basic accounting is over 500
years old and business and management has evolved along with it.
If the security industry is going to live up to the challenges of
security, it needs to evolve INSIDE a business environment – and it
needs to speak the language of business – that language is
numbers - metrics
Metrics
Met∙rics
/’metriks/
A method of measuring something, or the results obtained
from this.
3 Steps to Developing Metrics
1. Identify process or input
2. Identify quantifiable outputs or meta-data about process
3. Define rules to evaluate output and evaluation cadence
Top 5 Source Countries by Volume
Country
Event Volume (90 days)
United States
4,758,274,860
Germany
28,375,892
Japan
22,838,278
China
7,948,472
Spain
598,584
Let’s take a simple, common security metric and improve it.
First, we should remove the US from this top 5 list – its value is too
large to compare to the other data points.
Top 5 Non-US Source Countries by Volume
Country
Event Volume (90 days)
Germany
28,375,892
Japan
22,838,278
China
7,948,472
Spain
598,584
Ukraine
564,567
Now we are looking at data that is roughly at the same scale, but
large numbers don’t tell stories – they are security theatre. Modern
accounting rounds to the nearest $100k or even $1m. We should
modify this data to make it easier to relate to and compare.
Top 5 Non-US Source Countries by Percentage
Country
Event Volume (90 days)
Germany
35%
Japan
28%
China
10%
Spain
<1%
Ukraine
<1%
Rest of the World
>26%
This data now tells a better story. We added the “Rest of the
World” to show the rest of the “pie” and now our main data points
are comparable. But the data still doesn’t tell a story to anyone
OUTSIDE of security…
InfoSec Policy Change Recommendations
Country
Country Risk Event Volume (90 days)
Recommended Action
Germany
Low
35%
None
Japan
Low
28%
None
China
High
10%
Restricted Web Ingress
Spain
Medium
<1%
None
High
<1%
None
Medium
>26%
None
Ukraine
Rest of the World
Restricted Web Ingress for China
For non-2FA external facing applications used by employees and
partners, shun .CN based sources
We add two things and now we have a very different piece of
information. We add “Country Risk” based on our security
expertise. We also added a recommended action right in the table
and an explanation of that recommendation.
InfoSec Policy Change Recommendations
Country
Country Risk
Event Volume (90 days) Volume Growth (Q/Q)
Recommended Action
Germany
Low
35%
+5%
None
Japan
Low
28%
-1%
None
China
High
10%
+235%
Restricted Web Ingress
Spain
Medium
<1%
-50%
None
High
<1%
+512%
None
Medium
>26%
-2%
None
Ukraine
Rest of the World
Restricted Web Ingress for China
For non-2FA external facing applications used by employees and
partners, shun .CN based sources
We can go the extra mile, too. Let’s compare today’s data to our volume
growth quarter over quarter. Now we’re showing management that a)
we’ve thought about our recommendation b) provided evidence and c)
showed we’ve shown restraint in the recommendation.
All Metrics Are Not Created Equal
Metrics
Data Points
• Tactical
• Granular
• “So what?”
Key Performance
/Risk Indicators
• Strategic
• Roll-Up
• “So what!”
Service Level
Agreement/Objective
• Strategic
• Roll-Up
• “What now?!”
Key Performance Indicator (KPI)
A metric designed to accurately represent the
effectiveness or not of a particular process*
3 Steps to good KPIs
1. Have a defined process*
2. Have defined requirements or evaluation criteria for the
process
3. Have quantitative or qualitative measurement(s) for the
outcome of the process
*A control system is a fully automated process
KPI
KPI designed to reflect the aggregate effectiveness
of our monitoring and alerting strategy.
• Green – Aggregate True Positive >= 95%
• Amber – Aggregate True Positive >= 90%
• Red – Aggregate True Positive < 90%
Data Points
Key Risk Indicator (KRI)
A metric designed to accurately represent the risk or
“risky trend” in a process* or a metric/KPI
A metric can be both a KPI and a KRI…but probably not at
the same time.
3 Steps to good KRIs
1. Define the metric based on a KPI component or another agreed
upon business goal/objective
2. Balance KRIs holistically…not everything can be the biggest risk
3. Define KRIs better than KPIs…KRIs are scrutinized more
*A control system is a fully automated process
Service Level Agreement (SLA) and
Service Level Objective (SLO)
A set of quantitative and qualitative metrics tied to
specific processes with defined and agreed upon input and
output.
SLAs are generally punitive. SLOs are not.
3 Steps to good SLAs/SLOs
1. All stakeholders are involved in the definition process
2. Out of scope factors are defined AND included in process.
3. SLAs/SLOs should roll into KPIs
Dashboard
A single page/screen/view that explains the results of a
process or set of processes over a defined timeframe or
real-time.
3 Steps to good dashboards
1. You can explain all data on the dashboard
2. It contains mostly KPIs, KRIs, and their aggregate parts*
3. It contains brief commentary
4. It is a single page/screen/view…One!...I mean it!
*Or SLAs/SLOs with timers for real-time
The Parts
People
Process
Technology
The Plan
Design
Build
Run
The Payoff
KPIs
KRIs
ROI
Return on Investment (ROI)
In the security industry, a mythical goal that presumes a
level of performance can be achieved that saves/finds
more money than was given.
3 Things to Replace ROI with
1. Risk reduction
2. Productivity achievements
3. Program maturity goals
The Parts
People
Process
Technology
The Plan
Design
Build
Run
The Payoff
KPIs
KRIs
Outcomes
then
now
Security doesn’t have a 1,000 year history, or a 100 year history – it
probably has a 20 year history at best and let’s be honest, learnings
from more than 5 years about security have degrading value.
now
YOU ARE HERE!
That means the history of security hasn’t been written.
It means that it is up to everyone in the industry to mature our
practices. As long as businesses are responsible for their own
security, it is our responsibility to speak about security in the
language of business – that language is numbers – metrics.
then