Optimizing Performance Management
Gina Fisk, LANL Senior Cyber Security Manager
[email protected]

Adaptive Metrics


Fitness Functions



Develop metrics that determine how well we
are adapting to our ever-changing
environment.
Identify dependencies and requirements for
optimum productivity around the Laboratory.
Measure the impact of a localized failure of
one entity across the entire organization.
Balanced Score Card


Review our program from a balanced
perspective.
Provide metrics by which we can manage.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
Strategy
Measures
Budgeting
Initiatives
Starting Point – Remove the Clutter

Remove metrics that we can’t use to manage our
information security program.




How many customers called our help desk.
How many connections were deflected by our firewall.
How many times our network was scanned, etc.
Bin the remaining metrics into the BSC framework for
a Phase I BSC.




Financial.
Customer.
Internal Processes.
Learning and Growth.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
2
Determine Impacts of Failure

Conduct IT Impact Analysis

Determine the cost to an organization if various IT services
failed for variable lengths of time.


Calculate Impact Rating for each IT Service.



Network, Email, local storage, etc.
1/n, where n is the average number of days until an
organization has lost 100% of productivity.
Calculate the Daily Monetary Impact of the Loss of
that IT Service for an organization.
Calculate the overall productivity cost for the
Laboratory as a whole based on that loss.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
3
Focus Areas

From IT Impact Analysis results, identify IT Services
with largest impacts to productivity.







Loss
Loss
Loss
Loss
Loss
of
of
of
of
of
Accreditation of systems
local network access
Email
Oracle
Internet access
Goals that the CIO and CISO had set for the
organization in the Strategy Map.
Develop metrics based on these focus areas and
develop Phase II of the BSC.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
4
Information Security Strategy Map
Maximize mission enablement
by balancing risk and value (F1)
Operational Excellence
F2. Maximize
operational efficiency
Competitive Advantage
F3. Minimize IT
enterprise risk
F4. Facilitating acquisition of new business
through best-in-class IT security execution
Competency
C1. “Understand and
consistently deliver
what I need”
Contribution
C2. “Keep me out
of security and
compliance trouble”
C3. “Establish a
positive reputation
which will help me with
my customers”
C4. “Become a trusted
partner by helping me
solve my challenging
problems”
Achieve Operational
Excellence
Create and Support Internal
Programs and External Partners
Deliver Innovative Security
Solutions
IP1. Streamline compliance
program to achieve 100% of
scheduled accreditations
IP2. Optimize operations to
reduce KTLO by 10% per BU
IP3. Enhance performance
through implementation and
management of service
agreements
IP4. Mature IT governance
processes and increase partner
participation
IP5. Build a structured, transparent
and collaborative regulator
relationship
IP6. Promote transparency and
performance through holistic
metrics program
IP7. Propose and deliver
business-enabling information
security solutions
IP8. Mature IT risk program to
drive security, portfolio, and
governance decisions
IP9. Enhance red network
monitoring and vulnerability
management
LG1. Attract, develop, and
retain highly skilled security
professionals
LG2. Develop risk-focused
and customer-centric culture
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
FINANCIAL
PERSPECTIVE
LG3. Align employee
training with strategic
initiatives
CUSTOMER
PERSPECTIVE
INTERNAL AND
PROCESS
PERSPECTIVE
LEARNING
AND GROWTH
PERSPECTIVE
5
Balanced Score Card
Note: BSC target performance scores are represented here for explanatory purposes only
Customer (C1-C4)
Financial (F1-F4)
Security unit On-time rate of Enterprise risk
costs
accreditations
rating
Lower unit
costs
100% on time
Maintain .3
rating
Business
impact of
incidents
<25hrs/Q
Projects ontime/budget
Cyber PBI
ratings
Communication Compliance
Customer
Support
Program Input
Time per
accreditation
Customer
Satisfaction
>90%
>80% survey >70% survey >80% survey
>95% CA/avg >80% survey
governance
scores
scores
scores
times
scores
participation
<10% variance >95% green
Target
Target
Initiative
Initiative
Internal Processes (IP1-IP7)
Learning and Growth (LG1-LG3)
Training
roadmap
Planned role
rotations
Attrition
reduction
Strategic
training
X
X
AOE: Opex
reduction
<10%
schedule
variance
>=1/Q
Reduced
attrition rate
>50% training
mapped to
initiatives
X
X
>=2.5% Q/Q <10% variance
Target
Target
Initiative
Initiative
Hits target.
Initiative on track
Short of target.
Initiative recoverable
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
AOE: SLA
CSIPP:
DISS: AOP risk DISS: BP tied
performance unplanned work mapping
to risk
Failed process.
Initiative not recoverable
<=3/Q
>=80%
DISS: Red
capabilities
>=30% key
Positive trend
processes
Target not
defined. No
initiative
6


Fitness functions measure the overall
health of an organization by measuring
not only performance, but also the
performance of those organizations on
which we are dependent to achieve our
goals. If the performance of one of the
dependencies fails, there are
ramifications throughout the entire
organization.
Using the fitness scores of dependent
organizations, we can measure the
impact of a localized failure of one entity
across the entire organization, providing
valuable measurements of the actual cost
of security incidents, network outages,
etc.
We can trend these scores to evaluate
performance at various levels of the
organization.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
Contracts
Science and Engineering
Production
Cycles
Backups and
Storage
Visualization
Services
Reliant Organizations

Publications
Scientific Computing
Network
Services
Identity
Management
System
Administration
Dependencies
Fitness Functions
Patents
Core Services
Security
Infrastructure
C&A
Physical
Infrastructure
7
Example Fitness Function Framework






Fiscal Responsibility (weight: 20%)

Milestones and deliverables (quality, timeliness)

Expenditures (percentage over budget)
Customer Productivity (weight: 15%)

Services maximize productivity around organization (uptime, etc)
Customer Orientation (weight: 15%)

Responsiveness to the customer (SLAs, etc)
Improving Security (weight: 15%)

Progress made toward improving security against our current threat
environment (hardening tools, etc)
Institutional Responsibilities (Weight: 20%)

PBI deliverables and reporting (quality, timeliness)

CAP deliverables and reporting (quality, timeliness)

Metrics reporting (quality, accuracy)
Goal-Based Initiatives (weight: 10%)

Progress made against organizational goals.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
8
Fitness Function Example
Note: Fitness scores are represented here for explanatory purposes only







Fiscal Responsibility (weight: 20%)
.89

Timeliness of deliverables and milestones:
.83

% of projects +/- 10% of budget allocation:
.95
Laboratory Productivity (weight: 15%)
.98

Uptime of service:
.98
Customer Orientation (weight: 15%)
.89

Customer Satisfaction Rating
.89
Improving Security (weight: 15%) **
.56**

Progress made toward improving security against our current threat environment
(hardening tools, etc)
.56
Institutional Responsibilities (Weight: 25%)
.68

PBI deliverables and reporting (quality, timeliness)
.90

CAP deliverables and reporting (quality, timeliness)
.75

Metrics reporting (quality, accuracy)
.40
Goal-Based Initiatives (weight: 10%)
.98

Progress made against initiatives.
.98
FITNESS SCORE:
.806
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
9
Fitness Score Trends

Fitness scores allow us to watch for trends and to
manage by our metrics.

See how major changes affect our performance from
month to month.



Change in Management
Change of Platform
Change of Vendor, etc.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
10
Adaptive Metrics



IT Impact Analysis provide us with costs of the
failures of IT Services.
We have the data on our ever changing threat
environment.
The fitness functions allow us include “moving target”
metrics, which change each month, to measure our
performance against our current threat environment.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
11
**Adaptive Metric Example
Note: Threats and budgets represented here for explanatory purposes only

“Improving Security (weight: 15%)”




Identify top threats for the month.
 Phishing, Windows vulnerability, Oracle vulnerability.
Calculate cost of failure of these services across the organization
per month.
 Email: $200K, Windows: $500K, Oracle: 800K
 Overall Budget: 10% in jeopardy
Review % of security effort we are placing on these areas ($$$
spent).
 Email: 5%, Windows: 31%, Oracle: 20%
Weight the fitness function by how responsive we are to these
areas.
 56% of our budget is spent on our top threat areas.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
12
Managing by the Metrics



Our budget, metrics, and initiatives are actionable
and directly tied to our goals.
Our use of the Balanced Score Card helps us ensure
uniform management of our business.
Our use of the Fitness Functions help us trend our
metrics effectively and monitor the major changes.


We can trend our components individually or as a whole,
organizationally or institutionally.
Our use of Adaptive Metrics keep our outlook fresh
and defendable.
UNCLASSIFIED
Operated by Los Alamos National Security, LLC for NNSA
13