Air Force Institute of Technology
Integrity - Service - Excellence
Foundations for
Measuring Effectiveness
Major Richard ‘Kelly’ Bullock
Dr Richard F. Deckro
Department of Operational Sciences
Air Force Institute of Technology
Wright-Patterson AFB, OH
USA
30 August 2005
Research Motivation

Accurately predicting outcomes and assessing progress has challenged
leaders since the earliest use of national power to achieve political aims


Many theories on how to employ national power to achieve desired endstates


Especially true of military power
Douhet, Jomini, Trenchard, Mitchell, Schelling,…
Effects-based Operations

Not new, but efforts to ‘institutionalize’ EBO concepts are

US Joint doctrine and service doctrine, particularly USAF doctrine, has
undergone change to reflect EBO concepts

History has shown theory is of little value if not supported by an
empirically feasible measurement method
One accurate measurement is worth a thousand expert opinions.
– ADMIRAL GRACE HOPPER, 1906 – 1992
Integrity - Service - Excellence
2
Patton on MOEs…
Discussion between General George Patton and General Orlando Ward during WWII:
“How many officers did you lose today?” asked Patton.
“We were fortunate,” Ward replied. “We didn’t lose any
officers.” “Goddamit, Ward, that’s not fortunate! That’s
bad for the morale of the enlisted men. I want you to
get more officers killed.” A brief pause followed before
Ward said, “You’re not serious, are you?” “Yes,
goddamit, I’m serious! I want you to put some officers
out as observers,” said Patton. “Keep them well up
front until a couple get killed. It’s good for enlisted
morale.”
Integrity - Service - Excellence
3
Overview

Background

Measurement Concepts
 Measurement Theory

Application of Measurement
 Effects-based Operations
 Effects

Effectiveness Measurement Foundations

Definitions
 Concepts
 Framework

Takeaways
Integrity - Service - Excellence
4
Background
Measurement Concepts

Measurement is the objective representation of real-world objects, processes,
and phenomenon

Measurement captures information through attributes


A real-world system is defined by the attributes chosen to represent it


Attributes may not be directly observable
Measurement is an abstraction
Measurement is the process of assigning symbols to a system attribute such that
the assigned symbols reflect the underlying nature of the attributes
Set X
Set Y
E
1.2
A
2.5
B
8.9
3.1
5.4
3.7
C
6.2
D
measurement
Y  f (X )
Integrity - Service - Excellence
5
Background
Measurement Concepts
validity
– how
awell
measure
whatthe
it was
supposed
to represent
amplitude
– well
how
a measure
represents
higher
order constructs
reliability
– how
consistent
orreflects
repeatable
measurement
process
is
System
Measure
Assessment
Abstract
Attributes
Numbers
Reasoning &
Mathematics
Empirical
Object,
Process, or
Phenomenon
Numerals
Insights &
Information
= source for potential error
random
– ‘noise’
variation
from
any
impacting
the
system
observational
– oversight
of
a key
system
attribute
or source
using
the
wrong
measure
for a system process
attribute
systemic
– (bias)
derives
from
construction
of
the between
measure
or
definition
of the
measurement
Statistical
Theory
– concerns
making
inferences
from
data
Measurement
Theory
–
link
data
and
real-world
To measure is to know.
– LORD KELVIN, 1824 – 1907
Integrity - Service - Excellence
6
Background
Measurement Theory


Measurement Theory is a branch of applied mathematics that attempts to:

Describe, categorize, and evaluate the quality of measurements

Improve the usefulness, accuracy, and meaningfulness of measurements
Representational view of measurement built upon 3 theorems
Set Y
Set X

E
Representation – Proves the measure f
preserves the relationships in X after
mapping into Y (i.e. model validation)
1.2
A
2.5
B
8.9
3.1
5.4
3.7
C
6.2
D

Uniqueness – Characterizes the family
of measures that preserve the realworld relationships (i.e. scale type)
measurement
Y  f (X )
Nominal
Ordinal
Interval
Ratio

Meaningfulness – Shows resultant is
invariant to scale change for measures
meeting the uniqueness condition
Absolute
Integrity - Service - Excellence
7
Background
Application of Measurement


Application of measurement is…

an art

the mechanism for extracting information from empirical
observation

within a specified context

the translation of behavior into a set of ‘vital signs’ indicating
variations in behavior
Application guidelines

Frameworks

Vertical
 Horizontal

Effectiveness vs. Efficiency
 Measure Types & Characteristics
...things are to you such as they appear to you and to me such as they appear to me...
– PROTAGORAS, 485 – 421 B.C.
Integrity - Service - Excellence
8
Background
Frameworks
Measure of Effectiveness (MOE)
Objective31
Effectiveness
Vertical (or Hierarchical) Structure
What?
Fundamental
Objective
SubObjective
O1
Why?
SubObjective
O2
Value
V1
Value
V2
SubObjective
OX
Value
VY
Objective21
Objective32
Objective11
Objective22
Objective33
Objective12
Objective23
Objective34
Objective1M
Objective24
Objective35
Objective2N
Objective36
Objective3P
How?
Horizontal Structure
Attribute1
Attribute2
AttributeZ
Efficiency
Measure of Performance (MOP)
We must make the important measurable, not the measurable important.
– ROBERT MCNAMARA, 1916 –
Integrity - Service - Excellence
9
Background
Measure Types & Characteristics
Natural
Direct
Proxy
Constructed
- Commonly understood measures directly linked to
fundamental objective
- Example: Profit
- Measures directly linked to the fundamental
objective but developed for a specific purpose
- Example: Gymnastics scoring
- In general use measures focused on an objective
correlated with the fundamental objective
- Example: GNP (economic well being)
- Measures developed for a specific purpose focused
on an objective correlated to the strategic
objective
- Example: Student grades
Strategically-linked – traceable to fundamental objective; responsive to change and provides
indication of how much change can be attributed to an input
Timely – able to be collected and processed in a timeframe needed to be relevant within the context
Objective – 1) Collection: easy to understand, same regardless of the assessor, same under similar
circumstances; ‘face-value’ or whether the measure logically represents what it is supposed to
represent 2) Interpretation: unambiguous interpretation; distinguish between desired and
undesired consequences
Economical – Collection and processing should provide benefits that off-set the burden of
measurement activities
Complete – Measures should address all areas of concern in enough detail to discern reasons for
differences in actual and expected system results
Measurable – hold for Measurement Theory conditions; within a given context if the measure can be
feasibly obtained with available resources (i.e. operational )
Integrity - Service - Excellence
10
Background
EBO Overview

Effects-based Operations (EBO)

Theory for the employment of capabilities in dynamic and uncertain
environments in a manner to best attain objectives

Provides a conceptual framework for determining the integration and
application of capabilities to achieve specific effects to influence the
environment of interest yielding desired outcomes
“You begin with the objectives that indicate what you really care about. Then you
follow simple logical reasoning processes to identify the mechanisms by which the
objectives can be achieved. Finally, for each mechanism, you create alternatives by
asking what control you have over that mechanism.”
Keeney, Ralph, Value Focused Thinking: A Path to Creative Decision Making, Harvard University Press, Cambridge, 1992.

Key tenets

Focus on end outcomes

Reduced emphasis on weapon systems

De-emphasis on destruction as a sole means of achieving effects
Not everything that can be counted counts, and not everything that counts can be counted.
– ALBERT EINSTEIN, 1879 – 1955
Integrity - Service - Excellence
11
Background
EBO Overview
measures
measures
measures
measures
measures

measures
measures
measures
measures
measures
measures
Assessment
Planning
measures
Employment
Effects-based Operations
Measures
Effectiveness
–ifend-state;
tie
provide
feedback
on if
Assessment
–– determining
thepillars
intended
effects
were
achieved
and
Employment
efficient
employment
oftogether;
capabilities;
increased
Planning
–offocus
on desired
develop
strategy
(who, emphasis
what,
strategy
progress
towards
desired
end-state
they
arewhy,
shaping
the
desired
outcomes
on non-lethal
means
where,
when,
and
how)
Integrity - Service - Excellence
12
Background
Effects Overview

Effect

Consequence
System
state change
or result of a particular action
Tactical
Input
Input
Effect
Input
Direct
Parallel
Intended
Positive
Physical
Effect
Indirect
Parallel
Intended
Positive
Functional
Input
Input
Effect
Effect
Effect
Direct
Parallel
Intended
Positive
Functional
Effect
Indirect
Unintended
Negative
Psychological
Input
Input
Input
Effect
Direct
Parallel
Intended
Negative
Psychological
Effect
Effect
Input
Input
Effect
Input
Direct
Parallel
Intended
Positive
Psychological
Effect
Input
System
Boundary
Effect
Strategic
Operational
Input
Input
Attribute
Direct
Sequential
Intended
Positive
Functional
Indirect
Sequential
Cascading
Intended
Positive
Functional
Indirect
Sequential
Cascading
Unintended
Positive
Functional
Indirect
Sequential
Cascading
Intended
Positive
Functional
Effect
Direct (First-order)
Indirect (Higher-order)
Impact
Parallel
Sequential
Cascading
Direct (First-order) Cumulative
Intended
Intent
Un- intended (Collateral)
Indirect (Higher-order)
Positive
Indirect
Parallel
Cumulative
Timing
Effect
Intended
Positive
Systemic
Impact
Result
Parallel
Persistence
Sequential
Domain
Cascading
Level
Cumulative
Negative
Permanent
Non-temporal
Physical
Functional
Systemic
Psychological
Tactical
Operational
Strategic
Indirect
Parallel
Intended
IntentIntended
Effect
Positive Un-intended (Collateral)
Functional
Indirect
Unintended
Negative
Effect
Persistence
Psychological
Indirect
Sequential
Domain
Cumulative
Effect
Intended
Positive
Systemic
Indirect
Cascading
Unintended
Negative
Psychological
Types
Order
TypesTiming
Result
Indirect
Parallel
Intended
Positive
Psychological
Indirect
Unintended
Positive
Functional
Order
Attribute
LevelIndirect
Effect
Unintended
Negative
Psychological
Integrity - Service - Excellence
Positive
Negative
Permanent
Temporal
System
Behavior
Physical
Functional
Systemic
Psychological
Tactical
Mechanism
Operational
Strategic
13
Approach
Definitions
DEFINITION 1: A SYSTEM is a set of elements where relationships exist between the elements and the SYSTEM
has a purpose or normative behavior.
DEFINITION 2: A system ELEMENT, or SUBSYSTEM, is a system providing functionality or support to a parent
system.
DEFINITION 3: A MODEL is a formal image of an empirical structure.
DEFINITION 4: An ATTRIBUTE, or NODE, is a characteristic, feature, or property of a system that is directly or
indirectly observable.
DEFINITION 5: A MEASURE is a model of an attribute.
DEFINITION 6: A MEASUREMENT, or observation, is a particular manifestation, or instantiation, of an attribute.
DEFINITION 7: A system STATE is a particular instantiation of all system attributes, or state variables, at a
particular point in time.
DEFINITION 8: An EFFECT is a system state change.
DEFINITION 9: EFFECTIVENESS gauges the magnitude of a system state change.
DEFINITION 10:An END-STATE characterizes the desired measurements for all system attributes, or state
variables.
The beginning of wisdom is calling things by their right names.
– CONFUCIUS, 551 – 479 B.C.
Integrity - Service - Excellence
14
Approach
Product Structure
Product Structure Development Process
Empirical System
1. System Identification
• Determines system boundary
• Covers all pertinent aspects of desired end-state
2. Sub-system Identification
• Decomposes into smaller parts
• Only relevant sub-systems are needed
3. Sub-system Importance
• Discerns relative importance among selected sub-systems
• Amounts to weighting each sub-system
4. Attribute (Node) Identification
• Characterizes each sub-system by its salient features
• Only attributes relevant to context are needed
5. Attribute (Node) Importance
• Discerns relative importance among selected attributes (nodes)
• Amounts to weighting each attribute (node)
Attribute1
6. Measure Development
• Quantification of attributes (nodes)
• Constructed measures may be needed
Measure1
• Measures should hold for metric properties
Formal System
(model or product structure)
n

System
Sub-system1
i 1
sub-system
weight
Sub-systemn
m
Attributem
 attribute
weight
=1
i 1
q
Measure
Measure
q m
 measure = 1
i 1
Integrity - Service - Excellence
15
=1
Approach
Concepts
Time
t
Empirical
System
A = < aA
1,…, an >
t=0
Element
(Subsystem)
ai
t=T
Effect
(of inputt = 0 at t = T)
t = TEnd-state
Attribute
(Node)
Link
Input
Mechanism
System
Elements
Formal
(Subsystems)
System
Attributes
(or Model)
(Nodes)
xA = < x1, xx
xn-1, xn >
2,…,
A
Measures
System
Measurement
xA
x1
x2
1
ά1
xn-1
2
ά2
xA
xA
xn
n-1
άn-1
x1
x2
m
άm
1
ά1
xn-1
2
ά2
x1
xn
n-1
άn-1
1
m
άm
x2
ά1
xn-1
2
ά2
xn
n-1
άn-1
m
άm
Measurements
(Observations)
Measurements
(Observations)
Measurements
(Observations)
xA = < 0, 0,…, 0, 0 >
xA = < .2, .7,…, .3, .5 >
xA = < 1, 1,…, 1, 1 >
Effectiveness
(of inputt = 0 at t = T)
Count what is countable, measure what is measurable, and what is not measurable, make measurable...
– GALILEO GALILEI, 1564 – 1642
Integrity - Service - Excellence
16
Takeaways

Measurement turns real-world system behavior into a set of ‘vital signs’ for the
purpose of monitoring the system

Measurement Theory provides a robust basis for turning qualitative observations
into quantitative data

Effectiveness vs. Efficiency


Measure of Effectiveness (MOE)
 External measure
 Invariant to means of achievement
 Are we doing the right things?
Measure of Performance (MOP)
 Internal measure
 Coupled to means of achievement
 Are we doing things right?

EBO is an instantiation of an existing, structured, decision-theoretic framework
(VFT) for approaching problems with abstract or ill-defined objectives

An effect is a system state change

Effectiveness gauges the magnitude of the change
Integrity - Service - Excellence
17
Questions
&
Discussion
A prudent question is one-half of wisdom.
– SIR FRANCIS BACON, 1561 – 1626
Integrity - Service - Excellence
18