KPIs: Do They Really Work as Strategic Tools for Evaluation and RBM?
Revised Discussion Paper based on a Presentation to the MES Evaluation Conference
Kuala Lumpur, March 28, 2006
Jerome A Winston (Jerry) & Ranjit Bhagwandas
Program for Public Sector Evaluation International, Australia
E-mail: [email protected] & [email protected]
Table of Contents
Table of Contents
1
Abstract
1
Introduction
2
Issues Raised by the Attachments
2
Attachment A
Has there been Much International Experience Using KPIs and PIs as
Strategic Tools?
2
Attachment B
Are KPIs, PIs and PM in Common Use around the World?
4
Attachment C
Why is there Confusion about the Meaning of the Terms Performance,
Indicator and Measurement?
6
Attachment D. Why is there so Much Disagreement about the Features of a True KPI,
PI or PM?
8
Attachment E: Do Governments and Agencies Have Problems when Attempting to Use
KPIs as Strategic Tools?
9
Attachment F: Bibliography/References
11
Abstract
KPIs (Key Performance Indicators) are one of a range of tools that are intended to be used to – among other
things – improve strategic aspects of program planning, monitoring and evaluation. This paper reviews some
international experience using KPIs. This review highlights confusion about the meaning of the terms
performance, indicator and measure/ment. This confusion leads to multiple and conflicting standards for a
true KPI. Recent manuals and guidelines published by donor agencies have tended to avoid referring
explicitly to KPIs and performance indicators. The review of international experience identifies widespread
concern that using KPIs in conjunction with quantitative, operational targets promotes “data gaming” and “goal
displacement”. There is also widespread concern that – regardless of the stated goal to focus RBM (ResultsBased Management) and KPIs on strategic aspects of planning, monitoring and evaluation – what occurs in
practice is that KPIs often focus on little more than numerical, operational measures. One solution to this
problem is to define KPIs so that they focus on evaluation questions that enquire about all levels of program
and project result, rather than on predetermined, numerical measures. In this way, both evaluation questions
and KPIs will focus monitoring and evaluation on both strategic and operational aspects of program results.
This approach is consistent with using the ProLLTM model and the ProLLTM Results Ladder (Rasappan, 1994;
2003) to design KPI systems that do, indeed, work as strategic tools for evaluation and RBM.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 1 of 14
KPIs: Do They Really Work as Strategic Tools for Evaluation and RBM?
Introduction
Our conference presentation responds to the question, “KPIs: Do They Really Work as
Strategic Tools for Evaluation and RBM?” There is a simple – and trivial – response to this
question: “Yes, of course, KPIs really work as strategic tools to improve evaluation and
Results-Based Management (RBM) – providing we select the ‘correct’ KPIs.” This is a
trivial response to a serious question, because we would then be left to ask, “And how to we
identify the ‘correct’ KPIs that really work as strategic tools to improve evaluation and
RBM’?” To help us to answer this latter question, we assembled the six Attachments to this
paper, based on our review of literature from different decades, different sources and
different countries.
Issues Raised by the Attachments
Three inter-related issues caught our attention as we prepared the Attachments to this
discussion paper.
Issue 1. Inappropriate “data gaming” and “goal displacement” are now widely recognized
for their ability to undermine any attempt to use KPIs to improve strategic management,
monitoring and evaluation.1 This is particularly likely to become a major problem when a
program is funded or when managers and staff are rewarded on the basis of how well the
program meets simple, numerical objectives and targets.
Issue 2. Most of the approaches to Results- Based Budgeting (RMB) that we reviewed
provide little, if any, advice about how to reduce the risk of inappropriate data gaming and
goal displacement.
Issue 3. Guidelines, manuals and training often provide managers and staff with
considerable advice about how to measure and report simple, numerical operational inputs
and processes (i.e., management and staff operations). By contrast, managers and staff
appear to receive comparatively little, if any, instruction – or resources – to prepare them to
measure, analyze, interpret and report on the achievement of a program’s strategic
outcomes and impacts.
One possible way to reduce the risk of inappropriate data gaming and goal displacement is
to define KPIs so that they are well integrated with operational and strategic key result
areas (KRAs) as well as key evaluation questions (KEQs). This approach is particularly
useful when a program is described in terms of a systems model that includes an INPUT
system, PROCESS system, OUTPUT system, OUTCOME system, ENVIRONMENT and
FEEDBACK. For example, this approach to linking KRAs, KEQs, and KPIs is used by the
Program Logic and Linkages (ProLL) Model and the ARTD Results Ladder (Rasappan,
1994; 1999, 2003).
1
The impact of data gaming and goal displacement is not limited to their impact on strategic management, monitoring and
evaluation. Data gaming and goal displacement can just as readily undermine attempts to use KPIs to improve
operational management, monitoring and evaluation.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 2 of 14
There is an ongoing need for examples – based on real-world experience – that
demonstrate how KPIs may be linked to KRAs and KEQs. MAMPU has contributed to
meeting this need in its recent KPI circular (MAMPU, 2005). This circular includes an
outstanding, real-world example of how to develop and use KPIs to report, in detail, on
operational activities and results. The same attention to detail should be extended to
reporting on other aspects of program results, in order to extend the benefits of using KPIs
to reporting on strategic outcomes and impacts.
Attachment A.
Has there been Much International Experience Using KPIs and PIs
as Strategic Tools?2
Exhibit A1, below, describes a range of international experience using performance indicators and
measures in the public sector. Their use has been promoted since the 1970s as one way to
encourage public sector managers and staff to give more attention to asking, and answering,
strategic questions such as: Why is the government or donor agency funding this program? What
problems will this program solve, or at least modify? Which stakeholders' needs will be met as a
result of this program? Which policy outcomes and impacts should be – and have been –
achieved?
Throughout the 1960s and 1970s, in many parts of the world, public sector managers and their staff
were not focused on these strategic questions. Rather, they saw their primary job as responding to
short-term, operational questions such as: How much is left in our budget? What are our unit
costs? How many service providers were on duty? How many clients did we process? How
many files did we close?
Many public sector management reforms have relied on rational planning frameworks such as
program logic (e.g., Örtengren, 2004; Peters, 1989) to identify linkages between operational and
strategic aspects of program planning, monitoring and evaluation. Once these linkages had been
identified, program logic frameworks were often used to identify KRAs (key result areas) as well as
relevant indicators and measures. Some, or all, of these indicators and measures might be further
described as performance indicators (PIs) and performance measures (PMs).3 Some or all of
the PIs and PMs might also be described as KPIs (key performance indicators) and KPMs (key
performance measures).
Once the relevant indicators and measures had been identified, the program logic framework could
then be used to identify the methodologies that would be used to collect relevant performance data
(PD). The performance data would need to be analyzed, interpreted and reported. This
performance information would, among other things, shed light on the extent to which programs
were achieving their strategic (i.e., policy) goals. This information would be particularly useful to
senior managers and executive, policy advisors, budget and finance officers and auditors, as well as
to line managers and staff. Such at least was the theory that underpinned many of the management
reforms that are described in Exhibit A1.4
2
Although the focus of this paper is on the use of KPIs as strategic tools, we also include discussion of Performance
Indicators (PIs), Performance Measurement (PM), Performance Measures (PMs), as well as KPIs. We broaden our scope
because what is called a KPI in one location or agency may well be known elsewhere by another label, e.g., Indicator,
Performance Measure or PI.
3
Some organizations referred to these indicators as performance indicators or performance measures, while others did
not. See Attachments C and D regarding the impact of competing interpretations of the terms performance, indicator,
measure/ment.
4
Translating this theory into practice was often fraught with problems. See Attachment E.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 3 of 14
Exhibit A1: Examples of International Experience Using KPIs, PMs and PIs
USA
Various levels of government in the USA have required the reporting of KPIs, PIs or PMs since the
1960s. The “Government Performance and Results Act (GPRA)” is a recent example of mandating the use
of strategically-focused “outcome performance measures”, rather than operationally-focused “process and
output measures” (Radin, B. A, 2000).
Victoria
In 1982 the Victorian state government in Australia introduced Performance Indicators in the
context of the state government’s new Program Budgeting system. This “management reform” was taken
so seriously that the Victorian Parliament commissioned an independent evaluation of the success of the
first five years of this initiative.5
Australian Public Service
In 1984, the Australian federal government introduced Performance
Indicators in the context of the federal government’s new Program Budgeting system. The Commonwealth
Department of Finance, like the Canadian Comptroller-General, sought repeatedly to distinguish carefully
between true strategic performance and other, operational performance. For example, DoF published the
following advice in 1999, which repeated advice that had been published in 1986 in an earlier set of
publications on Program Budgeting (DoF, 1989):
New South Wales
In the early 1990s, the New South Wales state government in Australia introduced
the use of Performance Indicators as part of their commitment to better program evaluation. Somewhat akin
to the distinction made by the Canadian government (see above), NSW government training distinguished
between true, strategic Performance Indicators, on the one hand, and operational measures of processes, on
the other. These operational measures were referred to measures of “busyness”, not of true performance:
Performance indicators define how the success of a program is to be measured. (NSW, 1992:45)
Canada
By the mid-1970s the Canadian government, through the Comptroller-General, was
actively promoting the reporting of Performance Indicators. The Canadian government sought to
distinguish between two kinds of performance reporting: Administrative performance was associated with
indicators and measures of program operations, while true performance was linked to indicators and
measures of strategic impacts and outcomes. The government sponsored a number of reviews of the
Canadian initiative, including a 1991 review. (Comptroller General, 1991) In Australia, some years later,
the NSW and Australian federal governments sought to make a similar distinction between true
performance indicators, on the one hand, and operationally-focused indicators and measures of inputs and
processes, on the other.6
New Zealand
In the early 1990s – under guidance provided by the NZ State Services Commission
and following the recognition of problems associated with earlier New Management Reforms (NMRs) in
Europe, the NZ government combined a version of program logic with the reporting of performance
measures of nominated “output classes” to monitor and evaluate progress towards achieving each Ministry’s
strategic goals and objectives. This approach expected line managers and staff to focus primarily on
operational measures, because each Ministry was intended to have already established the link between the
operational outputs that government purchased and the achievement of the government’s strategic goals
and objectives.
International Donors
For more than 25 years, international development agencies have incorporated
KPIs or PIs as part of their requirement that projects and programs use Program Logic (ProLog)
Frameworks for ensuring that strategic goals and objectives are being met. For recent examples of linking
Program Logic (ProLog) to KPIs, PIs or PMs, see W. K. Kellogg Foundation (2001) and Örtengren, K.
(2004).
5
See Corbett (1989) and Economic and Budget Review Committee (1990).
6
See other Attachments for additional information about attempts to distinguish between operational and strategic
measures and indicators.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 4 of 14
Exhibit A1 (continued)
Worldwide Training During the last 25+ years, there has been a seemingly endless flow of conferences,
workshops and lectures, spanning much of the globe, promoting the use of one or another version of KPIs,
PIs and PM to focus the attention of managers and staff on strategic aspects to program planning,
management, monitoring, evaluation and reporting.
Exhibit A1 Program logic frameworks, KPIs, PIs and PM have been central features of numerous government
reforms during the past 25+ years.
Attachment B. Are KPIs, PIs and PM in Common Use around the World?
Attachment A provided evidence that KPIs, PIs and PMs have been central to various public sector
management reforms that have emphasized RBM and program evaluation. However, in recent
years, in many countries, and in many agencies within countries, and within many publications by
donor agencies and governments, the terms KPI, PI and PM are all but absent.
Exhibit B1, below, describes publications by donor agencies in which the terms key performance
indicator (KPI), performance indicator, and performance measure/ment are either completely
absent or are used only once. The absence of these terms may surprise delegates to a conference
that highlights the potential contribution of KPIs to strategic aspects of planning, monitoring and
evaluation.
The publications described in Exhibit B1 are also interesting to us, because some of them make
recommendations that we believe represent significant improvements in the way KPIs are defined
and applied. Some publications recommend the use of both qualitative and quantitative methods
when reporting on program performance. This is a change from earlier times when many authors
and consultants presumed that SMART measurement does not allow for the use of qualitative (nonnumerical) methods. These Exhibits may indicate that there is growing acceptance that SMART
measures need to be verifiable, which does not mean that they must be numerical. There is no
reason why performance measures and indicators could not be verifiable whether they are measured
and reported qualitatively or quantitatively.
These publications are also important because they demonstrate, as a group, the difficulty involved
in trying to explain the relationships among the terms indicator, measure (as a noun or verb),
measurement, data, information and report. For example, it was not always clear how many of the
meanings shown below were intended when the term indicator was used in these publications:

An indicator is a concept or attribute such as
client satisfaction or poverty reduction which
– potentially – could be measured and
reported in number of different ways.

An indicator provides not only the name of a
concept or attribute, but also identifies the
general methods that must be used to
collect the data.

An indicator is not only the name of a
concept or attribute, but also identifies the
specific tools, techniques, or instruments
that must be used to collect the data.

An indicator is not only the name of a
concept or attribute, but also identifies the
specific raw data that must be collected.
 An indicator is not only the name of a
concept or attribute, but also provides a
summary of the raw data that were
collected.
 An indicator not only the name of a concept
or attribute, but also provides a report of the
analysis of the raw data that were collected.
 An indicator is not only the name of a
concept or attribute, but also provides an
evaluation of aspect of a program or
project.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 5 of 14
In our opinion, the publications mentioned in Exhibit B1 tend to underplay the contribution of data
analysis to transforming raw performance data into useful performance information. This lack of
discussion about data analysis may lead some readers to conclude that performance data will all but
speak for themselves, i.e., that their main task is to collect (valid and reliable) data about
performance, because transforming this data into useful performance information will be reasonably
straightforward. One government manual, some years ago, offered just such advice. According to
this manual, performance data – presumed to be quantitative – can almost always be transformed
into useful performance information by calculating nothing more complex than ratios, rates,
percentages and totals (NSW, 1992).
We do not agree that this advice is helpful. According to empirical research, the use of such simple
numerical measures – especially when used to reward program managers and staff for meeting
(numerical) performance targets and goals – is likely to increase the risk of inappropriate “data
gaming” and “goal displacement”. Feeling under pressure to meet simple numerical targets and
goals, managers and staff may focus on how they can ensure that their performance measures “look
good” – regardless of the impact this narrow focus has on their ability to meet stakeholders’ needs.
Fortunately, some of the publications mentioned in Exhibit B1 refer to need to reduce the risk of data
gaming and goal displacement. We welcome these references. However, these publications do not
explore in any detail how to reduce these risks by changing the way in which performance indictors
and measures are defined, or how they are reported.
Exhibit B1: How KPIs and Similar Indicators & Measures are Described by International Donors
Denmark
Danish “Evaluation Guidelines,” dated 1999 and published in 2001, do not use the terms
KPI, PI or PM (Ministry of Foreign Affairs, 2001). Rather, these Guidelines refer only to “indicators”,
which are described as proxies for the “intellectual construct or the things we measure empirically”.
Indicator are described as:
Quantitative or qualitative statements which can be used to describe situations that exist and to
measure changes or trends over a period of time…[T]hey are used to measure the degree of
fulfillment of stated objectives, outputs, activities and inputs.
Sweden
In 2004, the Swedish International Development Cooperation Agency published a discussion
paper summarizing theory behind the “Logical Framework Approach” (Örtengren, 2004) which does not
mention the terms KPI, PI or PM. There are some references to indicators and measurement, but only one to
data. The term analysis appears often, but the links between measurement, information, and data are vague.
Quotes:
Step 7 Indicators – measurements of results
Is the project achieving its goals? To answer this question, the project group needs to identify indicators,
which make it possible to measure the progress of the project at different levels. Establishing a suitable
indicator for an objective is a way of ensuring that an objective becomes specific, realistic and tangible.
An indicator shall be objectively verifiable. In other words, anybody shall be able to measure the results. It
shall be clear where data for measurement purposes can be found.
Objectively measurable and verifiable indicators [column heading in a LFA matrix]
Australia
In 2000, AusAID, the Australian Agency for International Development, published a draft
guide to preparing evaluation reports, dated 10-03-00. Although the guide refers frequently to KRAs (key
result areas), data, outcomes and impacts, the guide does not use the terms PI or PM even once. The term
indicator is used three times, but only when referring to an economic or financial indicator. The term KPI
appears once, when reporting the values of economic and financial indicators.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 6 of 14
Exhibit B1 (continued)
USA
The W. K. Kellogg Foundation Evaluation Handbook, dated 1998, does not use the terms
KPI, PI or PM (W.K. Kellogg Foundation, 1998). The Handbook refers frequently to indicators, without
providing a formal definition. The Handbook advises that indicators may be measured and reported either
quantitatively or qualitatively. The first quotation, below, is included, because of its reference to goal
displacement, a serious problem that plagues the use of KPIs. The second quotation is included, because of
the support it provides for “triangulation”, that is, for using substantially different methods of data
collection when reporting on the status of just one indicator.
How will you measure outcomes? In order to determine how effective a program is, you will need to
have some idea of how well outcomes are being achieved. To do this, you will need ways to measure
changes the program is supposed to effect. This is another place where program staff and stakeholders
can lose sight of individual participant outcomes and begin to focus exclusively on the criteria or
indicators for measuring these outcomes.
This program might also have constructed teacher assessments of a child’s self-esteem to be administered
quarterly. Here the indicator has changed from a standardized, norm-referenced test to a more openended, qualitative assessment of self-esteem; however, the outcome remains the same – increased selfesteem.
Outcomes and indicators are often confused as one and the same, when they are actually
distinct concepts. Indicators are measurable approximations of the outcomes you are
attempting to achieve. The indicators you select for each outcome will depend on your
evaluation team’s philosophical perspective about what is the most accurate measure of your
stated outcomes…
To date, little work has been completed to establish clear, agreed-upon measures for the less
concrete outcomes attempted by comprehensive, community-based initiatives
…the field of measurement of human functioning will never be able to provide an accurate
and reliable measure for every outcome…
Physiological health status measures can be used to reveal priority health needs, or indicate
the extent of particular health problems in a target population or community.
If [observations] are to be useful as project outcome measures, these observations should be
documented
USA
The Logic Model Development Guide” (W.K. Kellogg Foundation, 1998) does not use the terms
KPI, PI or PM. Indicator is used often. Indicators are described variously as types and sources of data,
measures and information.
Indicators are the measures you select as markers of your success. Indicators…(types of data) against
which you will measure the success/progress of your program….[R]ecord the sources of data you plan
to use as indicators (where you are likely to find or get access to these data).
INDICATORS: Describe what information could be collected...
For each of the specific activities you have planned to do, what short-term and then long-term
outcomes do you expect to achieve as indicators of the progress made by your program toward its
desired results?
…outcome measures enhance program success by assessing your progress from the beginning and all along
the way.
Exhibit B1 Examples of how some international donor agencies have described indicators and related concepts such as
performance, measure/ment, data and information. Most of the publications cited did not use the terms performance
indicator or to KPIs.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 7 of 14
Attachment C.
Why is there Confusion about the Meaning of the Terms Performance,
Indicator and Measurement?
International authorities on evaluation and RBM provide conflicting advice about the meaning of the terms
performance, indicator and measurement. In this attachment we discuss some of the consequences of this
lack of consensus. In particular, we discuss two conflicting interpretations of the term performance, two
conflicting interpretations of the term Indicator, and two conflicting opinions about the appropriateness of using
non-numerical (i.e., qualitative) data to record or describe a measure/ment or indicator.
C1.
What is True Performance?
From an international perspective, there are at least two different interpretations of the term performance. See
Exhibit C1.

Global interpretation: according to some authorities, the term performance may refer to almost any aspect
of a program – including the program’s inputs, operations and strategic outcomes and impacts. We call
this the global interpretation or focus of the term performance. (Wholey, Joseph S., 1979).

Strategic interpretation: according to other authorities, the term performance has a much narrower
meaning. According to these authorities, the term performance may only be used to describe or predict a
program’s strategic results, for example, a program’s strategic outcomes and impacts (NSW, 1992;
Comptroller General, 1991; DoF 1989, Poister, 1983). We call this the strategic interpretation of the term
performance.
An important thing to remember about performance indicators is that they must measure performance not
“busyness”. Always check carefully any ‘performance indicator’ that reads ‘the number of...’ It is very likely to be
only a measure of how busy people have been. (NSW, 1992:46 & 48)” [Bold emphasis in the original.]
Accepting one or the other interpretation of performance will change the way that program operations are
described and reported.

According to the global interpretation of the term performance, operational measures such as completed
workload and operational efficiency may be reported as performance measures (PMs).

However, according to the strategic interpretation of the term performance, operational measures should
not be assumed, without evidence, to measures a program’s true performance.
There is one situation when the strategic interpretation of performance will allow operational results to be
reported as not only measures of program operations, but also evidence of program (strategic) performance.
This special situation occurs when an operational measure provides not only an accurate description of
program operations, but also provides an accurate prediction of the program’s strategic outcomes or impacts.
In this special situation, operational measures, such as completed workload and labour efficiency may be
reported as measures of (strategic) performance. However, this use of operational statistics must be justified.
In the absence of evidence that the operational measures do, indeed, predict a strategic outcome or impact,
then – according to the strategic interpretation of performance – such measures must not be reported as
evidence of true performance.
Exhibit C1: Alternative Meanings Associated with the Terms Performance and Indicator
Meaning 1
Performance
Indicator
Meaning 2
Global
The term performance may refer to
any aspect of a program.
Strategic
The term performance may only refer to a
program’s strategic outcomes and/or impacts.
Direct
The term indicator refers only to a
direct measure or description of a
concept that is associated with a
program or activity.
Proxy / Indirect
The term indicator refers only to an indirect or
proxy measure or description of some aspect of a
program. We may refer to such an indirect
measure as a predictor.
Exhibit C1 The terms performance and indicator may vary their meaning, depending upon who is using them.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 8 of 14
C2.
What is the Difference between a Direct Measure and an Indicator?
Not only do authorities differ about the meaning of the term performance, they also differ about the focus of an
indicator.

Proxy/indirect interpretation: according to some authorities, an indicator is a proxy (or indirect) measure or
description of any aspect of a program or activity, i.e., it indicates or points to – but is not a direct measure
or description of this aspect of the program or activity. According to this interpretation, a measure of height
might serve as an indicator of a child’s age, since height points to age, but is not a direct measure of age.
Proxy (or indirect) measures and descriptions are often used when direct measurement or description is
too difficult, expensive, or time consuming, or, perhaps, simply unlikely to yield accurate data.

Direct interpretation: according to other authorities, an indicator is a direct measurement or description of
some aspect of a program or activity. It is called an indicator because it refers to some aspect of a
program or activity. According to this interpretation, years since birth is an indicator of a child’s age,
because it refers to age. Those who adopt the alternative, proxy interpretation of the term indicator would
say that years since birth is a direct measure of the child’s age, not an indicator.
C3.
Do Measurements and Indicators Need to be Numerical?
There is one final debate that affects the use of the terms measure, measurement, performance indicator, and
performance measure/ment. This debate concerns what methodology is used to collect and analyze
performance data.

According to the quantitative interpretation of the terms measure, indicator and/or measure/ment, data
collection, data analysis, and performance reporting must be carried out using numerical methods. For
example, according to this interpretation, a photograph is not a valid measure or indicator unless the
contents of the photograph are described numerically. Similarly, according to this interpretation, a written,
professional opinion does not qualify as a valid measurement until the opinion has been coded as a
number or a series of numbers.

According to the mixed-method interpretation of the terms measure, indicator and measurement, any
methodology may be used for data collection, data analysis, and performance reporting. From the mixmethod interpretation, both qualitative and quantitative (i.e., numerical) methods – singly or in combination
– may be used to obtain a performance measure or indicator. For example, according to this interpretation,
a professional opinion, a photograph and an observer’s verbal description of a teacher’s tone of voice are
all valid measurements – even though they are not coded as numbers.
Attachment D.
Why is there so Much Disagreement about the Attributes of a True KPI, PI or PM?
Attachment D draws upon the alternative interpretations of performance, indicator and measurement that are
found in Attachment C. We use these alternative interpretations to identify eight different standards that could
be used to identify a valid or true KPI or PI. A similar analysis could be used to identify alternative standards
for a true performance measure (PM).
D1. What are some minimum standards for a true PI or KPI?
When alternative interpretations of the term performance are combined with alternative interpretations of the
term indicator, the stage is set for multiple and conflicting descriptions of the attributes a true PIs or KPI. In
Attachment D, we develop minimum standards for a true PI or KPI in stages. Initially, we use alternative
interpretations of the terms performance and indicator to obtain four different, incomplete, minimum standards
for identifying a true PI or KPI. See Exhibit D1.

When the term performance is assumed to refer to any aspect of a program (the global interpretation) and
the term indicator is assumed to refer only to a direct measurement or description, then a true KPI or PI is
described by minimum, incomplete standard 1 (MIS1).
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 9 of 14

When the term performance is again assumed to refer to any aspect of a program (the global
interpretation), while the term indicator is assumed to refer only to indirect or proxy measurement or
description, then a true KPI or PI is described by minimum, incomplete standard 3 (MIS3).

When the term performance is assumed to refer only to strategic outcomes and impacts, while the term
indicator is assumed to refer only to direct measurement and description, then a true KPI or PI is described
by minimum, incomplete standard 2 (MIS2).

Finally, when the term performance is again assumed to refer only to strategic outcomes and impacts,
while the term indicator is assumed to refer only to a indirect or proxy measurement or description, then a
true KPI or PI is described by minimum, incomplete standard 4 (MIS4).
Exhibit D1: Minimum, Incomplete Standards for Identifying a True PI or KPI
Interpretation of the
Terms Performance and
Indicator
Direct Indicator
Direct measure or
description
Proxy Indicator
Must be a proxy or
indirect measure or
description
Performance: Global Focus
Any relevant aspect of a program
Performance: Strategic Focus
Must focus attention on a program
outcome or impact
Minimum, Incomplete Standard 1
Minimum, Incomplete Standard 2
The PI or KPI provides a direct measure The PI or KPI must only provide a
or description of any aspect of a
direct measure or description of a
program
strategic outcome or impact
Minimum, Incomplete Standard 3
Minimum, Incomplete Standard 4
The PI or KPI must only provide a proxy The PI or KPI must only provide a
or indirect measure or description of
proxy or indirect measure or description
some aspect of a program
of a strategic outcome or impact.
Exhibit D1 These minimum standards are incomplete as they do not take account of how the data would be collected,
analyzed and reported.
D4.
What is a True (K)PI?
The minimum standards in Exhibit D1 are incomplete, because they fail to take account of the way that data
would be measured, analysed and reported. As we saw in Attachment 3, Section C3, the term measure/ment
may be interpreted in two alternative ways:

measure/ment only refers to the use of quantitative methods,

measure/ment may involve the collection and analysis of qualitative data, quantitative data, or both.
Exhibit D2: Minimum Standards for Identifying a Valid KPI or PI
Minimum, Incomplete Standards (MISs) for Identifying a Valid (K)PI from Exhibit D1
Data
Specification
Quantitative
Data
Specification
Mixed-Method
MIS 1.
The PI or KPI provides a
direct measure or
description of any aspect of
a program
MIS 2
The PI or KPI must provide
only a direct measure or
description of a strategic
outcome or impact
MIS 3
The PI or KPI must
only provide a proxy
or indirect measure or
description of some
aspect of a program
MIS 4
The PI or KPI must only
provide a proxy or
indirect measures or
descriptions of a
strategic outcome or
impact.
Minimum Standard 1
A numerical, direct
description or measures of
any aspect of a program may
be called a (K)PI
Minimum Standard 2
A numerical, direct
measure or description of a
strategic outcome or impact
may be called a (K)PI
Minimum Standard 5
A mixed-method description
or measure of any relevant
aspect of a program may be
called a (K)PI
Minimum Standard 6
A mixed-method measure
or description of a strategic
outcome or impact may be
called a (K)PI
Minimum Standard 3
A numerical, proxy or
indirect measure or
description of a any
aspect of a program
may be called a (K)PI
Minimum Standard 7
A mixed-method,
proxy or indirect
measure or description
of a any aspect of a
program may be
called a (K)PI
Minimum Standard 4
A numerical, proxy or
indirect measure or
description of a any
aspect of a program may
be called a (K)PI
Minimum Standard 8
A mixed-method, proxy
or indirect measure or
description of a any
aspect of a program may
be called a (K)PI
Exhibit D2 These eight, alternative standards take some of how the data are collected, analyzed and reported.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 10 of 14
Exhibit D2 identifies the eight alternative standards for a valid or true (K)PI. These standards take into account
some aspects of how data were collected, analyzed and reported, as well as the focus of the indicator.
Additional analyses could be carried out to identify a range of alternative standards for what qualifies as a valid
performance measure/ment (PM). Such analyses go beyond the scope of this paper. Even the analyses that
are summarized in Exhibits D1 and D2 may be too detailed for discussion at a conference.
Attachment E Do Governments and Agencies Have Problems when Attempting to Use KPIs as
Strategic Tools?
The MAMPU Circular (MAMPU, 2005) demonstrates that although posing a number of challenges, there are
conceptually few barriers to using KPIs to measure a wide range of attributes associated with program or
project operations. As shown in Exhibit E2, authorities suggest that problems emerge when operational
measures are reported and used as a basis for making financial and other decisions. Authorities also pose
serious questions about how well governments and agencies have been able to use KPIs to identify and report
on the achievement of strategic outcomes and impacts. The evidence provided in Exhibit E2 suggests that
more research and development is required if KPIs are going to be used effectively – across a wide range of
programs and projects – as “strategic tools”.
Exhibit E1: Problems when Using (K)PIs and Similar Management Tools
Goal Displacement and Data Gaming
“Measuring Government Performance: A General Test of Gaming” Abstract: An important lesson
from the economic literature is that explicit incentives may elicit dysfunctional and unintended responses,
also known as gaming responses...this behavior is typically hidden …Much of the recent policy and
public administration literature is concerned with performance measurement, as interest in accountability
and efficiency has waxed in recent years…5 Conclusions An important lesson from the economic
literature is that explicit incentives may elicit dysfunctional and unintended responses, also known as
gaming…One can discover how good a performance measure is only after it has been implemented and
the gaming responses that it generates have been observed. (Courty & Marschke, 2004).
A major problem in assessing bureaucratic performance is the difficulty in judging the final social
outcomes stemming from the work of public agencies. As a result, public agencies are frequently
evaluated based on the outputs they produce. Agency outputs are easier to measure than the actual
contributions agencies make to social outcomes. When agency performance is evaluated in terms of
numerical outputs, bureaucrats have an incentive to maximize outputs, regardless of whether maximizing
outputs is the preferred strategy for achieving desired social outcomes. This incentive to maximize
outputs may lead to organizational cheating, in which public agencies purposely manipulate output levels
to portray their work in the best light possible. This study examines the problems of goal displacement
and organizational cheating in…public schools. (Bohte & Meier, 2000)
Many times we have...been left with performance indicators, functioning as an end in themselves –
sometimes to the detriment of the organisation and the community. (Botsman, 1993:31)
…lead to goal displacement, as managers and staff work to make indicators look good, regardless of the
consequences. (Office of the Comptroller General, 1991).
Over-Reliance on Measures of Operational Output
“An important thing to remember about performance indicators is that they must measure
performance not ‘busyness’...Always check carefully any ‘performance indicator’ that reads ‘the
number of...’ It is very likely to be only a measure of how busy people have been. (NSW, 1992:46 & 48)”
[Bold emphasis is in the original.]
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 11 of 14
Exhibit E1 (continued)
It should be noted that many published indicators are of workload. Workload indicators are about levels
of inputs or activities, not performance. (Department of Finance, 1989: 29)
Unfortunately, because the criterion problem is so difficult, evaluation projects tend to measure workload
rather than true performance. (Quade, 1989:278)
Performance indicators are indicators of the extent to which programs are achieving the desired results...It
should be noted that many published indicators are of workload. Unfortunately, because the criterion
problem is so difficult, evaluation projects tend to measure workload rather than true performance.
(Quade, 1989:278)
Workload indicators are about levels of inputs or activities, not performance. (Department of Finance,
1989: 29)
Another common shortcoming is to report levels of work activities as indicators of performance.
Workload measures may indicate how hard people are working but not whether they are accomplishing
anything.(Comptroller General, 1991:18)
It is important to distinguish between program performance – the reason resources are provided – and
what we shall call administrative performance, matters more related to the internal health and continued
viability of an organisation...Good administrative performance is essential to, but does not assure, good
program performance. (Comptroller General, 1991:1)
Need for Mentors, Training and Skills
Obtaining an experienced professional who can direct the technical aspects and provide training to others
involved in the evaluation is a most important priority. government salary structures are a major
constraint here....(Hatry et al., 1973: 12)
Few [analysts] do the one thing they are best equipped for -- examine critically the assumptions of the
myths to determine whether the policies promoted in their name really accomplish what the myth implies.
(Innes: 1990:30)
Impact of Program Budgeting Reforms that Involved the Use of KPIs
It is highly doubtful whether "the Government itself, Parliament, party machines, unions, employers,
specific interest lobby groups, Government employees [or] anyone else interested in what the priorities
and activities of Government are" is better informed now, after five years of program budgeting, than at
the outset ... (Corbett, 1989:17)
[a] problem that besets ... any system attempting to install "rational" policy analysis in government is that
operational indicators of the attainment of objectives must be developed in order for the system of
performance evaluation to be effective.
Unfortunately, the search for such indicators rivals the search for the philosopher's stone in its apparent
futility. (Peters, 1989:229)
Over-Reliance on Numbers
Rarely will numbers alone tell the story. Qualitative information on performance is almost always
essential ... A common error made in developing performance indicators is to concentrate solely on easily
measured numerical indicators, typically the ones that just happen to be around. (Office of the
Comptroller General, 1991:18)
A common error made in developing performance indicators is to concentrate solely on easily measured
numerical indicators, typically the ones that just happen to be around. (Comptroller General, 1991:18)
Limited Understanding of How to Report on “Efficiency”
[One] way of describing program performance is to use the three "Es" of economy, efficiency and
effectiveness. While this approach has a [long] history, experience has shown that it is difficult to get
agreement on exactly which aspects of performance these terms cover. (Office of the Comptroller
General, 1991:4)
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 12 of 14
Exhibit E1 (continued)
Leathers (1979) examined what he calls the "language barriers" that surround the use of the terms
effectiveness and efficiency. One barrier is the failure to recognize that efficiency is an umbrella concept
that refers to different types of efficiency, e.g., allocative, technical, operational and non-allocative
efficiencies. When efficiency is used as if it were only referring to operational or technical efficiency, then
performance measurement systems may provide extensive information about the efficiency of program
operations, while giving little or no information about other aspects of efficiency.
Leathers also points out that the concept of cost-effectiveness was (and may still be) poorly understood.
Some managers and consultants appear to assume that cost refers only to dollar outlays, which ignores
non-financial opportunity costs and allocative efficiency. Others managers and consultants appear to
assume that cost-effectiveness is a measure of effectiveness, because of the presence of the term
effectiveness. Of course, cost-effectiveness and allocative efficiency can be focused on strategic aspects of
efficiency.
A typical example of “efficiency” is “(the average) cost of serving a client” or “(the average) time
required to complete a procedure”. Operational measures such as average cost per service and average
time per procedure are example of what economists refer to as operational or technical efficiency
(Leathers, 1979).
Limitations of Program Logic Models and Theories
{F}ormal theory and existing information seldom provide certainty about the results of future action. A
myth provides a source of faith that a risk is worth taking. (Innes, 1990:25)
The logic of myths, moreover, is magical. They offer analogies which persuade simply by demonstrating
in a vivid, emotionally charged way what the work is like. Cause and effect in a scientific sense are not at
issue. Because the myth does not spell out cause and effect, the listener can neither evaluate the
reasoning nor consider critically its application to the case in point... (Innes, 1990:29)
Appendix F.
Bibliography/References
Government and NGO Reports and Handbooks
ANAO (1998). “Better Practice Principles for Performance Information,” Australian National Audit Office, Canberra
(electronic copy, creation date July 1998).
AusAID (2000). “AusGUIDElines 14 – Preparing project ex-Post evaluation reports”, Australian Agency for
International Development, Commonwealth of Australia, Canberra.
Comptroller General (1991). "Measuring and Monitoring Program Performance: Developing Useful Performance
Indicators - Draft," Comptroller General, Canada.
Corbett, David (1989). "Discussion Paper on Program Budgeting". Economic and Budget Review Committee,
Parliament of Victoria, Melbourne.
Courty & Marschke (2004). Courty, Pascal and Gerald Marschke, “A General Test of Gaming,” Discussion Paper No.
4514 August 2004, Centre for Economic Policy Research, London; www.cepr.org/pubs/dps/DP4514.asp.
DoF (1989). "Program Evaluation: A Guide for Program Managers, March 1989." Department of Finance, Canberra,
especially the Glossary.
DoFA (2003). “Glossary of Terms and Acronyms”, “Annual Report, Department of Finance and Administration 20022003,” Canberra.
Economic and Budget Review Committee (1990). “Twenty- fifth Report to the Parliament: Program Budgeting, April
1990, “Parliament of Victoria, Melbourne.
MAMPU (2005). “Garis Panduan Bagi Mewujudkan Petunjuk-Petunjuk Prestasi Utama Atau Key Performance
Indicators (KPI) dan Melaksanakan Pengukuran Prestasi di Agensi Kerajaan”, Lampiran kedadea Pekeliling Kemajuan
Pentadbiran Awam Bil. 2 Tahun 2005, Unit Pemodenan Tadbiran dan Perancangan Pengurusan Malaysia (MAMPU),
Jabatan Perdana Menteri, Kerajaan Malaysia [“Guidelines for Establishing Key Performance Indicators (KPIs) and
Executing Performance Measurement in Government Agencies”, Appendix to the Public Administration Progress Circular
No. 2, 2005, Malaysian Administrative Modernization and Management Planning Unit (MAMPU), Prime Minister’s
Department, Government of Malaysia.]
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 13 of 14
Ministry of Foreign Affairs (2001). “Evaluation Guidelines, February 1999”, Ministry of Foreign Affairs, Denmark.
NSW (1992). "Planning and Monitoring Your Program: First steps in program evaluation," Office of Public Management,
New South Wales Premier's Department, Sydney, especially discussion of “busyness” indicators, pp. 46 & 48.
Office of the Comptroller General (1991). "Measuring and Monitoring Program Performance: Developing Useful
Performance Indicators," Draft: March 1991, Program Evaluation Branch, Canada.
Örtengren, Kari (2004). “A summary of the theory behind the LFA method: The Logical Framework Approach”. Sida,
Swedish International Development Cooperation Agency, Stockholm.
W.K. Kellogg Foundation (1998) “W.K. Kellogg Foundation Evaluation Handbook”, W.K. Kellogg Foundation, Battle
Creek.
W.K. Kellogg Foundation (December 2001) “Using Logic Models to Bring Together Planning, Evaluation, & Action:
Logic Model Development Guide” Item #1209, W.K. Kellogg Foundation, Battle Creek, Michigan.
Journal Articles
Bohte & Meier (2000). Public Administration Review 60(20:173-182), “Goal Displacement: Assessing the Motivation for
Organizational Cheating,” Mar/Apr 2000,
Bernstein, D. (1999). “Comments on Perrin’s Effective Use and Misuse of Performance Measurement,” American
Journal of Evaluation, 20, 85–93.
Feller, Irwin (2002). “Performance Measurement Redux,” American Journal of Evaluation 23(4): 435–452.
Leathers, C.G. (1979). "Language Barriers" in Public Productivity Analysis: The Case of Efficiency and Effectiveness."
Public Productivity Review: 3(4): 63-68.
Perrin, B. (1998). “Effective Use and Misuse of Performance Measurement,” American Journal of Evaluation 19 (367–
379).
Radin, Beryl A. (2000) “The government performance and results act and the tradition of federal management reform:
Square pegs in round holes?” Journal of Public Administration Research and Theory 10(1): 111-135.
Rasappan, Arunaselam (1994), “How to measure success”, Khidmat: 27-29, October 1994, Kuala Lumpur.
Sallack, David and Allen, David N., 1987. "From Impact to Output: Pennsylvania's Planning-Programming Budgeting
System in Transition," Public Budgeting and Finance (Spring 1987): 38-50.
Winston, J. (1993). “Performance Indicators: Do They Perform?” Evaluation News and Comments 2(3), 22-39.
Winston, J. (1999). “Performance indicators – Promises Unmet: A response to Perrin.” American Journal of Evaluation 20
(95–99).
Books
Botsman, Peter (1993). "Performance Indicators and Public Sector Effectiveness", pp. 31-35 in Guthrie (1993).
Guthrie, James (1993). The Australian Public Sector: Pathways to Change in the 1990s, IIR Conferences, North Sydney.
Hatry, Harry P., Richard E. Winnie, and Donald M. Fisk (1973). Practical Program Evaluation for State and Local
Government Officials. The Urban Institute. Washington, D.C., especially Hatry’s discussion of the need for a
supplementary objective for all programs that expresses the aim to avoid unplanned results that do more harm than good.
Innes, Judith Eleanor 1990. Knowledge and Public Policy: The Search for Meaningful Indicators (Second Expanded
Edition), Transaction Publishers, New Brunswick, especially the introductory discussion of the assumptions that often
underpin the recommendation to use one or another set of indicators.
Poister, T. H. (1983). Performance Monitoring, Lexington Books, Lexington, Mass., especially Poister’s discussion about
the need to distinguish between administrative and true performance measurement and reporting.
Quade, E.S. revised by Carter, Grace M., 1989. Analysis for Public Decisions (3rd ed.), North-Holland, New York.
Wholey, Joseph S. (1979). Evaluation: Promise and Performance, The Urban Institute, Washington, D.C.
Models & Frameworks
Rasappan, Arunaselam (2003). ProLL Results Ladder. Center for Development and Research in Evaluation (CeDRE),
Kuala Lumpur, Malaysia.
Rasappan, Arunaselam (1999). Program Logic & Linkages Model (ProLL Model). “Results Oriented Budgeting
Reforms: Malaysia & Singapore”, World Bank, Washington, DC, July 15, 1999.
© 2006 Jerome Winston & Ranjit Bhagwandas Do KPIs Really Work as Strategic Tools? (Revised)
Page 14 of 14