Performance Management Systems
and Evaluation:
Towards a Mutually Reinforcing Relationship
Jacob Alex Klerman (Abt Associates)
APPAM/HSE Conference
“Improving the Quality of Public Services”
Moscow, June 2011
Abt Associates | pg ‹#›
Performance Management Systems
and Evaluation
Performance Management
Evaluation
Abt Associates | pg ‹#›
Performance Management Systems
and Evaluation
Performance Management
 The need is clear
Evaluation
– “What gets measured gets
done”
– If you know what you want
done; you need to manage
against it
– To manage against it, you
need to measure it
Abt Associates | pg ‹#›
Performance Management Systems
and Evaluation
Performance Management
 The need is clear
– “What gets measured gets
done”
– If you know what you want
done; you need to manage
against it
– To manage against it, you
need to measure it
Evaluation
 But what do you want done?
– Are you sure?
 That’s the role of rigorous
impact evaluation
– Dirty little secret: much of
what we do—much of what
seems “plausible”—has
minimal impact (or even
hurts)
Abt Associates | pg ‹#›
Outline
 Current Practice
 A Better Way
 Closing Thoughts
Abt Associates | pg ‹#›
Rigorous Impact Evaluation Is Crucial
 Everyone wants better program outcomes
– We might even be willing to spend more if we could prove
better outcomes
 Proving “better outcomes” requires rigorous impact
evaluation
– Many apparently plausible programs (and program
innovations) don’t work
– Naive evaluation methods give the wrong answer
 Rigorous impact evaluation is challenging
– Requiring large samples
– And, the smaller the projected incremental impact,
the larger the required samples
Abt Associates | pg ‹#›
Current Evaluation Practice Isn’t Very Useful
 Asks the wrong
questions:
Does the program
“work”?
– i.e., Should we shut the
program down?
– Big programs address
major social problems
– The programs aren’t
going away
Abt Associates | pg ‹#›
Current Evaluation Practice Isn’t Very Useful
 Asks the wrong
questions:
Does the program
“work”?
– i.e., Should we shut the
program down?
– Big programs address
major social problems
– The programs aren’t
going away
 The right question is
often:
How can we make the
program better?
– Which program model
works better?
– Would some minor—and
affordable—change in
program design help?
– For which subgroups
does our program work?
Target the program at
them
Abt Associates | pg ‹#›
The Realities of Sample Size and Cost
 Answering up/down evaluation question requires
(relatively) small samples
– For a training program, perhaps 500-2,000 case
 Answering practitioners’ questions requires much large
samples
– For a training program, perhaps 10,000+ cases
 At current evaluation cost—$1,000+ per case—we can’t
afford to answer practitioner’s questions
– Especially if the change in outcomes will be at best small
– And that’s a big problem because CQI/kaizen suggests that major
improvement often comes from lots of small improvements
To answer practitioner’s questions,
we’re going to need to get the cost way down
Abt Associates | pg ‹#›
Steps in a Current Evaluation
 Negotiate access to sites, including convincing them
to deny service to some applicants
– <time consuming and expensive>
 Customize randomization for each site
– <time consuming and expensive>
 Detailed process analysis at each site
– <expensive>
 Detailed survey follow-up
– <very expensive>
Is there another way? Sometimes, yes …
Abt Associates | pg ‹#›
Outline
 Current Practice
 A Better Way
 Closing Thoughts
Abt Associates | pg ‹#›
At the Back End—Leverage Ongoing
Performance Management Systems
 We have just argued that collecting information on
outcomes drives costs
 Performance measurement systems already collect
information on outcomes
– Presumably on the key outcomes
 So, when we can measure outcomes through the
performance measurement system
– Costs will be much, much lower
– Allowing large samples
– A key requirement for evaluating incremental changes
 Will only work when both treatment and control are
“in the system” (e.g., incremental changes)
Abt Associates | pg ‹#›
At the Front End—A Learning
Organization
 Currently research is “top down”
– Someone outside the system decides to evaluate X
– Then, evaluator tries to convince sites to adopt X;
and to deny all services to a control group
Abt Associates | pg ‹#›
At the Front End—A Learning
Organization
 Currently research is “top down”
– Someone outside the system decides to evaluate X
– Then, evaluator tries to convince sites to adopt X;
and to deny all services to a control group
 Alternative is “bottom up”
– Ask sites to suggest what to evaluate
– Form a committee—site representatives, central program
staff, substance experts, evaluation experts
– Ask them to select from among the suggestions
– Ask sites to volunteer to implement the selected
suggestions
– Randomize at the site level;
control condition is “current practice”, not “no service”
Cutting time and costs
Abt Associates | pg ‹#›
In Summary
Now
Better
Negotiate access to sites (expensive They volunteer
and time consuming)
Customize randomization for each
site (expensive and time consuming
Site level randomization
Detailed process analysis
(expensive)
Skip this
Collect detailed survey outcome
data (very, very expensive)
Use Performance
Management System data
And when your costs drop sharply,
CQI if feasible; i.e., you can test little changes
Abt Associates | pg ‹#›
Outline
 Current Practice
 A Better Way
 Closing Thoughts
Abt Associates | pg ‹#›
A True Learning Organization
 Performance measurement is an ongoing task
 CQI/Continuous Quality Improvement; i.e.,
– Proposing small changes to SOP/Standard Operating Procedures
– Rigorously evaluating those small changes
– Adopting those that can be shown to “help”
 … Should also be an ongoing task
 The key insight of “kaizen” is that improved outcomes arise
from the accumulation of lots of such small changes
Data collected as part of Performance Management
Systems makes such CQI feasible
Abt Associates | pg ‹#›
When Will this Work?
 Site level randomization needs lots (50-200) of,
relatively similar, sites
 Central organization controls resources
– Much easier to get volunteers, when volunteering is the
only way to get more resources
We’re looking for test cases. Any volunteers?
Abt Associates | pg ‹#›
Performance Management Systems
and Evaluation:
Towards a Mutually Reinforcing Relationship
Jacob Alex Klerman
APPAM/HSE Conference
“Improving the Quality of
Public Services”
Moscow, June 2011