Evaluation capacity
building:
the international experience
Scott Bayley,
AusAID
Overview
1. What is evaluation capacity
building (ECB)?
2. Why do it?
3. Strategies for ECB
4. International examples
5. What works? – guiding principles
What is ECB?
• The UNDP (2002) defines capacity as
the ability of people, institutions and
societies to perform functions, solve
problems, and achieve their goals.
• ECB refers to activities and initiatives
taken to mainstream a regime that
supports the ongoing production and
use of evaluations.
Why do it?
Governments and organizations build
evaluation systems because they
believe such systems will help them
to improve their means and methods
of governance.
In particular …
Why do it? ...
… ECB can be used to support:
1. Planning & policy making
2. Program management
3. Resource allocations, budgeting
4. Government control, coordination
5. External reporting & accountability
6. Participation by civil society.
What is successful ECB?
• The production of appropriate quality
evaluations
• A high level of utilization of evaluation
findings
• Country/agency ownership
• Sustainability over time as
governments & officials change.
Strategies for ECB
1. Whose capacity and for what purpose?
•
Legislatures
•
Federal/state governments
•
Sectors, eg. health, education, water,
agriculture, justice
•
Agencies that commission and fund
evaluations
strategies for ECB …
•
Individual evaluation practitioners
and networks
•
Those who use the results of
evaluations to help guide their
policy making and management
activities, govt agencies, donors,
NGOs
strategies for ECB …
•
Groups affected by the programs
being evaluated
•
General public, the academic
community, civil society.
Whose capacity and for what purpose?
strategies for ECB …
2. Diagnostics
• current practices
• politics of reform
• desired functioning
strategies for ECB …
Current practices
-current policies and systems, existing
M&E activities, institutional capacities,
linkages to planning and budgeting
strategies for ECB …
Politics of reform
-drivers for change, supporters and
opponents of reform, resources,
incentives
strategies for ECB …
Desired functioning
-intended users and uses, supply and
demand factors, supporting institutional
infrastructure
2. Diagnostics
X
strategies for ECB …
3. Tactics
•
Overseas fact finding tours
•
Training workshops for
practitioners
•
Production of policy and
operational manuals, evaluation
plans
•
Legislation/regulation
sequencing ECB …
1.  Demand for and ability to use
evaluative feedback, then
2. Supporting institutional
infrastructure &
3. Supply side considerations
- Guiding principles (ECDG group):
ownership; relevance; integration; usefulness.
strategies for ECB …
•
Identify and work with powerful
champions of reform
•
Link ECB to other significant reforms
•
Actively manage the change process
- ECB develops over time and in stages
strategies for ECB …
•
Incentives:

the example set by leaders, rewards,
education and information, practical
support
 Incentives to increase demand and
use
 Incentives to reduce perceived costs
and increase benefits within agencies
(eg. guarantee of no job losses or
budget cuts)
strategies for ECB …
•
Training to raise awareness of those
who might demand evaluations to
enhance service quality i.e. internal
senior managers; external bodies
such as NGOs, consumer advocates,
media
•
L/T partnerships and joint
evaluations for skill transfer/learning
strategies for ECB …
•
Train the trainer (to provide on the
job training to practitioners)
•
Hold managers accountable for
learning/making improvements
•
Capitalize on existing demand and
windows of opportunity
•
Link evaluation into systems for
policy making, planning, budgeting,
reporting, etc.
International examples
First wave, 60s & 70s
Second wave, 80s & 90s
Third wave 90s+
What works? – guiding principles
1. ECB is a political activity with
technical implications.
• and not vice versa!
• ECB is about organizational change,
link it to other significant reforms
• ECB creates winners and losers
- which means supporters and opponents.
principles…
2. Start with good diagnostics.
• local context and history matter!
• aim to build upon local strengths and
interests; target functional needs
rather than ‘correcting’ deficiencies.
principles…
3. Building and capitalizing on
demand is the key.
• seek to match the supply of evaluations
to current demand, build demand (and
capacity to use eval info) over time
• attempting to force the creation of an
evaluation culture through legislation or
a supply side push simply doesn’t work
• a limited availability of internal
evaluation skills is not a fundamental
constraint, use contractors if need be.
principles…
4. Institutionalize evaluation in a
way that aligns supply to demand.
• the evaluation function needs to be
located (or anchored) where the
demand and users are
• we have a matrix arrangement to
support accountability in government.
Evaluation would benefit from a similar
approach.
principles…
5. Learn from others but avoid best
practice models.
• different countries/agencies have
different starting points, aspirations,
constraints & opportunities
• search for adaptable ideas, not
blueprints for reform
• Cautious evolving experimentation is the
way to go (along with M&E of ECB activities)
- ECB is not a linear process.
principles…
6. ECB is a long term process
• 5 to 10 years for agencies in more
developed countries
• 10 to 20 years in developing countries
• political and organizational interest in
evaluation is cyclical
- ECB is ongoing, not a “one time” event.
Summary
1. What is evaluation capacity
building (ECB)?
2. Why do it?
3. Strategies for ECB
4. International examples
5. What works? – guiding principles
Questions & Discussion
For further information:
[email protected]
* Abonyi, George. 2002, Toward a Political Economy
Approach to Policy-based Lending. ADB (on the web).
* Boyle, Richard, and Donald Lemaire, eds. 1999,
Building Evaluation Capacity. New Jersey: Transaction
Publishers.
* ECDG, 2011, International workshop agreement on
evaluation capacity development (on the web).
* Lahey, Robert. A Framework for Developing an
Effective Monitoring and Evaluation System in the
Public Sector (on the web).
* Mackay, Keith. 2007. How to Build M&E Systems to
Support Better Government. World Bank (on the web).