Improving the Evidence Base on Results Based Approaches
A pladoyer for an eye-level exchange with the partner side
For the use for the "Working group on Result Based
Approaches" of the Development Practitioners' Network
2nd Draft, 24.03.2013
By Kerstin Meyer
Commissioned by Deutsche Gesellschaft für
Internationale Zusammenabeit GIZ,
Berlin and Eschborn. March 2013
CONTENTS
Introduction .................................................................................................................. 2
Context
This paper
1. State of the evidence on Results Based Approaches .............................................. 3
1.1 Current conclusions
1.2 Implicit challenges within the current evidence base
a) Disentangle perspectives
b) Need to understand the quality of processes: structured implementation dialogue
c) Evaluate donor inputs in terms of contribution
d) Separate accountabilities
2. Learning from implementation : The governance base of results ....................... 6
a) Policy embeds results
b) And a culture of social responsibility/accountability
c) Is aid complementary to the on-going partner processes?
d) Whose Cost-efficiency? Separating accounts
3. Dialogue based evidence: possible next steps ......................................................... 9
Summary Table: Improving the Evidence on Result Based Approaches ............. 10
Annexe 1: Synthesis of recommendations made by reviewers for improving the
quality of evidence ..................................................................................... 11
Notes
................................................................................................................... 11
Bibliography ................................................................................................................. 13
1
Introduction
Context
To pay tranches for a credit or a financial subvention only on delivery of results, is an
element of New Public Management as it has been introduced in numerous conuntries
and sectors since the 1980s. In international development the approach has become
more frequent since the 2000s in a variety of forms and contexts: Conditional Cash
Transfers, for instance, target demand side bottlenecks and give incentives directly
from government to citizens, e.g. to parents for schooling their children; for targetting
constraints on the supply side, other varieties of incentive schemes have been tested,
e.g. to improve health service or infrastructure provision, and for which the World
Bank and other donors in 2003 coined the term Output Based Aid; likewise, Global
Funds set up contracts with public institutions worldwide for delivery of e.g.
vaccination services or reduced CO2-emissions, and at the macropolitical level,
donors look back on a decade's experience with budget support's variable tranches tied
to the evolution of particular development or sector indicators.
Results Based Approaches are taken by many, both donors and partner governments,
to be auspicious new tools in international development, not only because they are
expected to give the "right" incentives to the involved systems, but also because they
promise to address key shortcomings of aid with partner governments exerting full
discretion over the means how to achieve the agreed results. As DFID puts it : "Will
Payment by Results be a game changer?"
Indeed, today, donors are decidedly going ahead with Results Based Approaches: The
European Union is further developing its budget support tool towards payments
against achieved MDG results (MDG-Compact), the Worldbank published their new
Payment-for-Results Policy (P4R) and corresponding lending tool; European bilateral
donors have created a Working Group on Result Based Approaches as part of their
Practitioners' Network with the aim of sharing and systematising their diverse
experience with results based projects and tools. The Working Group is also interested
in inviting the partner side to the debate. 1
This paper
Following the recommendations of the meeting of the above mentioned working
group on November 26th in Brussels, GIZ has commissioned this conceptual
proposal. It's objective is to make a headstart on criterias for assessment of Results
Based Approaches.
What is specifically proposed here is to look closer at the change in direction that
Results Based Approaches propose: they are not taken to be projects themselves, but
to be contributions to reform processes on the partner side. Coming from an
organisation that has since long been understanding its work in terms of
accompanying partners' processes, the following arguments are stakeholder/processoriented and presented as systemic perspectives. Their main aim is to encourage the
working group to initiate a dialogue with the partner side on their experience and
assessment of Results Based Approaches.
Kerstin Meyer, March 2013
2
1. State of the evidence on Results Based Approaches
1.1 Current conclusions
To understand donors' current evidence base, it is referred here to five metaevaluations that have been commissioned in the recent past. Because they aggregate
and systematise a large number of individual studies, syntheses and reviews, they
constitute a primary source of information and analysis for the Practitioners' Networks
needs.2
These reviews agree that there seems to be general positive evidence that one achieves
the results one had set out to meet (objectives effectiveness), but that the evaluation
studies do not give an understanding of quality of outcomes beyond the small focus on
targets3. It is actually the quality of the evidence base that is a strong concern for the
reviewers, and which leads to their recommendations to be cautious in assuming
benefits are sustainable4.
Perrin (2013) looks at the quality of the evidence in detail: he documents a 'limited
awareness or attention in these studies to generally accepted evaluation standards and
quality criteria', and in particular 'limited attention to the five DAC evaluation criteria,
except to objectives achievement' 5 . Furthermore, he confirms what has also been
found in the earlier reviews, that current evidence neglects, irresponsibly, to consider
negative effects at all, although they are to be expected in incentivised environment;
there is no documentation on cost effectiveness, no comparison with other potential
approaches to improve performance, no narratives of factors that help or hinder the
achievement of result, or how incentives are supposed to bring about development
outcomes (logic model or theory of change). Moreover, equally signaled by the earlier
reviews, long-terms effects, for instance on equity or in terms of poverty reduction,
are not documented, nor is made known if the improved results or performance
changes were expected to be sustainable.
1.2 Implicit challenges of the current evidence base
a) Disentangle perspectives
The above reported limitations concern an evidence base that is produced by/for
donors and not by/for the partner side6. This puts heavy constraints on perspective. To
give an example: when the partner system, during the implementation of the Results
Based Approach reacts first internally to emerging negative effects (i.e. that services
may become inaccessible to some, create social conflict, debts, corruption, squeeze
out other basic services), this is not something that donors will be necessarily aware
of or that would be reported to them immediatly, which explains some of the above
noted limitations on donors' evidence. There is also a tendency of the evaluation
reports to seemingly represent the whole of the reform while actually focussing on a
financing instrument and related elements. Considering this limitation of the current
documentation, an eye-level exchange with the partner side on their evidence and
experience with Result Based Approaches seems a commendable avenue.
Kerstin Meyer, March 2013
3
b) Need to understand the quality of processes: structured implementation
dialogue
Result Based Approaches strictly tie payments (or not) to achievement (or not) of the
previously targeted indicators as agreed in the terms of the contract, putting emphasis
on the independent verification of actually achieved results which then trigger (or not)
the payments. Yet, does meeting targeted results reflect progress in overall system
development towards desired outcomes? It is not a surprise to anyone that it does not.7
But implementers of course do have knowledge about how systems are moving or
advancing. What is proposed here is that donors and partners
make room for a sustained dialogue on process which gives
an occasion to articulate and map actual and wanted process
outcomes in terms of agreed criteria, long before one has a
full technical view on final outcomes. Made explicit or not,
process mapping is what allows implementers to steer both
with orientation and flexibility. For donors, the process
dialogue is an opportunity to gain and document knowledge
about systemic qualities of the reform and their contribution,
Diagram #1 :
beyond the narrow focus of the incentivised results.
Implementers' process &
dimensions of quality
It is in this vein that one can understand
the recommendation made by several
reviewers (see annexe for a synthesis of
their recommendations) to have Results
Based Approaches include an internal
M&E capability - provided this does not
introduce new controls on the use of
resources. Joint process monitoring and
continuous participation in stakeholder
dialogue, amplified through interfaces
with the other programs that accompany
the reform at other levels or from other
perspectives, will facilitate donors'
#2 : Implementers' process map:
understanding of the dynamics and logic Diagram
Orientation and flexibility
of the involved systems and improve
program or mechanism design and adjustments. Moreover, with the gains in
understanding how the system responds to the incentives and how results are being
achieved, the risk - and the fear - of "gaming" will be reduced and some pressure be
taken off the instances of verification.
c) Evaluate donor inputs in terms of contribution
Perrin confirms that there is a methodologcial challenge when evaluating Results
Based Approaches as isolated elements. He proposes more appropriate methods, i.e.
"realist" and theory-based, which allow an assessment of contribution rather than
linear cause-effect arguments, and proposes lean evaluation formats to be used as part
of internal monitoring. From GIZ experience one can add, as a recommendation for
situations when stakeholders need to estimate systemic long term effects of an
intervention, a practical ex-ante impact assessment methodology elaborated by the
Kerstin Meyer, March 2013
4
OECD/DAC. The tool can be used for desk-study or collective analytic work,
depending on the needs of the decision process, and it is transparent and
communicable with stakeholders and decision makers.8
d) Separate accountabilities:
The key question though remains, which accountability process and whose scrutiny
the evidence is produced for. With regard to the reform, it is evidence, that is
produced from working accountability relationships within the partner system that
will provide information which can be used to "regulate" the system. Donors could
provide room (time, capacity/capability, finance) for improving partner side internal
(or independent) auditing and social accountability. With regard to the donor-partner
process, donors can then reduce their verification to the information from the
structured implementation dialogue for the purpose of giving account to their own
political authority.
Diagram #3: A structured dialogue on
implementation process complements the contractual
relation and strengthens both Donor (1) and Partner
country (2) accountability.
Summary - Proposed steps of action for improving
the quality of evidence on Results Based Approaches:
 Give more priority to partner-produced evidence
 Evaluate donor inputs in terms of contribution rather than linear cause-effect
relations
 Make available to stakeholders appropriate ex-ante impact assessment
methodologies
 Make space for structured implementation dialogue and process mapping
 Provide room (time, capacity/capability, finance) to improve partner side auditing
and social accountability.
 Review evaluation and field reporting policy to gain evidence from:
- Complementarity of aid with respect to partner processes
- Implementation process dialogue
Kerstin Meyer, March 2013
5
2. Learning from implementation: The governance base of approaches
a) Policy embeds results
In the past decade many partner institutions have introduced results based policy
frameworks as a foundation for coherent policy making. And since results orientation
is being promoted on many levels simultaneously, there has been so much evolution
that by now some partner institutions have more experience in results based
management than has a donors' home organisation. Not only are national policies and
their implementation organised by results, they are also more and more politically
integrated laterally, at subnational level in decentralisation policies and at the
supranational level in respective governance frameworks.
In West Africa and the Sahel, for instance, common policies and policy standards 9 are
being planned - by results - and implemented and coordinated across national borders
(with the ECOWAS Commission, UOMEA, CILSS). In practice, as in the case of
ECOWAS "strategy to inculcate the culture of results orientation", long term interinstitutional policy dialogue and increasing
part of their strategy for inculcating
accountability between member states and the As
the culture of RBM and RBM&E the
regional governance body had the effect of raising ECOWAS’ Commission and Court of
synergetic interests of stakeholders in the region, Justice are pursuing inter-institutional
in an on-going process that
and to draw them into the policy sustaining cooperation
will provide the Community with a
processes. Incidentally, for West Africa, the common evaluation and monitoring
stakeholders active on these regional platforms are system. (The GIZ support program has
ECOWAS' introduction of a
the same that plan, coordinate and control policy accompanied
results-based management system since
measures at decentral, sectoral and national levels 2007)
and who are thus in an immediate position to
implement the integration agenda - and, incidentally, they are the same that coordinate
programs with donors.
As partners' policy choices are now being progressively embedded in matrices of
interconnected and integrating strategies, each with their implementation and
financing policies, their logic and timeframe, they become effectively conditionalities
on donors' contributions: But are donors able and prepared to accept these? Donors
could put it as a test to themselves if they will formally articulate their national
programs within the regional policy frameworks. Likewise, even in contexts where
donors have advanced in coordinating their inputs for decentralisation reform, the
same donors may still lag behind in reconciling their own sectoral supports with the
prerogatives of the decentralisation reform and thus inadvertedly hinder with the left
hand what they promote with the right, a situation which lead to evaluators' advising
donors to "treat existing institutions as legitimate entities"10.
b) And a culture of social responsibility/accountability
Improved accountability is the one positive long term systemic outcome for which
donors consistently expect Result Based Approaches to make a positive contribution,
and which evaluations even cite as an objective in itself. 11 Yet, accountability
structure is not a field that is systematically qualified, measured, analysed or given
orientation on in the evaluations. Accountability in a concrete, systemic sense,
namely, if and how the involved stakeholders themselves12 watch over the outcomes
Kerstin Meyer, March 2013
6
and negotiate responses is not, as yet,
Outcome Oriented Measurement Systems for
technically recognized as the setting from Subnational Governance in Practice. .
which payment-by- results schemes are -> A continuous process of adaption: The system
being derived and regulated in the partner to be developd must be dynamic enough to meet its
to create learning
system. How does close knowledge of the political function, i.e.
opportunities and make results connect with policy
system feed back to improve quality of or management options. On the other hand, it must
results? From Rwanda's Health reform be predictable and realistic enough for it not to be
where incentives payments are made to percieved by local government units as an
of control from upper level government,
health workers 13 , stakeholder discussion instrument
which could cause contradictory reactions.
forums oversee the incentive mechanisms ->Stakeholder Implication: Even more important
and work out adjustments. In Senegal, on than pre-testing is the involvement of all
a backdrop of a politically active civil stakeholders to ensure that the instrument being
society and local government empowered developed supports the interests and the common
ideas of local government units and the central
by meaningful elections and with government, and even of international donors in a
government and/or donors providing developing country context.
investment budgets to municipalities on ->Indicators: The performance measurement
by-results-schemes,
stakeholders system integrates policy indicators with indicators
of higher interest for citizens, taxpayers and local
dialogue publicly on policy and the accountability, i.e. on output.
quality of implementation, thus realising ->Capacity Development Performance measuretheir right to transparent and accountable ment systems have an enormous potential for
government; there is also an ongoing enhancing capacity development, not only at local
level.
process to implement a performance
Source: Taraschewski, Wegener (GIZ) "Assessing Public
assessment framework for municipalities, Sector Performance" Jakarta. 2011.
which is accompanied by GDC. Indeed,
GIZ reports that demand for developing outcome-oriented measurement systems for
subnational governance is increasing worldwide14.
c) Is aid complementary to the on-going partner processes?
Diagram #4
In the partner system, interdependent agents and levels manage the responsibility for
outcomes - and for institutional change - in a way that is integrative of multiple
policy mangagement threads (see Diagram #4) and of social, political and
implementational considerations. If a Results Based Approach with donors becomes
Kerstin Meyer, March 2013
7
an option, what matters, from the systemic perspective is less, how many or few
indicators are on the list, but more, if the processes that have been found consensus on
follow the internal logic, are coherent with each other and complete in terms of
timings, stakeholders, results and the internal allocation of resources. Such processes
of combined horizontal and vertical organisation do not happen in empty space, but
are forged under current capacities and constraints, and hence take their time, will be
measured in years.
The reviews agree that "good design" of RBAp seems to be crucial, but is contextand process-dependent 15 . Evidence on the right incentive-verification-disbursement
mechanisms will hence not be gained from systematising across sectors and
continents alone, but potentially from studying the process of, for instance, trials and
adjustments, within/by/for the systems that are managing their internal change by
applying the results based financing. Indeed, when reviewers look with hindsight at
the Results Based Approaches, it is "ownership"16 that is singled out as a critical, if
not the critical factor for Results Based Approaches to prospectively achieve relevant
and sustainable outcomes. Significantly, Gorter et al. (2012) use an information about
process -"Who initiatied the Results Based Approach, donor or government/partner
institutions?"- as a proxy for ownership.
And notwithstanding the magic of the term, today there are practical limits to
ownership. Donors could put the test to themselves, again, if they are able and
prepared, for instance, to co-finance, based on mutual understanding, local or regional
development programs which they have no authorship in and which have advanced
on partner-side mechanisms and resources.
d) Whose Cost-efficiency? Separating accounts.
Just as there is a need to separate accountabilities (p.5) there is a need to separate
accounts when gauging cost efficiencies. Because the criterion is usually looked at
from a donor point of view, it is associated with the increased cost of control and
verification, e.g. external assessments that donors incur when fulfilling requirements
towards their own political authority. From a partner perspective, the accounts look
different. What the partner side invests in terms of capacities in the entire process to
sustainably achieve the outcomes, time and other resources, is not perceived by the
donor. Monitoring and evaluation, or performance measurement, if it is introduced
more widely in the system, is costly, but is a general management decision and not
clearly attributable to payments-by-results tools. But other additional cost items are
attributable, such as the cost of pre-financing, of mitigating negative effects of the
incentives for instance, or the higher - or lower - cost of financial management and
coordination of the Result Based Approach. What will supposedly matter more to the
partner organisation is financial sustainability of the results as they are only
temporarily financed by the Results Based Approach. Hence partners' need to
conceive of changes that are both, internally cost efficient and complementary to
legitimate long-term allocation procedures and priorities.
Perrin (2013) and Pearson (2011), taking a step back, are remindful that tools other
than by-results-payments may be more efficient to increase performance in public
administrations. Yes, but, again, what does it mean to place this advise in the realm of
donors?
Kerstin Meyer, March 2013
8
Summary - Learning from implementation:
Proposed Fields of Study/Evidence
Relation of the RBAp to the Results Based Policy Frameworks and Social
Responsibility/Accountability:
 The processes by which the RBAp and the associated donor programs are mounted
within partners interconnected policy and monitoring frameworks (local, national,
regional, continental).
 The organisation, in time, of (the different) accountabilities within the RBAp, and of
the structured process dialogue on implementation
Complementarity of Results Based Aid (of which the RBAp is a part)
 The organisation of interfaces of Results Based Aid to on-going policy processes
 The process of coordination of technical and financial partners to improve
complementarity
3. Dialogue based evidence - possible next steps for the Working Group
Participating Agencies share & harmonise evaluative work
 Share information on upcoming or accomplished evaluations
 Collaborate on terms of reference, choice of expertise, make evaluative
work available for meta-reviews and discussion with peers
 Commission joint evaluative work
Improve the evidence base
 The Working Group discusses challenges
and proposes steps of action.
Learn from implementation:
 The Working Group discusses fields of
enquiries, agrees criteria and next steps
Meta-Reviews
summarise
single case studies usually for a
(sub)sector, but should also be
thematically
or
regionally
focused, and also gauge the
quality of the evidence. They
could become an important tool
not only for donors but in
particular by/to heads of
states, regional/continental
governance bodies and policy
makers in partner countries.
The Working Group invites a dialogue with the partner side on their
experience and recommendations regarding Results Based Approaches.
Kerstin Meyer, March 2013
9
Summary Table: Improving the Evidence Base on Results Based Approache
Objective of
Evidence
INFORM DESIGN PHASE &
DISBURSEMENT DECISION
INFORM DESIGN PHASE &
IMPLEMENTATION
PROCESS DIALOGUE
DIALOGUE AND
AGGREGATE EVIDENCE
BUILDING
Overarching
Criteria
(Objectives)
Effectiveness
To be agreed with
partner
e.g. (using
OECD/DAC)
Efficiency,
Relevance
Sustainability
Long term impact
Other
To be agreed
within Working
Group & with
partner side
Kerstin Meyer, March 2013
Guiding Question
Have the targeted
Results been met?
Will / Is the system
moving towards
politically/socially
wanted outcomes?
How do Result Based
Approaches perform as
new aid instruments?
Under what
circumstances is a
Results Based Approach
appropriate?
Criteria / Propositions
 Targeted Results as in the contract
 Factors of success and failure
Learning from implementation:
Proposed Fields of Study/Evidence (from p.8) :
The relation of RBAp to the Results Based Policy Frameworks and Social
Responsibility / Accountability:
 The processes by which the RBAp and the associated donor programs are
mounted within partners interconnected policy and monitoring frameworks
(local, national, regional, continental).
 The organisation, in time, of (the different) accountabilities within the RBAp, and
of the structured process dialogue on implementation
Complementarity of Results Based Aid (of which the RBAp is a part)
 The organisation of interfaces of results based aid to on-going policy processes
 The process of coordination of technical and financial partners to improve
complementarity
Improving the quality of the evidence on RBAp
Proposed steps of action (from p.5)
 Give more priority to partner-produced evidence
 Evaluate donor inputs in terms of contribution rather than linear cause-effect
relations
 Make available to stakeholders appropriate ex-ante impact assessment
methodologies
 Make space for structured implementation dialogue and process mapping
 Provide room (time, capacity/capability, finance) to improve partner side
auditing and social accountability.
 Review evaluation and field reporting policy to gain evidence from:
 complementarity of aid with respect to on-going policy processes
 implementation process dialogue
Source of
Information
Partner M&E +
Audit+
Implementation
dialogue +
Reporting from
field staff
Partner M&E+
Implementation
Dialogue +
Reporting from
field staff+
Independent
evaluation
Dialogue with
partner
Joint evaluative
work
10
Annex 1: Improving the quality of evidence on Results-Based Approaches
Operational synthesis of recommendations from five meta-reviews
1

Before implementation
 Potential unintended effects should be anticipated and articulated at the design stage, and
monitored on an ongoing basis. (Perrin)
 Conduct an aid effectiveness impact assessment before implementing a results- based approach.
The assessment should also look at long-term sustainability, apply to experimental initiatives
and be made publicly available. (Pereira et al.)
 Formulate the theory of change (Perrin)

During implementation
 Programmes should include an internal M&E capability
 Potential unintended effects should be monitored on an ongoing basis
 Explore and describe the process by which PBR initiatives are implemented in practice.
Methods can range from routine M&E to more comprehensive ad hoc evaluation studies (Perrin)
 We need to closely monitor impact on equity (Pearson)
 It needs to be possible to monitor and verify the results against which payments might be made
and these results need to be closely linked to the overall outcomes desired. (vivideconomics)

General recommendations (for aggregate evidence)
 Priority should be given to evaluation methods that can provide explanation (e.g. theory based
and realist evaluation approaches).
 A mixed method approach should be taken, involving both quantitative and qualitative methods.
 Identify the mechanisms and sets of circumstances under which PBR approaches can most
likely result in behavioural change
 Explore unintended consequences of incentive approaches
 Explore cost effectiveness leading to changes in outcomes.
 A hands-off approach should not be seen as a barrier to independent evaluation (although the
nature of the evaluation, so that it does not necessarily become intrusive or dictate the form of
programme approach, should be open to negotiation) (Perrin)
 Investigate their potential to improve overall sector outcomes / MDGs,
 Explore the possibilites of combining RBF models and/or mixes of assistance (Gorter et al.)
Report of the Working Group Workshop, on 26th November 2012, Brussels.
2
Gorter et al (2012) : focus on maternal and neonatal health and low and middle income countries, base of
evidence: 70 individual, 60 syntheses. Pearson (2011) : focus on health sector, incl. United Kingdom. Pereira et al.
(2012) : focus on aid efficiency; review 5 multi or large donor aid mesures. Perrin (2013) : focus on quality of
evidence; base of evidence: 19 individual evaluations, 15 syntheses. Vivideconomics (2012) : focus on the
energy sector and applying principal-agent theoretic modeling.
3
not even the second question that one should ask when assessing effectivenes, i.e. regarding factors influencing
the achievement or non achievement of the objectives. From Burt Perrin, ‘Evaluation of Payment by Results
(PBR): Current Approaches, Future Needs’ (Dfid, 2013), p. 12.
4
Mark Pearson, ‘Results Based Aid and Results Based Financing: What Are They? Have They Delivered
Results?’ (HLSP-Institute).
5
applying himself the Organisation for Economic Co-operation and Development (OECD)/Development
Assistance Committee (DAC) Evaluation Quality Standards. Perrin. p.ii.
6
Authors' estimate, since bibliographic information frequently does not inform about who commissioned the
study.
7
" Performance targets for health service delivery such as immunization rates and trained female birth attendants
have been chosen because they are thought to reflect the overall fitness of the system to attend to the needs of
maternal and child health. The danger is that, once selected as targets for PBP, they cease to reflect this broader
system goal and just become a measure of the ability of an organization either to meet this specific target, or fool
the purchaser into believing that they have done so."
The same authors on potential negative consequences:
Kerstin Meyer, March 2013
11
"The focus on specific performance targets encouraged by PBP may have several unintended consequences. More
easily quantifiable measures of performance may often dominate over measures of quality of performance. (...)
Performance targets may also inappropriately skew a provider’s focus. If a variable such as community
participation or coordination with the Ministry is not factored into the assessment of performance because it is
difficult to measure, it may be neglected despite its importance for the long-term development of a sustainable
health delivery strategy. Performance targets that are difficult to quantify may also be neglected because they
detract from the ability in the short term to deliver on specific targets. For example, Low-Beer et al. (2007)
question whether the Global Fund’s PBP based on disease-specific targets may have negative effects on general
health systems strengthening".
in: Eldridge, Cynthia, and Natasha Palmer, ‘Performance-based Payment: Some Reflections on the Discourse,
Evidence and Unanswered Questions’, p.164 f.
8
PIA (Poverty Impact Assessment) by OECD/DAC. It can be used for short or long desk studies or for a
stakeholder dialogue on prospective systemic impact. For description of an application as part of a stakeholder
dialogue see also Dia I. and Meyer K. "Thinking Things Through, Senegal's first PIA" Poverty in Focus,
International Poverty Centre. Brasilia. April 2008.
9
Recent examples of regional policy processes: trade, customs and free movement; regional infrastructural
development; extractive industries; citizens’ participation.
10
Land, T. and Hauck, V., ‘Building Coherence Between Sector Reforms and Decentralisation: Do SWAps
Provide the Missing Link? ECDPM Discussion Paper No. 49’ (Maastricht : European Centre for Development
Policy Management, 2003), p. 34.
11
Examples: "(...) a tool for governments who aim to improve accountability in the system " (a) ; "(...) the
process of verification improves overall transparency; the World Bank’s guidance on Program-for-Results
Financing in turn states that greater transparency ‘will greatly contribute to improved accountability (...)" (b+c) ;
"Due to its focus on monitoring and verification, PBR can improve the accountability of aid recipients, both of the
people they serve, and of donors" (d)
(a) Gorter et al. , p.40 ; (b+c) quoted in vivideconomics, p.44 ; (d) Birdsall & Savedoff, 2011; Eichler & Levine,
2009, quoted in : vivideconomics p. 38
12
What is meant by "involved Stakeholders" cannot be specified as it depends entirely on the system one is
referring to; organisations, departments, staff, clients, beneficiaries, colleagues, peers, donors, supervisors, civil
society, regulators, journalists, parliaments, interest groups, service providers or downstream contracting
partners... it may also include donors or technical and financial partners.
13
A monthly bonus is paid by an NGO (itself under contract to run the district health services) based on the
amount of patients seen. This bonus is divided up amongst staff. Source: Eldridge, Cynthia, and Natasha Palmer,
‘Performance-based Payment: Some Reflections on the Discourse, Evidence and Unanswered Questions’
14
For a structured discussion of this management tool and accompanying advisory services, also in the context of
Results Based Aid, see: Thomas Taraschewski and Alexander Wegener, ‘Assessing Public Sector Performance.
Outcome Oriented Measurement Systems for Subnational Governance in Practice. " Deutsche Gesellschaft für
Internationale Zusammenarbeit (GIZ), Jakarta. 2011.
15
"One case in point is the use of different monitoring regimes for incentives to reduce teacher absenteeism.
These were shown to work well when administered impersonally, and perceived to be monitored in that way (...)
However, in settings when supervisors were given discretion over administering incentives, these programs
became entirely ineffective’ (J-PAL, 2012). Thus similar incentives, monitored in different fashions, can have
radically different impacts – showing the importance of well-designed monitoring structures". vivideconomics,
p.45.
16
Incidentally, the term "ownership" is overly simplifying and has distracting connotations, when one considers
the collective, multi-layered and self-organised quality of what it should mean in practice in the development
context.
Kerstin Meyer, March 2013
12
Bibliography
Cockburn, Elli, ‘DFID Pilots on Payment by Results’ (Department for International
Development DFID, 2012)
Dia, Ibrahima, and Kerstin Meyer, ‘Thinking Things Through. Senegal’s First PIA.’,
Poverty in Focus, April 2008, 24–25
Eldridge, Cynthia, and Natasha Palmer, ‘Performance-based Payment: Some Reflections
on the Discourse, Evidence and Unanswered Questions’
Gorter, Anna C., Por Ir, and Bruno Meessen, ‘EVIDENCE BRIEF: Results‐Based
Financing of Maternal and Neonatal Health Care in Low- and Lower-Middle Income
Countries; Background Paper, Revised Draft’, 2012
Government of Rwanda Ministry of Health, ‘Rwanda Health Financing Policy’, 2009
Klingebiel, Stephan, ‘Ergebnisbasierte EZ-Ansä tze: Definitionen, Kriterien und
Konzepte. Ein Diskussionspapier.’, (Deutsches Institut für Entwicklungspolitik, Bonn,
2012)
---, ‘Results-Based Aid (RBA) New Aid Approaches, Limitations and the Application to
Promote Good Governance’ (Deutsches Institut für Entwicklungspolitik, Bonn, 2012)
Krempin, Michael, and Jörg Freiberg, ‘Aid on Delivery in the Context of Results-Based
Approaches. Discussion Paper.’ (GIZ, 2011)
Land, T., and Hauck, V., ‘Building Coherence Between Sector Reforms and
Decentralisation: Do SWAps Provide the Missing Link? ECDPM Discussion Paper No. 49’
(Maastricht : European Centre for Development Policy Management, 2003)
Pearson, Mark, Robin Ellison, and Martin Johnson, ‘Review of Major Results Based Aid
(RBA) and Results Based Financing (RBF) Schemes’ (DFID, 2010)
Pearson, Mark, ‘Results Based Aid and Results Based Financing: What Are They? Have
They Delivered Results?’ (HLSP-Institute)
Pereira, Javier, and Carlos Villota, ‘Hitting the Target’ (EURODAD, 2012)
Perrin, Burt, ‘Evaluation of Payment by Results (PBR): Current Approaches, Future
Needs’ (DFID, 2013)
Taraschewski, Thomas, and Alexander Wegener, ‘Assessing Public Sector Performance
Outcome Oriented Measurement Systems for Subnational Governance in Practice’
(Deutsche Gesellschaft fü r Internationale Zusammenarbeit GIZ, 2011)
vivideconomics, ‘An Operational Guide for Results Based Financing Approaches Report
Prepared for ESMAP Draft Report’, 2012
Worldbank, ‘A New Instrument to Advance Development Effectivenesss: Program-forResults Financing’ (Operations Policy and Country Services, 2011)
Kerstin Meyer, March 2013
13