Evaluating the
Performance of PBOs
Helaina Gaspard
Institute of Fiscal Studies and Democracy (IFSD)
June 6-10, 2016
World Bank GN-PBO Annual Meeting
Washington, DC
USA
Why evaluate PBOs?
1.
Support to parliament

2.
Parliament’s scrutiny function derives from the Magna Carta 1215
Legitimacy and survivability of the office

Credibility of outputs; perceptions of stakeholders; and organizational
factors can influence a PBO’s operation
Why a common evaluation framework?
 PBOs are emerging globally – What works? What doesn’t?
 An evaluation framework can work in two ways:
1.
To assess the work of an existing PBO
2.
To establish the parameters for a new PBO
 A common evaluation framework can be applied based on context,
inputs, outputs and outcomes.
An Evaluation Framework for PBOs
CONTEXT
INPUTS
 Legislation
 Budget
 Sources of
accountability
 Human resources
 Access to data
 Defining
stakeholders
 Independence
OUTPUTS
 Peer review of
reports (scientific
and
methodological
soundness)
 Transparency
gains
 Assessment
against
counterfactuals
 Assessment
against peers
OUTCOMES
 Stakeholder
perceptions (via
surveys)
 Media impact
analysis
 Expert interviews
Case study - the 2014 UK OBR review
 First review of the UK OBR
 Context, input, output,
outcome frame
 Defined performance – better
decision support and
enhanced transparency, rather
than being “right”
 Provided advice to support
organizational sustainability
 Helped to build legitimacy or
the fledgling institution
Context
 Constitutional and political ecosystem.
 Founding legislation; parameters of the PBO.
 Legislation guarantees a minimum.
Points of Departure
Parliament
Legislators
Legislation
Cabinet
PBO
Civil Service
Imperative
(t0)
Implementation
Civil Society
Media
Sustainability
Inputs
 Resources: both human and monetary that enable a PBO to fulfill
their mandate.
 Access to data: can a PBO get what it needs from the executive
branch to do its work?
 Independence: is a PBO free from political influences to select its
staff, head and fulfill its mandate?
 Don’t let the numbers fool you – you can do a lot with a little.
Assessment, Costing &
Forecasting
300
250
Assessment &
Forecasting
200
150
Assessment
100
Assessment &
Costing
50
0
5
10
15
20
25
30
Total staff and leadership (number)
Annual budget (£m)
Inputs of OECD IFIs. Source: Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 32.
Independence of the OBR. Source: Data sourced from the Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 34.
Outputs
 The products or reports prepared and released by the PBO.
 Evaluating outputs is a technical assessment of a PBO’s products.





Are they credible?
Are they methodologically sound?
Are they transparent?
Are they peer-reviewed?
Are they comparable to those of peers?
Process
Defined
Transparent
Disciplined
Economy
Indicators assessed over
multiple years
Errors analysed by level,
growth rate,
contributions to growth
and probabilistic
estimates
Public finances
Lessons learned
Details of receipts and
spending assessed over 3
year period
Transparency drives
agenda for forecast
Changes due to
improvement
economic and fiscal
forecast, policy and
classification assessed on
annual basis
Forecast evaluation transparency gains. Source: Review of the United Kingdom’s Office for Budget Responsibility (2014), p. 61.
Outcomes

Perceptions and impacts of a PBO matter for sustainability
 Identify a PBO’s key stakeholders (from legislation and convention)
 Survey them, interview them and track media results

A PBO can exist on paper but if it cannot communicate with stakeholders, it
may become irrelevant

Assessing outcomes:
 How do parliamentarians perceive the PBO? Do they leverage their
outputs?
 Does the PBO resonate in the media?
200
¨
180
160
140
120
100
¨
80
¨
60
40
Factiva analysis of OBR media citations and mentions (2010-2014). Source: Review of the United Kingdom’s Office for Budget
Responsibility (2014), p. 72.
Jun-14
Apr-14
Feb-14
Dec-13
Oct-13
Aug-13
Jun-13
Apr-13
Feb-13
Dec-12
Oct-12
Aug-12
Jun-12
Apr-12
Feb-12
Dec-11
Oct-11
Aug-11
Jun-11
Feb-11
Dec-10
Oct-10
0
Apr-11
20
 Note: Parliamentary survey results from the OBR evaluation were
unusable due to a lack of response rates
 Only 52 responses out of a potential 1,427 (650 Members of Parliament
and 777 Peers (House of Lords))
 5 from MPs, 47 responses from Peers
 A survey of Canadian parliamentarians (February 2015-August 2015)
yielded a 20% response rate (considered statistically relevant)
Establishing a PBO with the evaluation framework
 “Flip the framework on its head” and build evaluation into your
design
 It can imbue your office with credibility, motivation and institutional
sustainability
 Identify your desired outcomes; consider your context:
 What inputs and outputs are required to fulfill your goals?
Context
Outcomes
Inputs
Outputs
Why a common framework?
 Baselines
 Define a basic common standard of institutional organization and
products
 What are the fundamental common attributes of PBOs?
 Benchmarks
 Establish peer groups for assessment and comparison
 You know you have succeeded when…
 Aspirational institutions
 Models of successful PBOs can be useful examples for new institutions
Conclusion
 Cement dries quickly – you must act to set precedents of practices
 No legislation is perfect. Use your inputs and outputs to maximize
your mandate and your outcomes for stakeholders
 With strong leadership, you can maximize mandate by ensuring that
the office’s products are relevant, their content authoritative and
their release timely and foster sustainability
Thank you
Institute of Fiscal Studies and Democracy
University of Ottawa
www.ifsd.ca
[email protected]