Performance Improvement
Framework
Self Review Training
24 June 2010
Central Agencies
l DPMC l SSC l Treasury
Mythological Ministry of Magic
Mission: Making Money Materialise
50%
Staff Growth (FTE)
Personnel Expense per Employee (FTE)
Change in Real Price or Staff
40%
Personnel Expenses (ex. ACC)
Total Operating Expenses
30%
20%
10%
0%
99/00
00/01
01/02
02/03
03/04
04/05
05/06
06/07
07/08
HYEFU
08/09
-10%
Financial Year
PIF challenge: Showing value to a sceptical public (or reviewer)
2
Central Agencies
l DPMC l SSC l Treasury
#3.
Cost
Effectiveness
VALUE
FOR MONEY
(Best Result for NZers)
#1.
Technical
Efficiency
#2.
Allocative
Efficiency
Value for Money is hard to assess directly … but poor results
at ANY corner of the ‘triangle’ shows under-performance
Central Agencies
l DPMC l SSC l Treasury
#1. PRICE, QUANTITY & STANDARDS (TECHNICAL EFFICIENCY)
‘Did new spending
actually increase or
improve output ?’
Effect of Flying Hours on Maritime Patrol Prices
80%
Gulf of Oman (Output
D16 - 659 extra hours)
Total flying hours
Real change in price or output
70%
Real cost per flying hour
60%
Real cost (Personnel)
50%
Real cost (Total Expenses)
1.
40%
2.
30%
20%
10%
0%
FY 98/99
FY 99/00
FY 00/01
FY 01/02
FY 02/03
FY 03/04
-10%
-20%
Financial Year
FY 04/05
FY 05/06
FY 06/07
3.
4.
5.
How did prices of
big outputs change?
Did output volume
or quality change
proportionately?
Did quality change?
Did o’heads grow?
Can we show
citizens actually
benefit from this?
Central Agencies
l DPMC l SSC l Treasury
#2. DID OUTPUT GET TO THE PROBLEM AREA, & EFFECT CHANGE
Allocative Efficiency
‘Can we get better results
Extra Prisoners ('03 vs '07)
Large Proportion of New Prison Space Used
for Less Serious or Less Risky Offenders
 Can we link extra $$$
& output to priority
groups, areas, needs &
opportunities?
300
250
200
150
100



50
0
Seriousness Score
-50
-150
"0"
"1"
"2"
"3"
"4"
"5"
Seriousness Score 198
251
218
137
63
-1
-157
155
127
85
99
81
Reoffending Risk
"6"
129
Who has / is the problem?
Who got interventions or $$$?
Who missed out?
 How well do allocative
rules really work?
Reoffending Risk
-100
-200
without spending more ?’
"7"
"8"
15
104
164
118
" 9 " " 10 "
17
247
 Can we deliver more in
areas that will benefit
more, & limit waste?
Central Agencies
l DPMC l SSC l Treasury
#3. INCREASED PRICE (OR OUTPUT) IMPROVES CORE OUTCOMES
Look for diminishing returns.
Scale & timing ($ vs. outcome)?
20%
10%
/0
7
06
6
/0
05
5
/0
04
4
/0
03
3
/0
-10%
02
2
0%
/0
Did end outcomes improve much?
Did gains mirror / equal spending?
Did gains precede or lag changes?
Intermediate outcomes improved?
30%
01
1.
2.
3.
4.
Investing to reduce crime …
Change Relative to FY 01/02
‘Are results good, bad
or simply indifferent ?’
-20%
Real Funding
Violence
Drugs & AntiSocial
Dishonesty
ERO Approach - Context

ERO evaluates the performance of schools and early
childhood services

ERO’s self-review was undertaken as a pilot review
for the PIF (July 2009)

The context for ERO’s self-review was “the alignment
of services and supporting infrastructure to better
support Government priorities for education”

This organisation-wide context meant that ERO
tested all aspects of the framework
Brent McPherson- June 2010
ERO Approach - Team
Level
Position
(tier 2)
National Manager
Corporate Services
(tier 2)
National Manager
Review Services (Central)
(tier 3)
Strategy & Performance Manager
(tier 3)
Manager Information Services
(tier 3)
(tier 3)
The team:

was appointed by the
Chief Executive

had knowledge about the
business and governance

had access to information

had evaluation skills
Chief Finance Officer
Manager Standards & Contracts
Brent McPherson- June 2010
ERO Approach - Method

Critical areas identified in the framework were allocated
between review members - in pairs
(allocated areas were independent to current portfolios)

Each pair recorded their assessment, action points &
evidence in a template (to justify their judgement)

The team synthesised results & discussed judgements
(moderation)

The report (and recommendations) was discussed with
the Senior Management team
Brent McPherson- June 2010
Benefits to ERO
The self-review:

reinforced a commitment to all staff to promote a
culture of continuous improvement

reinforced ERO’s own credibility
(ERO promotes evaluation capacity externally)

identified a set of recommendations for action
(the recommendations have been accepted by SMT
and integrated into the work programme)

encouraged the Chief Executive to think about
ERO’s future approach to internal evaluation
Brent McPherson- June 2010
Why Do A Self-Review?

respond to Govt priorities (ie how to “maximise the
value received from its education services”)

promote continuous improvement through a formal
and structured approach

promote accountability – and a balance of internal
and external review (ERO ‘s approach to schools)

prepare for a formal review

review the organisation (do it in chunks?)

basis for engaging with the Minister (PIAs, SOI)
Brent McPherson- June 2010
Issues and Challenges
To improve the value of the self-review, next time ERO
would….. plan ahead
 schedule at a time appropriate to ERO
(and set time aside to conduct the self-review)

engage managers and staff in the planning
(and engage the senior management team throughout)

identify priorities – to reduce time spent on areas where
there are high levels of comfort

incorporate a greater field component - to test the cascade
of initiatives to the field (ERO is a field agency)

develop an associated communications plan
Brent McPherson- June 2010
Issues and Challenges
ERO removed bias by:

ensuring that review team members did not review
their own portfolio

using a moderation process to synthesise the results

ensuring that the views expressed were backed up
by supporting evidence
Brent McPherson- June 2010
Issues and Challenges
Other recommendations/challenges:

treat it as a formal review (ready the documents)

ensure the team has access to evaluation skills

plan for conflicting opinion

what is the current culture of the agency?
Brent McPherson- June 2010