TECHNICAL OVERSIGHT OF
ANNUAL REPORTING
Portfolio Committee of Public Works
11 March 2008
2
Session Outcomes
 Explaining PEM cycle
 Evaluating Measurable Objectives
 Critiquing Performance Measures &
Targets
 Legislative Oversight of Annual Reporting
 Alignment between SPP and AR
Public Expenditure
Management Cycle
3
Planning
Reporting
Budgeting
Expenditure monitoring
and management
4
Assessing Performance
Information
6
Overview: Components of SPP
Five-year SPP
Part A
Whole of Department
Situation Analysis
Strategic Goals
Overall Strategic
Objectives
Part B
Programme
Measurable Objectives
Situation Analysis
Subprogramme
Situation Analysis
(if required)
Resource Information
Part C
Background Information
Analysis of Service Delivery Environment
Organisational and Institutional Environment
7
Why Evaluate Performance?
• Achieving objectives?
• Outputs delivered?
• Institutional comparisons
• Productivity
• Output – outcome?
8
Defining objectives
“Measurable objectives” need to:
 Reflect organisational priorities.
 Be related to activities and resources.
 Adhere to S.M.A.R.T. principle:
Specific
Measurable
Appropriate
Realistic
Time-bound
E, and R?
9
Three components of a good
objective
1. Primary output that the programme
will achieve.
2. Intended impact that the programme’s
output will have on the public or
client.
3. Level of performance.
•
The desired level of service delivery
10
Objectives checklist
Primary output
Is it observable?
Intended impact
Can you see what
impact it will
have?
Level of performance
Is it measurable?
11
What’s wrong with this MO?
The objective of
Further Education and
Training (FET) is to
provide services in
terms of the FET Act.
12
It only refers to activities. It
does not refer to any level of
performance – and is therefore
not measurable!
And, even more
importantly, it does
not state the
impact of this
objective on
society.
13
Revised Objective
“To provide FET colleges with the
required resources necessary to roll out
and implement the requirements of the
FET Act in KZN.”
Further improvements?
14
Change in focus
 Measurable objectives change the focus from
activities to outputs and outcomes.
 For example:
From
Administer polio vaccines to children under 6
years old in certain hospitals.
To
To eradicate polio among children under 6
years in certain areas.
15
MO from Public Works
 Programme 3: National Public Works P.
MO/output: Formalized mentorship
programme as a regulated profession
 Is there an output?
 Is there an outcome?
16
Exercise
 Choose a programme from DoPW.
Examine MOs for that programme.
1.) Assess whether it contains all the
necessary components of an MO.
2.) Are all MOs in the control of the
department?
3.) Are they aligned to the strategic obj?
17
Performance Measures &
Indicators
 Performance measures and indicators are
statements that describe the dimension of
performance that is to be monitored.
 The dimension of performance to be
monitored must be the most appropriate and
under the control of the component.
18
Criteria for Performance
Measures and Indicators:
NT’s “Framework for Managing Programme Performance
Information” - 2007
 Reliable: accurate for intended use, respond to
changes in level of performance
 Well-defined: clear, unambiguous definition,
consistency
 Verifiable: validate the processes and systems that
produce indicator
 Cost-effective: usefulness of indicator to justify
cost of collecting data
 Appropriate: indicator must avoid unintended
consequences
 Relevant: relate logically and directly to aspect of
mandate
19
Dimensions of
Performance Measures
 Quantity
 Cost
 Quality
 Timeliness
20
Quantity
 Describe outputs in terms of how
much or how many.
 Require a unit of measurement (e.g.
kg, litres, km).
 Examples:
• number of students passing per year per
grade;
• number of schools build;
• number of earmarked FET colleges invested in;
• number of finance management personnel on
SCoA, BAS training.
21
Cost
 Should reflect full cost of producing an
output
 Should include unit cost for each
deliverable described under quantity
targets
 Examples
• Cost per unit of materials used;
• Average annual operating cost per learner per
year;
• Cost per ABET targeted individual served;
• Total operating expenditure.
22
Quality
 Reflect service standards based on customer needs
and contribute to government outcomes.
 Product or service should fit intended purpose.
 Balance efficiency with effectiveness so that price is
not predominant factor.
 May address:
• Parent relations, quality of schooling.
 Examples:
• Number of parent complaints filed;
• Minimum standards of electronic connectivity at schools;
• Minimum set of qualifications attained by teachers.
23
Timeliness
 Provide parameters for how often, or within what
time-frame, outputs will be delivered.
 Measured by turnaround times, waiting or response
times (deliver service yearly/quarterly).
 Examples:
• Whether the brief and instructions to the Minister have been
completed within the deadline;
• Proportion of case reviews conducted by due date;
• Percentage of responses answered within a given timeline;
• Children enrolled yearly.
24
PMI from DoPW
 SP 3.1: Constr. Industry Development Policy
& Monitoring
 PMI – Implement and Monitor HIV/AIDS
Policy
 Target – Ongoing
 Is the PMI measurable?
 Is it one of four performance dimensions?
25
Constraints & MOs/PMIs
 SP constraints to be addressed by MOs and
PMIs
 SP 3.1: Constr. Industry Development
Programme [P49 of SPP]
 Lack of co-ordination of infrastructure depts
 Unco-ordinated BEE programmes
 Obstacles such as access to credit for
emerging enterprises
 Lack of appropriate skills in the industries
26
Exercise – Measures &
Constraints
 Exercise: Choose a SP. Evaluate
whether SP address the constraints by
specifying MOs and PMIs for them.
27
Developing Performance Measures &
Indicators
Be aware of perverse incentives!
E.g. if a performance measure is the number of
schools built, property developers might use
cheap labour and cheap building materials in
order to increase productivity levels, but in fact
effectiveness and efficiency are compromised.
E.g.: “number of policies, guidelines, and
legislation formulated…”
28
Perverse Incentives
Activity: For a measurable objective in the Department of
Public Works, give an example of how focussing on
one dimension of performance (eg. cost, quantity) can
undermine other performance dimensions (eg quality,
equity etc).
Type: Group
Time: 20 min.
Require: Flipchart sheet and markers
29
Developing Performance Targets
Key Characteristics of Targets
 Defined in precise terms relating to delivery
of outputs
 Relate to a single performance measure of a
particular output
 Specify a time frame or milestone
 Are measurable i.e. actual numbers and
percentages (not terms like increase,
decrease or optimal unless quantified)
 Linked to baseline achievements
30
Performance Targets
 Set the quantity, quality, cost and timeliness
levels for output delivery
 Governments will use targets to:
• set delivery levels; and
• assess departmental performance.
 To ensure targets are achieved, need to involve
all stakeholders in the process
 SMART applies
31
Examples:
Performance targets
 Performance target (1)
• Less than 10% permanent staff turnover
rate
 Performance target (2)
• 1:34 teacher/learner ratio for KZN.
32
Legislative Oversight of AR (1)
 S55 (2) of Constitution outlines oversight powers
of NA
 AR allows parliament to evaluate performance of
dept after financial yr
 PFMA requires AO to table performance targets
for their dept – ENE & SP
 Challenge for PC is to get depts to provide good
quality perf. info. with tight perf. targets & then
get depts to report against these in AR
33
Oversight Process
34
Oversight Process (2)
 Key question: How did the executive
perform in using its budget effectively
to deliver services?
 Biggest weakness is the poor quality of
the non – financial performance
information
35
Consideration of AR
1. What is the technical quality of AR?
2. Does the dept report on each and every
performance target specified in ENE & budget?
3. What is the quality of perf. info. as highlighted by
performance audit by A-G?
4. Is the dept. achieving economy, efficiency &
effectiveness?
5. Equity in service delivery?
6. Evaluating mgt’s explanations of why dept’s
peformance did not attain targets set in SP and
budget
7. Investigating circumstances of under & overexpenditure & its impact on service delivery
36
Alignment of AR and SPP
 PMIs in SPP should be reported on in AR
 PMIs and planned targets should be
consistent
 Poor specification of PMIs in SPP
non-reporting in AR
 Require internal processes to capture non
– financial information
37
Example - DoPW
 P3 (NPWP);SP1: Construction Industry Development
Programme








Output 1 (ENE) – Construction Industry T. Charter
PMI 1 – Charter Gazetted
Target 1– Dec 2006
Output 2 – Contractors exit from incubator progr.
PMI 2 – No. of contractors graduated
Target 2 – At least 50 contractors by Dec 2008
SPP – Pge 53
AR – Pge 35
38
Exercise - DoPW
 Examine PMIs for a Programme in the
DoPW budget statement. Cross – check
SPP to see if PMIs are recorded there.
 Turn to AR and assess whether it
reports on these PMIs.
END
Questions?