TRANSPO
ORT RE
EFORM
M OUTC
COMES
S FRAMEWO
ORK
Worrking Pa
aper
Dece
ember 2010
2
National Transport Commission
TRANSPORT REFORM OUTCOMES FRAMEWORK
ISBN:
Date: December 2010
Report outline
Title:
Transport Reform Outcomes Framework
Type of report:
Working paper
Purpose:
Inform the development of a framework to improve
reform outcomes
Abstract:
In response to the Review of NTC 2009, NTC has
explored best practice methodologies to help the NTC
plan, monitor, evaluate and report reforms to improve
their impact.
Chief Executive Officer
National Transport Commission
L15/628 Bourke Street
MELBOURNE VIC 3000
Key milestones:
December 2010
Key words:
reform outcomes
Acknowledgements
This paper based on advice from Jess Dart, Clear
Horizon; Kieran Keleher, PMSA; Kym Davis,
Ausproject; AECOM and C-Marc
Contact:
Paul Sullivan, Chief Officer-Strategy
Table of contents
1. Introduction
3 1.1 NTC Review 2009 findings
3 1.2 CoAG Reform Council observations
3 1.3 Definition of an outcome (impact)
4 1.4 Management by Outcomes
4 1.5 Outcomes-Based Accountability (OBA)
4 1.6 Program Logic
6 Transport Reform Outcomes Framework
7 2.1 A framework to deliver outcomes
7 2.2 Components of the framework
8 2.3 Planning & Reporting Process
8 Planning for outcomes
2 3.1 Strategic Planning
2 3.2 Work Programme
2 3.3 Strategic planning in government
2 3.4 Planning for outcomes
3 Monitoring and reporting outcomes
4 4.1 Background
4 4.2 Hierarchy of objectives and outcomes
4 4.3 Objectives and outcomes
4 4.4 Intermediate outcomes
4 4.5 Monitoring IGA goals and intermediate objectives
5 2. 3. 4. Example: safe system logic
5 4.1 Monitoring IGA goals and intermediate objectives
6 4.1 Data availability
6 Evaluation of Outcomes
7 5.1 Background
7 5.2 Effectiveness Evaluation
7 5.3 Evaluating the implementation and effectiveness of reform
8 5.4 Implementation status reporting
8 Work program (outputs) reporting
9 Project status report
9 5. 6. 6.1 7. 7.1 Appendix A
10 Collaborative Performance Story Reporting - overview
10 1. Introduction
1. Outcomes-based accountability
1.1
NTC Review 2009 findings
National Transport Commission Review (December 2009)1 commentary on transport reform performance:
“The next phase of transport reform will be challenging as the focus shifts from improving individual modes to
lifting the performance of the national transport system as a whole. Reform development needs to
holistically consider impacts on productivity, safety, pricing, network access and land-use planning and
investment.”
“The overarching objective of the current IGA is to improve transport productivity, efficiency, safety and
environmental performance, and regulatory efficiency in a uniform or nationally consistent manner. Therefore
the NTC’s performance must ultimately be assessed in terms of its impact against these indicators.
“Governments and industry need to address significant transport safety, productivity, efficiency and
environmental performance issues to successfully respond to the predicted transport growth and to
underpin Australia’s further economic and social development …
“Addressing these issues is not entirely within the control of the NTC, so we believe responsibility for the
outcomes delivered must be shared by jurisdictions. However, the NTC’s performance must ultimately be
judged in terms of impact delivered, even though it does not have direct responsibility for either transport
policy or reform implementation …
“the IGA should be changed to require jurisdictions to provide information to the NTC on the outcomes for
transport safety, productivity, efficiency and environmental performance…
“(the lack of consistent data) impedes the ability to optimise performance across modes and throughout the
entire transport system
“Greater emphasis on outcomes-based reporting will provide a feedback loop to guide prioritisation and
resourcing of the reform agenda and associated work program …
“Our recommendations will enhance governance outcomes by ensuring that the SCOT is engaged in a
constructive dialogue with the NTC on its performance goals, work program and areas for improvement at
an operational level.”
Recommendation 7: The NTC should regularly report to the ATC on progress of priority projects, including
reform development, implementation and impact2.
ATC Response: Agree. The NTC, in consultation with jurisdictions, will develop an activity and performance
reporting system for approval by ATC in line with the review recommendation.
1.2
CoAG Reform Council observations
In September 2010, the CoAG Reform Council released a progress report, COAG reform agenda: Report on
progress 20103, which called on governments to focus on cooperation, outcomes and public accountability.
Recommendations for improved performance reporting include better data, clear goals and performance
indicators to sharpen the focus on performance against key outcomes.
1
Source: http://www.infrastructure.gov.au/transport/australia/ntc/
2
NTC’s submission to the review included recommendations for articulated goals and objectives (measureable); a public reporting framework; and an
outcomes-based approach to implementation.
3
http://www.coagreformcouncil.gov.au/reports/docs/COAG_reform_agenda_2010_exec_summary.doc
1.3
Definition of an outcome (impact)
“a final product or end result; impact; consequence”
In the language of planning and program theory, an ‘outcome’ is conceptualised as the desired end result or
impact of a program. It is where you want to get to, rather than how you will get there.
According to C-Marc (2011)4, outcomes are results valued by the community. In the transport context, this
includes safe journeys, less travel time, affordability, reliability, health and security.
1.4
Management by Outcomes
Over the last decade, the focus of transport reform has evolved from “break of gauge” regulatory issues to
“better regulation” in a more complex reform environment; often requiring complementary measures to
deliver an optimal outcome (eg: targeted rest areas to support fatigue reform).
In recent years the evaluation and reporting of reform effectiveness has also undergone a shift from
attempting to ‘prove’ whether a program has achieved outcomes, to a softer focus: striving to understand the
extent to which the program has contributed to outcomes and using this knowledge as a ‘learning tool’ to
improve programming (Mayne 1999).
As observed by the Review of NTC 2009, it is often unclear how program activities and outputs contribute to
outcomes and broader goals. The review signalled the importance of identifying how programs actually
impact on their intended outcomes. This reflects the movement towards ‘management by outcomes’.
1.5
Outcomes-Based Accountability (OBA)
Outcomes based accountability is a conceptual approach to planning and assessing performance that
focuses attention on the results – or outcomes – that the reforms are intended to achieve. It is also seen as
much more than a tool for planning effective programs; and making better outcomes the primary purpose of
the organisation and its employees (NTC Review Steering Committee 2009).
Much of the thinking and work done on outcomes (outcomes based accountability or results based
accountability) has been informed by the work of Mark Friedman from the United States5. Further
distinguishing features of the approach are
•
The use of simple and clear language
•
The collection and use of relevant data
•
The involvement of stakeholders, including service users and the wider community, in
achieving better outcomes
•
The distinction between accountability for program delivery performance on the one hand,
and accountability for outcomes among a particular population on the other.
4
Performance Reporting of Transport Reform Outcomes: Measuring the value of what can be managed (C-Marc 2011) – see NTC website under
publications
5
Friedman M (2005) Trying Hard is not Good Enough: how
Table 1: An analysis of outcomes models
Model
PLANNING TOOLS
PROGRAM &
RESOURCE
ALIGNMENT
Diagrammatic
representation of a
program, showing what it
is supposed to do, with
whom, and why
Key Concepts
Strong Points
Well suited for
Inputs, outputs,
outcomes; arrows show
relationships between
elements in the model
Easy to use; provides easily Program overview;
understood representation
presentations; program
of program’s theory of
and evaluation planning
change
Investor return, results,
customers, milestones,
performance targets,
outcome statement
Highly disciplined approach
that serves both program
investors and implementers;
Web-based software has
strengthened usability
Government and
philanthropic grant-making;
program and organization
management
ResultsBased
Accountability
Real-time approach that
describes what desired
results look like, defines
results in measurable
terms, and uses measures
to drive action plans for
improvement
Results, experience,
indicators, baselines,
strategy, action plan and
budget, accountability
Thorough system for
planning communitychange efforts and
improvements in program,
agency, or system
performance; uses lay
language and provides
direct link to budgeting;
useful for integrating
different outcome systems
Project planning and startup; development of
community report cards;
program/agency
improvement plans and
budgets; grant-making and
evaluation design
Targeting
Outcomes
of
Programs
Tracking progress toward
achievement targets;
evaluating degree to which
programs impact targeted
conditions
Knowledge, attitude,
skills, and aspiration;
process, outcome, and
impact evaluation
Fairly easy to use; helps
integrate program
development and
evaluation; implementers
and managers can use
same concepts
Program design and
evaluation
Balanced
Scorecard
Business-based model
designed to provide
integrated management
and accounting for
multiple variables
impacting organization
performance by
connecting them to a set
of performance indicators
Strategy, alignment,
short- and long-term
objectives; financial and
nonfinancial measures;
lagging and leading
indicators; performance
measures and drivers;
internal and external
indices of success
Allows for a graphic
assessment of the degree to
which an organization’s
resources and efforts
support its goals
Monitoring either a single
program with several
associated initiatives or
multiple programs within an
organization; analysing
alignment of resources and
initiatives to strategic
targets
Graphic tool that centres
Scales; mutually
around a series of scales exclusive, multiple, and
and their placement within floating indicators
a matrix designed to
illustrate progress along a
continuum of stages
Places a client, community,
or program on a continuum;
shows incremental and
relative progress,
stabilization, or decline;
individual data together tell
a complete story;
behaviourally anchored
description of levels of
change
Demonstration of
aggregate progress;
measuring concepts that
are not easily quantified
Results
Mapping
Outcome-based
Causal and synchronistic
evaluation tool designed to attribution; levels and
systematically capture
milestones
otherwise non-quantifiable
anecdotal evidence
Way to systemise,
standardise, gather, and
utilise lessons embedded in
anecdotal information
Turning anecdotal
information into a useful
tool for program
presentation, evaluation,
and assessment
Program
Results
Story
Uses stories to capture
Results, stories,
organisations’
anecdotal evidence
achievements and present
them in a results-based
format
Easily understood approach Presenting program and
for presenting results; brings results to multiple
outcomes to human interest audiences
level; captures and conveys
richness of information
Program
Logic
Model
PROGRAM REPORTING
Description
Key management focus on
the achievement of
Outcome
specific, sequential results
Funding
for customers of services;
Framework
emphasis on results, not
activity
Scales and
Ladders
Source: Robert Penna & William Philips, Rensselaerville Institute’s Centre for Outcomes
1.6
Program Logic
Program logic is a methodology to support OBA. It expresses how change is expected to occur by capturing
the rationale behind a program, probing and outlining the anticipated cause-and-effect relationships between
program activities, outputs, intermediate outcomes and longer-term desired outcomes.
Program logic is usually represented as a diagram or matrix (log frame) that shows a series of expected
consequences, not just a sequence of events (adapted from Dart 2007 and OECD 2002). It expresses how
change is expected to occur.
The concept of program logic has been applied since the 1970s to demonstrate value for money in delivering
outcomes in complex sectors, such as international aid. Since then it has been used by governments in
many different disciplines in a variety of formats. More recently it has been adapted for use in natural
resource management (NRM), environmental protection, health etc.
Program logic methodology engages stakeholders to collaboratively think, plan and communicate program
objectives and actual accomplishments. It provides a framework to:

guide planning and design of programs and projects

share understanding and ownership with stakeholders

monitor and evaluate program performance

inform learning and adaptation of strategies to improve program performance

inform partners, the community and stakeholders.
Program logic sets out a clear statement of anticipated outcomes, the assumptions underlying those
outcomes and how success will be measured. It articulates a roadmap and timetable, maintaining a focus on
the broader goals as well as the component parts.
2. Tra
ansportt Reforrm Outtcomes Fram
meworkk
2.1
A fra
amework to
o deliver ou
utcomes
The Review
w of NTC 200
09 recommen
nded the devvelopment off outcomes-b
based perform
mance reporrting for
transport reform, linked to NTC’s IGA
A goals. In itts response to
t the review
w, ATC also nnoted an incrreased
emphasis o
on “policy planning, development and implementation, data an
nd research, prioritisation and
transparenccy of consulta
ation process
ses”.
In order to rreport on outtcomes, it is important tha
at reforms an
nd the work program
p
are adequately planned
and designe
ed to achieve
e the outcom
mes. Program
m logic metho
odology links
s the activitiees, outputs an
nd
assumptions to deliver an
a outcome and
a sets obje
ectively veriffiable perform
mance indicaators.
An integrate
ed framework to deliver reform outcom
mes would in
nclude the fo
ollowing com ponents:
•
m logic)
planning (ussing program
•
monitoring (performance
(
e indicators)
•
evaluation and
a reporting
g (based on tthe logic); an
nd
•
improvemen
nt and adapttive managem
ment.
within the re
underlying principles of th
he frameworrk is that prog
gram logic is
s embedded w
eform life
One of the u
cycle (see F
Figure 1 belo
ow).
Figure 1: M
Methodology and
a reform lifecycle
PROGRA
AM
LOGIC
(OUTCOMES
M
METHODOLO
OGY)
2.2
Components of the framework
Table 2 (below) illustrates the framework’s key components and outputs. The framework’s iterative activities
recur throughout program planning, design and implementation.
The framework supports a highly participatory continual learning and improvement process involving NTC
and governments. It will facilitate a common understanding of the problems to be addressed and the
underlying assumptions about how change will occur, leading to sustained engagement and effort.
Table 2: Transport Reform Outcomes Framework
Component
Outputs
Planning (Program logic)
Desired changes, and the types and extent of changes expected
Monitoring and reporting
Evaluation and reporting
Improvement and adaptive
management
2.3

problem and objective analyses

key assumptions about how change will occur

anticipated outputs and outcomes

performance indicators

collation of qualitative and quantitative data

assess relevant data against reform outcome targets

assess relevant data to assess the effectiveness of reform
against program (intermediate) objectives and IGA goals

discuss transport system performance (with governments,
stakeholders) to identify strategic priorities for reform

conduct implementation reviews of national reforms (eg:
collaborative performance story reporting)

assess the impact, appropriateness, effectiveness, efficiency
and legacy of reforms

communicate evaluation
external stakeholders

reflect on what is working and what is not working based on
monitoring and evaluation reports

amend reform program strategies based on reflection on
monitoring results and outcome reports
results
to
shareholders
and
Planning & Reporting Process
Figure 2 and 3 (following) illustrate the planning and performance reporting framework presented to SCOT
and ATC in 2010. The proposed process aligns NTC, SCOT and ATC reporting cycles
Figure 2: NTC EXTERNAL PERFORMANCE REPORTING FRAMEWORK PROCESS
CONSULTATION on reform needs with governments,
industry, unions and transport users.
ATC/SCOT PERFORMANCE REPORTING (Jul-Dec)
MONITOR REFORM OUTCOMES
-
outcomes against IGA objectives and goals
-
outcomes against intermediate objectives and KPIs
-
outcomes against reform KPIs
INFORM
PLANNING
progress against key deliverables
-
strategic risks and interdependencies
-
future key meetings and deliverables
-
emerging new initiatives/proposals
-
NTC vision, mission, role, objectives and
supporting strategies and KPIs
BUDGETING
REFORM PROGRESS REPORT (updated regularly)
-
STRATEGIC PLAN (Jan-June)
-
based on the Commonwealth Budget Additional
Estimates CPI adjustment
WORK PLAN
REPOR
T
-
objectives-oriented project and resource plan to
implement the strategic plan
IMPLEMENTATION STATUS REPORT
-
report progress against implementation plans
-
strategic risks and barriers to achieving outcomes
ANNUAL REPORT BRIEFING
-
strategic summary (subject to timing and availability of audited
accounts)
ANNUAL REPORT (October)
- CAC Orders (Report of Operations), incl. - “material” highlights - audited annual financial analysis - corporate KPIs (over 3 years) - significant changes, trends and events Figure 3: SCOT/ATC PERFORMANCE REPORTING FLOW CHART & TIMING
SCOT (Jul-Dec)
Performance Report
strategic discussion to
inform NTC’s planning
process
ATC (Jul-Dec)
Performance Report
presented to Ministers
for noting/discussion
STRATEGIC
PLAN
consultation
ATC (Jan- June)
Final Strategic Plan,
Budget & Work
Program for approval
by Ministers
SCOT (Jan-June)
Strategic Plan,
Budget & Work
Program for final
consideration by SCOT
members
Transport Reform Outcomes Framework
December 2010
1
3. Planning for outcomes
3.1
Strategic Planning
As NTC is only one component of a broader multi-partner national strategy, the achievement of its
broad goals cannot always be directly attributed – a whole plethora of governments and
stakeholders also contribute. As a result, NTC’s strategic plan has a directional construct rather
than an operational one that articulates the organisation’s role in working towards a common set of
agreed goals.
Under this view, successful strategies result in the emergence of partnerships, with NTC’s strategic
directions aligned with its partners (eg: SCOT, Austroads, TCA, national regulators etc).
3.2
Work Programme
NTC’s work plan currently articulates how the programs are designed to achieve the high-level
goals articulated in the strategy. This includes budgets, resources and outputs. As the reforms are
often complex in nature and part of broader reform agendas, more work is needed to articulate the
intermediate outcomes, interdependencies, assumptions and risks (external factors).
3.3
Strategic planning in government
It is worth considering, what literature states about ‘good’ strategic planning in governments.
Strategic planning adopted approximately 20 years ago from the private sector as an innovation for
use within government sector (Eadie 19836; Berry 19947; Poister and Streib 20058). As discussed in
(Poister and Streib 20056):
“The conventional strategic planning process typically involves clarifying mission and values,
developing a vision of the future, analysing external challenges and opportunities, assessing
internal strengths and weaknesses, developing strategic goals and objectives, identifying strategic
issues, developing and evaluating alternative strategies, and developing action plans. Yet, a lively
debate continues regarding how to go about strategic planning in government in terms of scope,
content, involvement and participation, and approach.”
Literature of strategy tends to concur that while creation of strategies is relatively straightforward
(Poister and Streib 20056), the Achilles heel of strategy appears to be more concerned with how to
implement strategies:
“The more important issue, however, concerns putting plans into action. Strategic planning is an
action-oriented type of planning that is useful only if it is carefully linked to implementation and this
is often where the process breaks down.”
A draft report by Griffin NRM PTY LTD (2006)9 includes a table of what were thought to have been
elements of ‘advanced thinking’ with regard to best practice government strategies. The areas of
advanced thinking are well aligned to the Review of NTC 2009 recommendations and include:





consultation and partnerships
knowledge management
mechanisms for implementation
articulated as outcomes and targets
monitoring and evaluation
6
Eadie, D. C. (1983): "Putting a Powerful Tool to Practical Use: The Application of Strategic Planning in the Public Sector" Public
Administration Review 43 (5): 447-452.
7
Berry, F. S. (1994). "Innovation in Public Management: The Adoption of Strategic Planning." Public Administration Review 54(4): 322-330.
8
Poister, T. H. and G. Streib (2005). "Elements of Strategic Planning and Management in Municipal Government: Status after Two Decades."
Public Administration Review 65(1).
9
An assessment of the alignment of the 1996 National Strategy for the Conservation of Australia’s Biological Diversity and the National
Objectives and Targets for Biodiversity Conservation 2001-2005 with relevant nation, state/territory and industry biodiversity strategies,
NRM policy and codes of practice
Transport Reform Outcomes Framework
December 2010
2
3.4
Planning for outcomes
Program Logic is a powerful planning tool to consider how outcomes can best be achieved,
articulate a clear narrative for intervention, and later to help monitor, evaluate and report on
progress. It can be used at a strategy, program and project level.
Articulating logic often requires 'backward mapping' – that is we start with the intended
goals/outcomes and map downwards the preconditions to achieve each outcome.
Key steps in planning for outcomes linked to goals and objectives include:




situation context, trends and external influences (environmental scan)
future state analysis of transport outcomes (based on existing government strategies/plans)
NTC’s contribution to achieving those outcomes (based on its national regulatory and
operational reform role)
Developing a work plan, including program outcomes, intermediate KPIs, activities and
resources (using Program Logic methodology)
Logic has been used to support strategic planning by many organisations, including the Department
of Sustainability & Environment, Environment Protection Agency, AusAID, the natural resource
management sector etc.
Transport Reform Outcomes Framework
December 2010
3
4. Monitoring and reporting outcomes
4.1
Background
The Review of NTC 2009 observed that: “The overarching objective of the current IGA is to improve
transport productivity, efficiency, safety and environmental performance, and regulatory efficiency in
a uniform or nationally consistent manner. Therefore the NTC’s performance must ultimately be
assessed in terms of its impact against these indicators.”
According to a UK report, The Role of Indicators, Targets and Monitoring in Decision-Support for
Transport (Marsden and Snell 2009), mapping progress against objectives through outcome
indicators is central to decision support within transport. It concludes, “given the increasingly
fragmented delivery structures within which transport policy operates, this approach seems likely to
grow in importance over time.”
In response to CoAG endorsed recommendations from the Productivity Commission Report on
Government Services (2010), a review of performance reporting frameworks found a lack of, “an
explicit conceptual framework that follows the program logic … (that is, the conversion of inputs to
outputs and outcomes).” (Review of the Report on Government Services’ performance indicator
framework, PC, August 2010).
4.2
Hierarchy of objectives and outcomes
When planning a work program, a distinction is usually made between the broad shared goals that
the program will contribute towards (eg: NTC’s IGA objective of “safe systems”) and the specific
outcomes that the program, if successful, will make a significant contribution towards (eg: safe
driver behaviour).
In program logic the ‘program level’ of outcomes is referred to as the ‘purpose’ or intermediate
outcome.
Individual reforms (eg: heavy vehicle driver fatigue reform) will make a significant and unique
contribution to the relevant intermediate outcome.
Figure 4: Hierarchy of outcomes
Broader goals (IGA/ATC level) 
What success looks like at the broader society level? (goals NTC shares with governments). 
the unique contribution of NTC’s strategy and work program to the broader goals expressed as an outcome, (e.g. the impact of our strategy if we are successful). 
the unique contribution of NTC’s programs to the intermediate objectives expressed as an outcomes (eg: fewer tired truck drivers) Intermediate Outcomes (program level) Reform Outcomes (reform level) 4.3
Objectives and outcomes
Objectives are often confused with outcomes. ‘Objectives’ are the things that an organisation
promises to achieve (eg: NTC’s IGA goals); and is accountable for managing for these promises.
It is difficult to directly attribute the contribution of NTC reforms toward broad national goals (eg:
transport productivity) as the lead time is long-term and the specific contribution of one program or
initiative compared to another is complex. The Review of NTC 2009 concluded the accountability for
outcomes linked to IGA goals must be shared by governments and NTC.
4.4
Intermediate outcomes
Intermediate outcomes are the ways in which NTC’s program activities lead to longer-term impacts
against the broader goals. ‘Intermediate outcomes’ are effectively the stepping stones toward goals
and are generally medium term.
Transport Reform Outcomes Framework
December 2010
4
NTC has greater control over intermediate outcomes than longer term goals. Intermediate
outcomes are, therefore, key to NTC showing progress towards change at the goal level and
informing adaptive management. Articulating and monitoring intermediate outcomes therefore has
several purposes:



4.5
to demonstrate whether NTC’s reforms have (or have not) led to changes at the goal level.
provide NTC and governments with useful information for adaptive management
(incorporate learning into the reform planning cycles).
affirm and adapt the assumed theory of how activities lead to intermediate outcomes,
allowing program logic planning to become increasingly sophisticated.
Monitoring IGA goals and intermediate objectives
The Review of NTC 2009 noted that, “the next phase of transport reform will be challenging as the
focus shifts from improving individual modes to lifting the performance of the national transport
system as a whole. Reform development needs to holistically consider impacts on
productivity, safety, pricing, network access and land-use planning and investment.”
C-Marc10 coordinated advice on performance reporting of transport system outcomes at the
ATC/NTC IGA goal and program levels (informed by logic mapping – see example below). The
report proposes a hierarchy of KPIs for transport productivity, safety, environmental and regulatory
efficiency across all modes and users (see example, below).
This approach will support NTC’s logic model approach to work plan development, which links
goals, program objectives, projects and resources.
Table 2: safe system logic
Objective Measurement MOV Assumptions Accountability Goal: safe transport system KPI Data source NTC and all governments Impact: safer vehicles, drivers KPIs Data source National Road Safety Strategy implemented (incl. infrastructure) NTC reforms implemented All governments NTC Outputs: Milestones Work program status report Activities: Resources and activities Project plan KPIs 1) Project 1 2) Project 2 ATC approves reforms Resources Internal review available NTC NTC It is important to recognise that indicators are only a guide for informed discussion on national
reform priorities. Consistent with the NTC review findings, the proposed KPIs provide context for
governments to discuss strategic planning and transport reform priorities for NTC (and potentially
for other reform partners).
Based on C-Marc advice and logic-mapping an initial set of performance indicators has been
developed for the Strategic Plan. The indicators draw from existing publicly reported information
and will develop over-time as better data becomes available.
10
Performance Reporting of Transport Reform Outcomes: Measuring the value of what can be managed (C-Marc 2011) – see NTC website
under publications
Transport Reform Outcomes Framework
December 2010
5
4.1
Monitoring IGA goals and intermediate objectives
NTC is required by its IGA to monitor reform implementation. Functions of the Commission will be
to:
5.1: (f) monitor implementation of Agreed Reforms by the Parties and regularly report to the
Council; (g) maintain and review Agreed Reforms
13.1: “the Commission … is to establish a process for regular monitoring and reporting on the
Parties’ (governments) progress with implementation of Agreed Reforms.”
Specific, measurable, and time-bound (SMART) targets should express what the success of each
reform looks like. Evidence is then collected to determine whether you are on track to achieve these
targets, which is an expression of performance expectations.
NTC has implemented a maintenance and evaluation framework endorsed by governments;
however reform outcomes are not systematically measured and reported against agreed KPIs.
Research has been commissioned from AECOM11 to identify:

outcomes and indicators in previous NTC reforms (or recommend proxies)

a schedule for monitoring priority reforms

how reform monitoring can inform or trigger evaluation reviews
Key recommendations from the AECOM report include:

NTC RIS to identify reform outcomes, verifiable KPIs, a monitoring plan and mandatory
review point;

Baseline data should be collected before reform implementation (eg: National Heavy
Vehicle Law) for longitudinal comparison;

Reform outcomes monitored by timely tracking of KPIs (“canary in the coalmine”)

Evaluation review is triggered early if implemented reforms fail to deliver reform outcomes
AECOM also reviewed previous regulatory impact statements and retrospectively recommended
suitable KPIs for future reform monitoring. The heavy vehicle driver fatigue, compliance and
enforcement and mass limits reform were identified as priorities for reform monitoring and
evaluation.
4.1
Data availability
NTC is primarily a user of research, not a research provider. It is expected that NTC will draw on
data already reported by existing agencies (such as BITRE, Austroads, ARRB).
The Review of NTC 2009 observed the lack of consistent data “impedes the ability to optimise
performance across modes and throughout the entire system”. The review recommended a change
to NTC’s Act and IGA requiring governments to provide information (eg: similar to the National
Water Initiative).
ATC agreed that a formal amendment to the IGA did not appear necessary. Instead, NTC will work
with governments and research bodies to address any identified data gaps.
11
Reform Monitoring and Evaluation Framework (AECOM 2010) – see NTC website under publications
Transport Reform Outcomes Framework
December 2010
6
5. Evaluation of Outcomes
5.1
Background
In 2001, Transport Agencies Chief Executives (TACE) endorsed a formal process to evaluate the
implementation of existing national transport regulatory reforms.
Implementation reviews complement reform maintenance and identify where ‘on-the-ground’
reforms need enhancement or ‘fine-tuning’. It also enables agencies to consider their own positions
relative to reform variations.
The NTC IGA provision (13.1) for this work is detailed as follows:
“the Commission, in consultation with representatives of the Parties, is to establish a process for
regular monitoring and reporting on the Parties' progress with implementation of Agreed Reforms.
This process is to be approved by the Council and will include provision for the Commission to
report to the Council on the details of a Party's divergence from an Agreed Reform.”
The IGA also allows NTC to ask relevant Ministers for a response to variations (13.2) which can be
referred to ATC “to determine whether further action should be taken”.
In NTC’s own submission to the Review of NTC 2009, it was acknowledged that the current review
process is often completed by a legal consultant and focuses on an analysis of “black letter law”.
The planned establishment of national regulators administering national heavy vehicle law and rail
safety regulation will help to switch the focus of evaluation from “black letter law” to effectiveness,
implementation and impact.
5.2
Effectiveness Evaluation
The TACE endorsed framework also includes provision for 5 and 10 year reform evaluations or
“effectiveness reviews”. The evaluations are identified, prioritised and resourced by the strategic
planning process (and based on the views of governments, industry, NTC officers etc).
Some reforms may require only a straightforward review (e.g. possibly single issue technical
problems). Other reforms may require a more extensive review, usually because of their larger size
or more complex nature.
Stakeholder feedback on earlier evaluation reviews have suggested that NTC has often taken a
narrow approach to effectiveness reviews.
Summary:
Current focus is:

“black letter law” implementation review (within 18 months)

effectiveness review if policy issue or 5 and 10 yrs

no monitoring of KPIs to objectively guide or prioritise evaluation.
Transport Reform Outcomes Framework
December 2010
7
5.3
Evaluating the implementation and effectiveness of reform
A reform built around program logic to deliver outcomes provides a strong base for future
implementation and effectiveness evaluations because it:

clearly articulates the goals (what success looks like)

identifies measurable outcomes (indicators of success)

articulates the cause-and-effect relationships to ensure program activities deliver an impact
or outcome.
For existing reforms, the program logic will need to be developed retrospectively at the planning
meeting.
NTC has already begun to plan effectiveness reviews for Australian Road Rules and Australian
Vehicle Standards Rules by applying program logic methodology.
A similar methodology – Collaborative Performance Story Reporting (CPSR) – could be applied to
reform implementation evaluations (eg: ports strategy). Performance story evaluation is a relatively
new “collaborative” approach to reporting (see Appendix a).
CPSR moves away from the traditional ‘report card’ approach and can be used as a ‘learning tool’
for NTC and governments to act on findings and improve approaches to reform implementation. In
this way, the advice to ATC would outline recommendations for positive change.
CPSR is supported by quantitative and qualitative evidence and describes the causal links that
show the reform’s progress toward an outcome. It also provides an insight into what’s working and
what’s not and why.
In 2007–08, as part of the Building Better Data Project, the Australian Government sponsored a
performance story evaluation training program followed by 13 pilot studies in the Natural Resource
Management sector around Australia based on work by Dart and Mayne (2005).
It is intended to design implementation planning for the PBS reform using collaborative (peoplecentred) program logic to assist a future CPSR evaluation of the reform’s implementation.
5.4
Implementation status reporting
Currently NTC compiles a detailed implementation status report for ATC, which also incorporates
reporting for CoAG reforms.
In the longer-term, the implementation of a uniform national heavy vehicle law and rail safety law
will address many of the variations in the report.
It is recommended that the report is reviewed and updated to provide a more strategic overview
(NTC perspective) of the progress, barriers and key risks to delivering outcomes.
Transport Reform Outcomes Framework
December 2010
8
6. Work program (outputs) reporting
6.1
Project status report
In its response to the review, ATC asked NTC to provide greater visibility of the “specific products
and reports on which the NTC will consult publicly or seek ministerial approval” for the year ahead
(see report below). A similar format to CoAG project board reporting has been adopted.
Key features of the progress report include:




progress against key deliverables
strategic risks and interdependencies
future key meetings and deliverables
emerging new initiatives/proposals
Transport Reform Outcomes Framework
December 2010
9
7. Appendix A
7.1
Collaborative Performance Story Reporting - overview
Key features of Collaborative Performance Story Reporting adapted from the MERI framework for
Natural Resource Management include:
Scoping—inception/planning meetings are held to determine what will be evaluated, develop the
program logic (if not already existing), set evaluation questions and identify existing evidence and
people to be interviewed.
Evidence gathering—an evidence trawl is conducted to identify existing data that will provide best
evidence for expected outcomes. This is followed by the social inquiry process, where interviews
are conducted with people who can provide additional information about program outcomes.
Specific questions are asked and recorded to provide stories
of significant changes that have occurred as a result of the program.
Integrated data analysis—quantitative and qualitative data is analysed to identify evidence
corresponding to the outcomes in the program logic and integrated within the results chart.
Expert panel—people with expertise in relevant fields and scientific disciplines assess the evidence
of outcomes that has been gathered. They also judge and make statements about the extent to
which the evidence is adequate to assess the progress the program is making towards its stated
outcomes. The panel may also identify further evidence that may be needed to make a conclusive
statement about the achievement of program outcomes.
Following the panel meeting, the evaluator integrates all of the analysed evidence and assesses the
amount and quality of evidence available for each outcome in the program logic to inform a draft set
of recommendations.
Summit meeting—evaluation participants come together to consider and discuss the findings,
nominate the stories that best illustrate the impact of the program and make recommendations for
the program in future.
Integration, report and communications—the evaluator prepares the performance story report,
which is a synthesis of all the above steps including recommendations from summit meeting
participants. A plan is established to communicate the findings of the evaluation.
Revising the program logic—program managers, staff and other stakeholders meet to consider
the report and revise the program logic as needed to plan for the next phase of the program. The
next phase can incorporate the lessons and recommendations from the previous phase.
Transport Reform Outcomes Framework
December 2010
10
Transport Reform Outcomes Framework
December 2010
11