OFFICE OF DEVELOPMENT EFFECTIVENESS
JULY 2017
Office of Development Effectiveness Work Plan
1.
INTRODUCTION
The Office of Development Effectiveness (ODE) undertakes independent evaluations of the Australian aid
program. This document is ODE’s work plan for calendar year 2017. In addition to the discrete items set out in
the work plan, ODE will undertake a range of work to support DFAT’s internal performance assessment and
evaluation processes, to support and report on the conduct of program prioritised evaluations, and to build
evaluation capacity within the Australian aid program.
The 2017 work plan has been reviewed and endorsed by the Independent Evaluation Committee (IEC), an
external advisory body that oversees all of ODE’s work. The role of the IEC is to strengthen the quality,
credibility, influence and independence of ODE’s work through the provision of independent strategic and
technical advice.
This work plan includes all work scheduled to be completed in 2017, evaluations which are expected to
commence in 2017 but be completed in 2018, and a reserve list of evaluation topics which may be progressed
in 2017 or future years.
The following principles guide the selection of evaluation topics:
» Policy relevance: assessing progress on implementation of the Australian Government’s policy priorities.
» Potential learning benefits: focusing on areas where there is a strong demand for insight, information or
guidance.
» Coverage and materiality: over time ODE’s evaluations should provide appropriate coverage of
expenditure by sector, theme, country and region.
» Risks to aid effectiveness: examination of issues which pose a challenge to the Australian aid program
and its reputation.
ODE’s involvement in program evaluations is guided by the above principles in addition to the following
considerations:
» Readiness of program areas to collaborate with ODE.
» Clarity of evaluation purpose; be it accountability, learning or both.
» Feasibility of the evaluation timing and the availability of human and financial resources.
» Opportunity to model and promote ‘good evaluation practice’ within DFAT.
ODE expects to publish five strategic evaluations in 2017. The number of program evaluations ODE undertakes
will depend on residual staff capacity. ODE will also undertake six pieces of discrete performance and quality
analysis. Details of the work in each of these areas is elaborated below.
2.
OVERVIEW OF ODE WORK PLAN
The work plan is divided into four key areas: strategic evaluations, program evaluations, performance and
quality analysis, and evaluation policy implementation, advisory support and outreach.
2.1 Strategic evaluations
Strategic evaluations are broad assessments of Australian aid that focus on key policy directions, specific
development themes and sectors, or large programs. These evaluations are independently initiated, managed
and conducted by ODE. They typically examine a number of investments, often across multiple countries,
regions or sectors. By virtue of their scope, strategic evaluations tend to be time and resource intensive.
Strategic evaluations provide independent oversight of Australian aid. They inform change in both strategy
setting and operational practice and they assist in identifying new and innovative ways to deliver Australian
aid.
Table 1: Strategic evaluations to complete (publish) in 2017
Topic
Humanitarian assistance in the Pacific: The effectiveness of Australia’s response to Cyclone Pam
Combatting pandemics & emerging infectious diseases
Electoral systems strengthening
Disability-inclusive development
Climate change assistance
Table 2: Strategic evaluations to complete (publish) in 2018
Topic
Economic partnerships in Asia
Ending violence against women and girls
Facilities for effective aid delivery
Health support to the Pacific
Evaluation of Pacific infrastructure assistance
Table 3: Strategic evaluation reserve topics
Topic
Regional programming – relevance and effectiveness
Technical and vocational education and training
Country program evaluation (with thematic focus)
Gender mainstreaming in capacity building programs
The role of the private sector in Australia’s aid program
2.2 Program evaluations
Program evaluations are independent assessments of the performance of Australian aid investments. They
provide evidence to inform program management decisions, contribute to broader learning and help to
enhance the accountability of Australian aid. Program evaluations are generally commissioned and managed
by program areas and may be conducted at whichever point in time is of most value from a management
perspective.
Under the new DFAT evaluation policy, country, regional and thematic aid programs advise ODE of evaluations
they will finalise each year. These are compiled and, together with ODE strategic evaluations, comprise the
DFAT annual aid evaluation plan. There are minimum numbers of evaluations required for each program area,
proportionate to their size.
Consistent with its role in providing expanded support for program evaluations, and the selection principles
outlined above, ODE sometimes funds and leads, or otherwise participates in, a limited number of program
evaluations. ODE has offered to assist PNG Program with one evaluation in both 2017 and 2018.
Table 4: Program evaluations to complete (publish) in 2017
Topic
Economic infrastructure: Eastern Indonesia National Roads Improvement Program
Economic infrastructure: Papua New Guinea Transport Sector Support Program
Cambodia Agricultural Value Chain Program
2.3 Performance and quality analysis
ODE’s performance and quality analysis (PQA) work monitors DFAT’s aid performance management and
reporting systems on an annual basis, and independently quality assures the assessments they produce. These
are undertaken mostly within set timeframes, while other activities are ‘one-offs’ or undertaken with a
timeframe controlled by ODE. There are a number of components to this work, as indicated below.
Table 5: Work to complete (and publish where appropriate) in 2017 and 2018
Topic
Spot check of Aid Quality Checks
Validation of Final Aid Quality Checks
Annual Review of Aid Program Performance Reports
Annual Quality Assurance of Performance of Australian Aid report
Review of Uptake of ODE Recommendations
Review of DFAT’s approach to investment monitoring (2017 only)
2.4 Evaluation policy implementation, advisory support and outreach
ODE provides support for program evaluations in four main ways: monitoring evaluation activity within DFAT;
supporting programs to conduct better evaluations; direct engagement in individual evaluations; and
providing M&E training and other activities to help build DFAT’s evaluation capacity.
A new evaluation policy for the Department was approved and circulated in early November 2016. The new
Evaluation Policy seeks to increase the impact, transparency, and relevance of our evaluations. It introduces a
demand driven approach where aid programs determine what they will evaluate based on need and use after
consulting their Division Head. Programs are required to complete a minimum number of evaluations each
year, and they will be required to have a management response and be published within three months of their
completion.
Under the new evaluation policy ODE will monitor planned and completed evaluations across the Department.
ODE will compile an annual aid evaluation plan, which will include program evaluations and ODE’s strategic
evaluations, for consideration by the Independent Evaluation Committee and for the Secretary’s approval.
ODE will report to the Secretary on the implementation of the plan twice a year.
Table 6: Evaluation policy work from 2017 onwards
Topic
Progress report on 2017 Evaluation Plan
2018 Evaluation Plan and Final report on 2017 Evaluation Plan
ODE will continue to provide other support to DFAT staff managing evaluations in the following key areas:
» Provision of advisory support to program areas. For the most part, this involves reviewing evaluation
terms of reference or reports, and advice on completing management responses and publishing.
» Deliver training on DFAT monitoring and evaluation.
Further, ODE plans to undertake outreach to the evaluation community to strengthen understanding of the
policy and to address issues regarding the quality of evaluations (e.g. quality of recommendations).
Details of all planned work topics are at Attachment 1.
Attachment 1 - Description of all planned work topics
The following tables provide further detail about ODE evaluations and performance assessment work in 2017
and 2018.
Table 7: Topics to complete (publish) in 2017
Topic
Description
Humanitarian
Assistance in the
Pacific: The
Effectiveness of
Australia’s Response to
Cyclone Pam
The purpose of this evaluation is to learn from the experience of the Cyclone
Pam response to inform future humanitarian responses to rapid onset crises
in the Pacific. The evaluation will cover the funding and partnerships that
formed the initial response and early recovery phases of the Australian
assistance to Vanuatu following the cyclone.
(strategic)
Combatting Pandemics
& Emerging Infectious
Diseases
(strategic)
Electoral systems
strengthening
(strategic)
Disability-inclusive
development
(strategic)
Climate Change
Assistance
(strategic)
Economic
infrastructure: Eastern
Indonesia National
Roads Improvement
Program
(program)
The evaluation will contribute to the evidence base on strengthening health
systems to prevent, detect and respond to emerging infectious disease
threats with pandemic potential in Asia and the Pacific, and inform decision
making about future DFAT investments and policy engagement on regional
health security.
The evaluation will investigate the contribution of Australian electoral
assistance to Australia’s foreign and development policy objectives in
selected countries, the efficiency and cost-effectiveness of Australian
assistance and the contribution to gender equality and social inclusion.
This evaluation aims to assess how well the ‘Development for All 2015-2020’
strategy is being implemented across the aid program. It will identify ways to
strengthen disability inclusion and to inform implementation of the strategy.
The evaluation will offer specific operational lessons on successfully
incorporating climate change into design and implementation by assessing
the performance of a number of past and current climate change investments
with a focus on effectiveness, sustainability and value for money.
This evaluation will document the achievements and lessons learnt from the
Eastern Indonesia National Roads Improvement Project, a 10 year, $330m
loan financing investment which supported twenty road and bridge
construction packages. Lessons will inform approaches to other infrastructure
investments in Indonesia and in the broader aid program, and inform current
thinking about the potential for expanded use of loan financing.
Economic
infrastructure: Papua
New Guinea Transport
Sector Support
Program
(program)
Cambodia Agricultural
Value Chain Program
(program)
Spot check of Aid
Quality Checks
(performance and
quality analysis)
Annual Review of Aid
Program Performance
Reports
(performance and
quality analysis)
Validation of Final Aid
Quality Checks
The PNG Transport Sector Support Program (TSSP) is envisaged as a 15 to 20year commitment to improve transport infrastructure in PNG. The second,
five-year, $400 million phase of the program started in late 2014. The purpose
of the evaluation is to inform the approach taken by TSSP over the remainder
of its current phase and into the next, and to inform thinking about how it can
best support other Pacific Island countries with maintaining and improving
their road networks.
This evaluation will examine the Cambodia Agricultural Value Chain (CAVAC)
program Phase 1 (concluded December 2015, $46m) and CAVAC Phase 2
(ongoing, $98m). The evaluation will assess the effectiveness and
sustainability of phase 1 irrigation and civil works and look at whether
interventions with private sector actors are providing intended benefits to
smallholder farmers.
ODE quality assures DFAT’s self-assessments of the performance of individual
aid investments, which are made through annual Aid Quality Checks (AQCs).
This quality assurance work involves checking a statistically significant sample
of AQC reports each year, to assess whether the quality ratings within them
are robust.
ODE assesses the quality of annual APPRs which report on the performance
of country and regional aid programs, highlighting areas of strength and
opportunities for improvement. APPR quality reviews have been carried out
every year since APPRs were introduced in 2008 and have been published on
the ODE website since 2013.
In 2017, ODE will for the first time validate the ratings contained in all Final
Aid Quality Checks.
(performance and
quality analysis)
Annual Quality
Assurance of
Performance of
Australian Aid report
ODE independently analyses the performance assessments made in the
Performance of Australian Aid report to ensure that they are robust.
(performance and
quality analysis)
Review of Uptake of
ODE Recommendations
(performance and
quality analysis)
Review of DFAT’s
approach to investment
monitoring
(performance and
quality analysis)
The management response to ODE’s strategic evaluations are cleared at a
high level and become a public commitment to implementation of accepted
recommendations. ODE reviews the implementation of actions to achieve
recommendations around twelve to eighteen months after each evaluation
and reports on progress to the IEC and the DFAT Executive.
ODE intends to make a detailed assessment of investment monitoring and
evaluation which has been consistently identified as a weakness in DFAT aid
investments (through Aid Quality Checks since 2008). Factors enabling and/or
inhibiting high quality monitoring of investments will be identified.
Table 8: Topics to complete (publish) in 2018
Topic
Description
Economic partnerships
in Asia
A number of programs in South-East Asia are transitioning to economic
partnerships involving Australia’s diplomatic, trade and aid resources. The
evaluation will contribute to learning about how we can engage effectively in
partner government policy and budget cycles and about the effectiveness of
other tools which can be used in policy reform.
(strategic)
Ending Violence
Against Women and
Girls
(strategic)
Facilities for Effective
Aid Delivery
(strategic)
Health Support to the
Pacific
(strategic)
Evaluation of Pacific
Infrastructure
Assistance
(strategic / program)
Addressing Violence Against Women and Girls (VAWG) is a priority for the
Australian Government, both domestically and in our foreign policy and aid
program. This evaluation will examine Australia’s support to prevent and
respond to VAWG in developing countries—particularly in the Pacific where
gender-based violence rates are among the world’s highest.
In DFAT the prevalence of facility aid investments - a flexible project
implemented through a contract where objectives are specified, but the
pathway to delivering on those objectives is left unspecified – has
substantially increased in recent years. This evaluation will document
experience to date and inform current management of facilities and well as
future designs to enable greater aid effectiveness and strengthen risk
management.
ODE will examine the quality of DFAT’s health programs, whether and how
they align with global good practice and whether they are ‘fit’ for the unique
context of the Pacific. The findings will inform bilateral and regional
programming in future, improve cross-sectoral collaboration, and guide
support provided by Canberra.
This evaluation will complement other ODE infrastructure evaluations
completed in 2017 (EINRIP and PNG TSSP). It will examine DFAT’s use of
multilateral partners in delivering infrastructure and supporting reform and
capacity development in the Pacific. The evaluation is likely to involve analysis
of one, or possibly two, major regionally focussed infrastructure investments
delivered through the ADB and/or the World Bank.
A short synthesis publication and seminar will draw together findings from
this and related evaluations.
Spot check of Aid
Quality Checks
Same as for 2017
(performance and
quality analysis)
Validation of Final Aid
Quality Checks
(performance and
quality analysis)
Same as for 2017
Annual Review of Aid
Program Performance
Reports
(performance and
quality analysis)
Annual Quality
Assurance of
Performance of
Australian Aid report
Given that ODE assessment of the quality of annual APPRs have returned
relatively stable findings for a number of years, ODE will consider taking a
different approach in 2017, focussing on a more detailed and in depth
assessment of a small number of APPRs.
Same as for 2017
(performance and
quality analysis)
Review of Uptake of
ODE Recommendations
(performance and
quality analysis)
Same as for 2017