Managing for Results:
A primer on monitoring public diplomacy
grant projects
Eric Abdullateef
Performance Advisor
USAID/DCHA/ASHA
Annual Partners Workshop
March 30, 2016
10:00 am to 10:50 am
Crystal Gateway Marriott
This workshop informs ASHA’s approach to performance management. It focuses on
the necessity of monitoring and evaluating project performance and the importance
of measuring project impact. It offers reasons why and when these processes
should be planned and implemented, what’s involved, and who should be engaged
in the process and how the data should be presented.
Overview
What is public diplomacy (PD)? Each organization will have its own nuanced
definition of PD. Working synonyms for PD are advocacy or influence. Advocacy,
and by association PD, is the strategic use of information to influence the policies,
actions, opinions, attitudes and behavior of those in positions of power or authority to
achieve positive changes in people’s lives. Much of diplomacy is relationship building,
getting others to do what you want. ASHA funds infrastructure, fixtures and
equipment inputs ultimately to gain overseas influence. The results of these inputs
are not easily quantified.
Communication is the main currency of PD. Assessing the effectiveness of PD relies
heavily on measuring perceptions and potential attitude and behavior changes
among target stakeholders. Effective and strategic communications means that the
information is provided in the right format at the right time to the right audience with
the right impact. Efficient communications means providing only the information
needed. The catch phrase “American ideas and practices” basically boils down to
information that is communicated to receptive overseas audiences.
Approaches to PD
• Heightened awareness
• Changed policies
• Positive changes to people’s lives
• Contribution to debate
• Policy changes implemented
Defining Success for an ASHA-funded Project
A goal is the overall purpose, end state or reason for the project. ASHA projects
contribute to strengthening the overall project/program/ institution (hereafter project).
Note that activities are components of projects, which are components of programs,
portfolios and so on. It is important to specify exactly what is being monitored. Is it
the activity or the program? For example an ASHA funded activity might make a
“contribution” to a higher level program goal, which is composed of input from other
non-USAID sources. Also keep in mind that projects might be planned over multiple
phases or sub-projects that take years. As such, accomplishing the stated goal in
this scenario would not be directly attributable to ASHA support or be realized
immediately.
The USAID/ASHA program provides resources for construction and commodities
that support an organization or institution that promotes the best in American ideas,
innovations and practices in education and medicine. Also keep in mind that ASHA
assistance does not come with the guarantee of follow-on funding. Each ASHA
award aims to build partnerships and foster greater understanding between the
1
American people and people overseas. In many cases these partnerships are
decades old, but each award is tracked in isolation over the grant life cycle.
An objective is a specific change a project seeks to achieve with the clearly defined
inputs. Objectives may be short or long term. Use your project’s objectives to define
indicators. Example: By December 2020, we aim to have made a 40% increase in
the number of registered student groups using the ASHA funded Student Center
building. A strong ASHA application will feature objectives that are S.M.A.R.T. which
stands for specific, measurable, achievable/attainable, realistic and time bound.
Performance and process indicators
A project is successful when it meets its goal as measured against its performance
indicators. Key performance indicators can be financial, customer, people, product or
process focused metrics. They can track inputs, process, outputs, outcomes or
impacts.
Example: Renovate an engineering training facility to expand access for persons
with disabilities.
Goal
Expand professional
opportunities for
persons (faculty, staff
and students) with
disabilities (PWDs) to
gain expertise and
leadership skills in
engineering.
Performance Indicator
An increase in the
professional engineering
achievements of student,
faculty and staff PWDs,
based on knowledge and
skills acquired in the XYZ
Program.
Target
By the end of 2020, 75% of
the disabled beneficiaries will
rate their experience in the
XYZ Program as highly
favorable.
25% of the XYZ Program
disable beneficiaries will have
demonstrated advancement in
their careers within five years
after returning from the
program.
ASHA’s Theory of Change
A Theory of Change is a specific and measurable description of a social change
initiative that forms the basis for strategic planning, on-going decision-making and
evaluation. The theory behind the ASHA program can be summarized as follows.
Providing needed resources to sustainable overseas organizations and institutions
that model the best in American education and medicine will establish and foster a
positive picture of the United States, improve goodwill and increase mutual
understanding between people. ASHA is a public diplomacy program that uses
health and education development tools to support overseas schools, libraries and
medical centers that have a strong link to American ideas, innovations and practices.
The tricky part entails evidencing whether a project does or does not work.
PD is multidimensional and topical (multidisciplinary). The work of ASHA grantees is
cuts across a host of outcomes including:
2
• Human resource outcomes
• Civil society organizational outcomes
• Democratic outcomes
• Communications outcomes
• Infrastructure outcomes
• Environmental outcomes
An example of a high-level PD objective follows: Strengthen the capacity of civil
society organizations so they are able to carry out their mission independent of the
host government, mobilize/leverage funding or improve their techniques and tools.
A central element of ASHA’s PD strategy is civil society strengthening, which
includes building the capacity and voice of key stakeholders – OSI and USO
partners, grass roots movements etc. – so that they are able to campaign for and
represent the interests of both their stakeholders and Americans more effectively.
Promoting American Ideas and Practices
Traditionally PD is carried out through communications activities such as advocacy,
campaigning, awareness raising, direct actions and working with the media. ASHA
argues that infrastructure, fixtures and equipment, as tangible symbols of American
generosity, also have the ability to send powerful signals or help an institution attract
attention. A grantee’s outreach activities and its Branding Strategy and Marking Plan
(BSMP) further amplify these PD activities and initiatives.
Geographic Target Area
PD initiatives can be targeted at several geographic levels:
• International/Worldwide
• Regional
• National
• Local
As a worldwide program, ASHA grantees work across an array of geographic
audiences.
Issues of Monitoring and Evaluating PD Efforts
Monitoring
Monitoring a project involves the collection of routine data that measure progress
toward achieving project objectives. Data are used to track changes in project
performance over time. Monitoring is sometimes referred to as process evaluation
because it focuses on the implementation process and informs key process
questions such as:
• How well has the program been implemented?
• How much does implementation vary from site to site?
• To what extent did the program benefit the intended people? At what cost?
Monitoring involves assessing whether results are being achieved as planned, for
both decision-making and accountability purposes.
3
Evaluation
Evaluation measures how well the project activities have met expected objectives
and the extent to which changes in outcomes can be attributed to the project
intervention. Note that evidencing contribution is a much lower bar to reach than
evidencing attribution or causality. Given the difficulty of proving attribution (isolation
of impact), ASHA operates under the broad assumption that its support contributes
to –rather than causes –any particular PD result.
Example: Say a medical center was providing only modest neo natal care training
and services before ASHA support and is now teaching three more courses post
award and providing 20% more contact hours of neo natal patient services. Then
there is a good chance of contribution from ASHA support –but not necessarily
attribution.
Although often mentioned in the same breath, monitoring and evaluation are, in fact,
two distinct activities, related but not identical. What they have in common is that
they seek to learn from what you are doing and how you are doing it. ASHA does not
require evaluation plans and evaluations are not the main focus of this presentation.
What follows are particular challenges and opportunities that the monitoring and
evaluation of PD work presents.
Challenges
•
•
•
•
The chain of cause and effect can be long and complex.
Partnerships (like ASHA/USO/OSI networks) can make it difficult for any
individual organization to say exactly what its influence has been.
The objectives and focus of PD projects are non-linear, fluid and
incremental over time, which makes it hard to measure progress against
initial objectives.
Assessing the value of PD is art more than science.
Opportunities
•
•
•
Approaches that allow for flexible scope, objectives and tactics.
Reliance on simple common-sense tools and techniques such as the use
of qualitative narratives and success stories.
Recognition that organizations need to be subjective in their assessment
of change, and the factors that drive PD results.
ASHA M&E Requirements
Grantees are expected to monitor, control and report on project implementation
(activities and processes) and performance (outcomes) over their grants life cycle
(versus over the project or program life cycle, although the latter is best practice).
ASHA’s truncated reporting horizon is mainly due to the fact that an award recipient
is not under any legal obligation to report once an award ends. Similarly, recipients
are not required to plan or commission evaluations, but again, evaluation is a best
4
practice that is highly encouraged at USAID. If commissioning an evaluation is part
of your project design, make ASHA aware of it. Formative, mid-term or summative
evaluations are at the awardee’s discretion and expense. They can focus on process,
performance or impact, for example. Evaluations can be led internally or externally
(independent). Note that evaluations take time to arrange and are costly.
Why, When and How M&E Should Be Planned and Implemented
Why –The most important deliverable and implementation tool you are required to
develop during the ASHA grant start-up phase is your Activity M&E Plan. It enables
collaborating, learning, improving and adapting. It is used to inform what a project
has done. Time built in for intelligent reflection on progress or failure is invaluable.
Learning should be more important than whether the project worked or failed to work.
Performance reporting helps signal whether a funded project was money well spent.
When - An M&E plan needs to be in place at the beginning of any new project. This
will include the identification of clear and staged objectives and indicators that will
enable staff and other stakeholders to track progress towards achieving these
objectives. Each PD project should be monitored on a continuous basis over its
lifetime. ASHA’s AMEP tool requires an initial submission (within 90-days of an
award), annual updates and a final AMEP update once the award closes.
How – Defining clear goals and outcomes and logical steps to achieving them are a
pre-requisite at the outset of any PD project. Keep in mind that over time and in
responding to developments in the project, some objectives will need to be adapted.
M&E planning is iterative rather than static and elaborated based on new information.
1. Assess to what extent the project is successful – and to adapt it
accordingly – using key performance indicators and baselines that are
relevant to ASHA and your institution.
a. Isolate impacts that related to ASHA assistance to the extent possible.
b. The baseline reflects the number associated with an indicator at the
time the grant is awarded, which may be zero (0).
c. Change (increase or decrease) is the difference between the baseline
and actual results.
d. Set targets that reflect how the baseline situation is expected to
change over time and state key assumptions.
i. One year later, the grantee will record the progress made
toward reaching the target set. Comparing actual performance
against targets to make judgments about the value of the project.
2. Respond to unpredictable events by progressively elaborating your plan.
3. Provide regular opportunities to communicate and collaborate and share
with stakeholders.
4. Document the process in order to be able to learn from experience to
improve future work.
5. Demonstrate results to donors, supporters, policy-makers, and others.
Define Timeframe for Your Project
From both the perspective of writing a grant proposal and reporting to stakeholders,
5
the project life cycle is the ideal timeframe to choose. The typical life cycle of an
ASHA grant (which is 3 or 4 years) is typically shorter than the life cycle for a project
involving infrastructure, fixtures and equipment, which may mean that initially there
would be little or no actual results realized. While ASHA agreements only stipulate
performance reporting until closure of an award, a project manager should ideally
plan for results over a timeframe that fits the unique information needs of its
management and stakeholders.
Milestones versus Key Performance Indicators
The development of a few specific milestones and key performance indicators in the
early stages of the project makes the tracking of progress towards the achievement
of objectives and the overall goal much easier. Milestones 1 are significant points or
events in the project, such as the completion of a major deliverable. Remember that
milestones have no duration; they simply represent the completion of activities. A
milestone helps to make sure nothing falls behind or is forgotten. Performance
indicators are clues, signs or markers that measure one aspect of a program and
show how close a program is to its desired path and outcomes. They are used to
provide benchmarks for demonstrating the achievements of a program. They provide
objective evidence that an intended change is occurring. Performance indicators are
at the heart of developing an effective performance management system.
Figure 1: A graphic illustration of program monitoring over time might look like this.
Many grantees confuse milestones and performance indicators. Milestones show up
frequently in implementation plans and schedules rather than M&E Plans. Note that
a complete set of milestones can easily be converted into a performance indicator
such as “percent of milestones completed,” for example.
1
For example, a completed design, certain deliverable due dates from the customer, or a
company-required checkpoint could be milestones. The project manager can also impose additional
milestones as checkpoints to help control the project. If a checkpoint in the schedule is reached and
any of the planned work has not been completed, it indicates the project is not progressing as
planned.
6
Activity Monitoring and Evaluation Plan (AMEP) Tool
ASHA’s AMEP tool is a systematic and objective approach for monitoring project
performance toward its objectives over time. ASHA developed this tool to simplify
reporting for grantees. Grantees provided important input and feedback to our
design. The tool is a standardized Microsoft Excel template that identifies a set of
four (4) standard indicators. Standard Indicators are not required, but are strongly
recommended. Award recipients receive training and material on AMEP reporting
requirements in the initial phase of their award.
Standard Indicators
ASHA’s Standard Indicators are discussed at length in ASHA’s Annual Program
Statement and in its Strategic Plan, which are available online. They measure
project progress and impact as they align with several USAID strategic objectives.
They are at the output level, and can usually be reported in the short or intermediate
term of a grant’s life cycle and with data that most awardees have at their disposal.
ASHA steered away from requesting original data collection where possible to
minimize cost and maximize usefulness. Standard Indicators help to harmonize
overall reporting and enable ASHA to quantify and consolidate its program results.
Custom Indicators
The AMEP also requires a minimum of two (2) Custom Indicators that are proposed
by and unique to the award recipient. Tips on Custom Indicator data collection
follows below.
Frequently Asked Questions:
•
•
•
What if I cannot be sure my institution can measure the standard
indicator? Answer: Pick another one. Or collect original data.
How do I select custom indicators? Answer: Based on usefulness and
importance for your institution.
What does ASHA do with performance report data? Answer: Manage its
portfolio by comparing baselines and targets to actuals; Report program
performance to USAID and Congress.
Measurement Tools and Techniques
A strong AMEP will provide a sense of the number of gender separated beneficiaries
receiving project support as well as what changes this support brings about in their
lives. Separate people-level project data based on other characteristics (i.e. age,
ethnicity or income) based on your unique needs for information. Choose data
collection methods that will answer key questions about the project, inform future
decisions and help tell the story of your project. Remember: An impact story trumps
a story about inputs or outputs any day!
7
Performance monitoring can be based on a broad range of qualitative, quantitative
and mixed methods. Data collection methods should match the indicators developed
for the project, be appropriate to the context, and be easy to use.
•
•
•
•
Surveys
Meetings
Activity recording
Logs
•
•
•
•
Media tracking
Policy tracking
Timelines
Quotes log
• Interviews
• Polling
• Focus group
discussion
Engage Stakeholders in the M&E Planning Process
PD activities are likely to involve members of staff, students, alumni, partner
organizations and members of the community all working towards change –such as
enlarging democratic space; improving service delivery and so on. ASHA
encourages grantees to engage a wide variety of stakeholders in the strategic
selection of all key performance indicators and setting of targets.
Final Point
Effective public diplomacy activities can prove to be a very powerful and empowering
strategy -- benefiting both overseas partners and the American people. By this same
token, ineffective PD strategies can be hugely wasteful of time and resources and
leave stakeholders feeling bewildered, disempowered or uninterested. The onus is
on each of us, and our organizations, to ensure that we make these interventions as
successful as possible.
Closing Questions
• How can ASHA make performance reporting better, faster, smarter?
For More Info
• USAIDLEARNINGLAB.ORG
Contact Information
Eric Abdullateef
Performance Advisor
Macfadden & Associates
USAID | DCHA | American Schools and Hospitals Abroad
[email protected]
202 758 9283
8