Theory of Change, Outcome Mapping and AcT’s Logical Framework
Kate Dyer, Programme Director
May 2012
1. Background
The AcT Programme is designed to increase government responsiveness and accountability through
a strengthened civil society. The Programme was originally conceived as a 5 year programme it will
now run for 6 years from 2009 to 2015, with a budget of £31m, and now also includes ring-fenced
finance for governance and accountability issues linked to environment and climate change, which is
partially funded by DANIDA.
The original proposal was to use outcome mapping (OM), in particular a specific adaptation of it,
called the RAPID Outcome Mapping Approach 1 (ROMA) which introduced a focus on political
economy. We were originally drawn to this tool to counter what we saw at the time as the in
incentive for CSOs in Tanzania to focus their work more at the output level (on training, workshops,
carrying out pieces of research), without thinking through how to achieve transformational change,
which required a more nuanced contextual understanding, clear strategic thinking and calculated
risk taking. In the first couple of years of implementation, the programme team developed a working
understanding of how to integrate OM with logical framework reporting to DFID, documented in a
think piece written in January 2011 2. Several months further of implementation, an output to
purpose review and a great deal of thinking later, this short piece is designed to share with
interested parties how our thinking has moved on. In particular, we have seen the value of making
our theory of change explicit, and making substantial use of the OM results achieved by partners,
developed a log-frame that collates progress markers into conventional indicators
2. Lessons from Programme implementation 2009- early 2012
2.1 Overall Approach
Overall we were confident that our approach was on track. This has been to select CSO partners with
the commitment, knowledge and skills to work in governance and accountability, and to provide
them with individual support, alongside financial grant assistance to implement their programmes.
AcT seeks to promote learning for its partners, individually and collectively, for itself, and for DFID.
Understanding the context and process of change to increase accountability is itself a new and
dynamic process, and to support this learning and to help measure results we are using outcome
1
www.odi.org.uk/RAPID/Tools/Toolkits/Communication/Outcome_mapping.html
Dyer, Kate (2011) Think Piece: Working with Log-Frames and Outcome Mapping in the Context of the
Accountability Tanzania Programme (AcT)
2
mapping. In essence, AcT is a governance and accountability learning programme that uses grants
as a modality for action, rather than a simple grant making programme with a learning component.
2.2 Challenges
By early 2012, our thinking on integrating outcome mapping with the logical framework approach
had moved on and it was obvious that we needed to commit ourselves to some results, and results
reporting, that were a better reflection of what the programme (us, as programme managers, and
our CSO partners) were actually doing and needed to be held accountable for, beyond what was in
the original programme documents and Logframe. This section outlines some of the areas where we
needed to change.
This logic model illustrates the early thinking about the AcT Programme. The output and purpose
were the principle focus of reporting to DFID. OM was seen as a ‘black box’ in which attitudes and
behaviour change would be achieved as a means of enabling the achievement of higher level results.
Goal
To contribute to the achievement of MDGs by ensuring that Tanzanians are
increasingly able to claim and exercise their rights as citizens
Improved Outcomes at sectoral level (e.g. inc enrolment, etc)
Purpose
Output
To increase the accountability and responsiveness of government
to its citizens through a strengthened civil society
Citizens’ access
to information
is improved
CSO engagement in
Strengthened CSO
policy and budget monitoring of service
formulation
delivery and public
processes at local resource management
and national levels
increased
Improved
understanding by civil
society of what works
in improving
accountability and
fighting corruption
Using Outcome Mapping, AcT partners seek to influence attitudes and behavior of boundary
partners in order to bring about higher level results
AcT Partner Organisations
AcT Programming
Input
Grants, Technical advice, Support for organisational learning,
Information in Log-frame
Figure 1: Original AcT Programme Logic Model
Rather than being simply a stage to be passed through along the results chain, it became clear that
attitude and behaviour change would be a pre-requisite for the achievement of all the higher level
results. This was particularly important in terms of explaining OM to partners, and facilitating them
to see how attitude and behaviour change connects to the results they were reporting to AcT, and
other donors, using conventional indicators.
Just and
equitable
society
Practices policies and
legis fr’work of gov &
bus improved
Access to
justice for
poor &
marginal’d
Capacity of civil society,
Media and parl to address
violations and monitor inc.
Legal and
policy
framework
conforms to
HR
principles
Gov efforts to
address
corruption
and free and
fair elections
Compliance to labour
rights, environmental rights
& land rights by corporate
sector improved
CHANGES IN ATTITUDE AND BEHAVIOUR OF KEY PARTNERS
Outputs
Outputs
Outputs
Outputs
Outputs
Outputs
Outputs
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Inputs:
Resources
Inputs:
Resources
Inputs:
Resource
s
Inputs:
Resources
Inputs:
Resources
Inputs:
Resources
Inputs:
Resources
Figure 2: Example of Partner Logic Model, showing how attitude and behaviour change underlies the
achievement of higher level results
In terms of our own reporting to DFID, a number of challenges had become obvious, in the first two
and a half years, partly as observations from the output to purpose review, partly in our efforts to
respond to its recommendations 3, and partly from experience of programme implementation. The
broader results agenda played its part too – with its focus on higher level results:
•
First and foremost, it was obvious that the grantees of the programme were achieving results in
government accountability, but that the conventional log-frame indicators we were using
weren’t picking them up. For example, collecting numbers of partner research and analysis
products cited in official government documents, doesn’t really get to the heart of whether
there is any meaningful influence 4. We did try to pick up actual results achieved, but it was in an
unsystematic way based on some fairly arbitrary sampling of partners. For example, under CSO
engagement in policy and budget formulation, the indicators included ‘Number of documented
cases of partner and partner associates influence on plans and budgets at the national level’,
with an assessment to be based on three of the then partners. To be fair, at the time, we didn’t
have enough results – particularly results using progress markers - in from partners, to be able to
see the potential for something more comprehensive. There was a strong sense of ‘missing
middle’ – something between the outputs and purpose levels that we needed to be picking up.
3
Neil Macdonald /Gondwana Development Associates (2011) Output to Purpose Review of the AcT
Programme
44
And arguably plays into some largely unproducitve CSO work when they lobby hard to get special wording
into a particular government document, and celebrate its achievement, without adequate focus on whether
that document is really the one that is going to drive programme implementation.
This challenge we addressed through working on a theory of change, and in selection of
indicators.
•
Some of the indicators we had were at the wrong level. For example, using World Bank
Governance Indicators and the Mo Ibrahim Index at purpose level – though speaking directly to
the governance and accountability issues at the heart of the programme, and arguably providing
something more objective - was inappropriate, since the scope and scale of the programme were
such that AcT was not likely to be able to have a bearing on whether scores changed. Similarly
the example, cited above, of partner influence on government plans or budgets can better be
seen as a result at purpose level, and something we had the potential to influence, if we could
pick up more systematically – something which outcome mapping enabled us to do (see more
below).
•
The wording of the purpose level caused some confusion: ‘you seem to be a demand side
governance programme, seeking a supply side result’. This was part of the original wording of
the programme document, and not of KPMG’s making. Rather it signaled that this was the
demand side of DFID’s governance programme under their former country strategy for Tanzania,
complementing other work seeking increased government responsiveness and accountability
through the wide range of supply side reforms that DFID was supporting through other
programming. It did, however, require some clarification to potential grantees, who came up
with applications focused solely on demand side work, without a mention of government –
despite the fact that they themselves and the programme management team all knew that
working on demand side governance without any reference to its context and to relations with
government would be fruitless – arguably an exercise in getting CSOs to bang their fists on the
table without consideration to whether this would achieve any constructive results. Working on
our theory of change (ToC) addressed this challenge, and speaks to the value of making the ToC
explicit in the design phase.
•
A further challenge was that the purpose statement has two levels: ‘increased government
responsiveness and accountability through a strengthened civil society’. Similarly at goal level,
there are two levels ‘To contribute to the achievement of MDGs by ensuring that Tanzanians are
increasingly able to claim and exercise their rights as citizens. It became apparent that we
needed to make our ToC explicit, about which more below, in order to clarify what we really
thought we were doing and what results we were aiming for.
•
We needed a measure that would enable us to monitor and learn from results of working with
partners as a sub-set of strengthening civil society. In practical terms, the programme
management team was spending a great deal of time with partner organisations, supporting
them initially over strategic planning, but also over ensuring a tighter link between plans, budget
and reporting. We needed to be sure that partners plans constituted value for money (VfM) – in
all the dimensions of economy, efficiency, effectiveness and equity. This, we felt, was also a
contribution to strengthening civil society. We decided to ‘upgrade’ a programme management
tool we called ‘progress markers for partners’ to address this point – see below 3.3
3. Responses
As can be supposed, what was required was some fundamental rethinking. Many individuals
contributed to this, including AcT partners, staff in the AcT Programme Management Team, staff in
DFID Tanzania and from the Fragile States and Good Governance team in London, participants and
organisers at the Outcome Mapping workshop held in Beirut in February 2012, where some ‘work in
progress’ was presented, and Paul Thornton of Verulam Associates.
3.1 Theory of Change
Theory of change sets out the assumptions and causal links necessary to achieve the desired
outcome/s rather than simply setting out the steps (inputs, outputs, outcomes) in the process of
delivering the programme. It is an outcomes-based, participatory method for planning, evaluation
and organisational capacity building, and hence we felt it was appropriate for the AcT Programme,
as a working tool. Assumptions change as learning develops or as the context responds to external
influences including, but not limited to, the programme itself. This is different to the conventional
Logframe that is used to describe a programme and to identify the indicators of progress. A logical
framework is more static and a ToC more dynamic. However, the two can be related, and it was felt
that by making our ToC explicit, we would be able to translate key elements of it into the Logframe,
to enable effective management of the programme.
After broad consultation, we decided on a short form ToC as follows:
‘ Supporting civil society partners to implement context-specific strategic interventions
will enable them to influence positive change in the attitudes and behaviour of citizens,
civil society and government, making government as a whole more responsive and
accountable.’
The simple outline focuses on the outcome of a more responsive government. It shows that the way
AcT expects to achieve this is through behaviour change amongst citizens, civil society and
government actors. Thus all the CSO grants are to support strategic interventions that influence the
attitudes and behaviour of these actors.
Below is the fuller narrative which gives more detail about the inputs, process, outputs and
outcomes. This narrative does not spell out the assumptions in detail – these are attached as an
annex.
‘If civil society grantees are carefully selected and respond to individual support tailored to
their programming and internal systems, they will be able to develop targeted strategic
interventions which are sensitive to changes over time and in the broader political
economy, as well as their geographic location, their sector, institutional mandate and
values.
And if grantees also commit to systematic learning individually and collectively the work
they do will be more the effective.
CSOs implementing programmes will engage in a range of information generating and
disseminating activities as well as developing the capacity of other stakeholders to
articulate their roles and responsibilities.
Some participatory activities build directly into citizen action and civil society
strengthening, whereas others focus on influencing the behaviour of elected and
appointed officials and of the judiciary – at local and national levels.
Influencing activities can be formal or informal, inside track or outside track, and CSOs
become more adept at selecting which is going to be most effective under what
circumstances.
The result of the behaviour changes on the part of key stakeholders is the purpose level of
the programme: ‘Increased responsiveness and accountability of government through a
strengthened civil society’
Focusing on ToC in this way helped clarify the issue of two levels in the wording of purpose. Our
purpose is “increasing government responsiveness and accountability” and “strengthening civil
society” is a means to that end, and hence assessing changes in the strength of civil society need to
be picked up lower down the logic chain. Similarly at the goal level, it became clear that the
achievement of MDGs can be better seen as a ‘super goal’. ‘Tanzanians are increasingly able to
claim and exercise their rights as citizens’, the second half of the original goal statement, can better
be seen as a means to this end, and is effectively being picked up lower down the model in terms of
access to information and citizens taking action, rather than right at the top.
The next step was to translate the ToC into the log-frame, and in doing so the following logic model
was found extremely useful, derived from the longer narrative ToC.
Goal
MDGs Achieved
Purpose
Outputs
Process
Outputs
Increased Accountability and Responsiveness of
Government
Behaviour Change
Civil Society
Citizens
Citizen
Action
Influence on behaviour of
elected representatives,
government officials, judiciary
Knowledge Generated
Process
Inputs
Government
Information Disseminated
Civil Society
Strengthened
Capacity Built
Targeted Strategic Interventions
Individual and Shared Learning
Selection, Tailored Individual Support,
Grants
Figure 3: Revised Logic Model for AcT Programme
It highlighted that in collecting results we needed to be picking up what are here called process
outputs, and that we could do it either at the level of ‘Knowledge generated/information
disseminated/capacity built’ or at the level of ‘citizen action/behaviour influenced/or civil society
strengthened’. The former risked being too low down in the model, with the risk of replicating the
problem of the missing middle experienced before, and the latter too high with the risk of not
picking up enough of the process that we knew our partners spent much of their time and energy
on. To be pragmatic, we used a mixture of both, and the following section describes how they were
translated into indicators.
3.2 Design and Selection of Indicators – making the link with Outcome
Mapping
The value of the OM results partners had been collecting over the previous months of the
programme were an immensely valuable resource here. Not surprisingly, many partners had
identified similar boundary partners:
• elected representatives at national level (MPs) and at local level (councilors),
• officials/civil servants at national and local level
• Citizens and communities, NGO/CBO partners
• The Media
Very often their outcome statements talk about changes in policy and practice – the main
differences being by sector or by part of the country, according to the mandate and focus of the
organisation. Results reported by partners were already being stored on a database, and by pulling
out all the results by a particular boundary partner, overall patterns became clear. The early focus of
partners’ work, usually expressed at ‘expect’ or ‘like to see’ level was on getting other stakeholders
to engage with the CSO’s concern, to develop knowledge or share information with them and to
build a shared understanding with them. At ‘like to see’ or ‘love to see’ level were changes in
behaviour of varying degrees of magnitude, depending on the ambition and scope of the
organisation concerned5. Strategies of different organisations, unsurprisingly differed. For example,
for some the emphasis was on building capacity for citizen action at grassroots level, or others to use
a ‘research and advocacy’ approach. Nevertheless, by early 2012, we had enough results from
partners to see that by collating them in terms of number and description we would be able reflect
the richness and diversity of what was being achieved. This would capture the scope and scale of
5
At one point we experimented with the idea of trying to report on results by boundary partner – drawing
directly from the database of progress markers, to see what was being achieved at expect to see/like to
see/love to see level, as reported by partners. This was presented as work in progress at the Beirut OM
workshop. The failing of the approach was that partners’ levels of achievement and ambition differed – what
was transformational ‘love to see’ change for one, might only be ‘like to see’ and hence more attainable by
another. Collating them together didn’t generate a clarity about whether there had been any real movement
from process to actual changes in policy and practice. Hence we focused in the end on the actual results we
wished to see from key stakeholders – the changes in policy, planning, legislation, bye-laws, budget and so on,
that was brought about by partner action, even if one partner counted the change as a love to see and another
as only a like to see.
the programme, but also be the stimulus to further reflection and learning about what was actually
being achieved and how – provision for which is made through AcT’s Learning Strategy.
It may prove sensible, in time, to see whether the indicators can be further refined 6, but at this stage
we are proposing the following:
Purpose Level
•
•
•
•
Number (quantitative indicator) and description (qualitative indicator) of examples from AcT
Partners or their associates of elected representatives at national level who make specific
decisions or take action as a result of citizen engagement (covering, responding to complaints, or
policies, practices and budgets successfully influenced).
Number (quantitative indicator) and description (qualitative indicator) of examples from AcT
Partners or their associates of appointed officials at national level who make specific decisions
or take action as a result of citizen engagement (covering: responding to complaints, or policies,
practices and budgets successfully influenced).
Number (quantitative indicator) and description (qualitative indicator) of examples from AcT
Partners or their associates of elected representatives at local level who make specific decisions
or take action as a result of citizen engagement (covering: responding to complaints, or policies,
practices and budgets successfully influenced).
Number (quantitative indicator) and description (qualitative indicator) of examples from Act Partners or
their associates of appointed officials at local level who make specific decisions or take action as a result
of citizen engagement (covering,responding to complaints, or policies, practices and budgets successfully
influenced).
Output Level
Programme outputs relate to the behaviour change amongst citizens, civil society and elected representatives,
officials and the judiciary. The process outputs arising from CSO partner activity provide the knowledge
generation, information dissemination and capacity building that all result in:
Output 1: Citizen Access to Information Increased.
Indicators for output 1 are:
•
•
•
6
Numbers reached through most extensive means of information dissemination: TV, National Radio,
National Newspapers
Numbers reached through targeted information dissemination: Local Radio, large print runs of
publications, outreach through community meetings, website hits, u-tube downloads
Number committed to AcT partner programmes, through targeted training, taking on identified roles in
CSO activities, signing up as campaign members etc.
In fact, at a ‘Learning Event’ for AcT partners, in May 2012, partners made the observation that our results
were very ‘individualised’, and didn’t do sufficient justice to generating systemic change. Two concrete
suggestions were: (i) separate out citizen action into individual actions (for example, sending an sms to make a
complaint or to raise an issue with an MP), and collective actions (for example, joining a PETS Committee, or
taking part in a campaign) (ii) bringing partners together to review the ‘state of governance and accountability’
in a particular sector or around an issue, based on, and complementing the quantitative reporting and ensure
that there is a fuller view of the ‘bigger picture’. We are now working on how to do this.
These correlate with the information it has been agreed to collect in relation to DFID’s corporate indicator on
accountability and empowerment.
Output 2: Civil society strengthened
There is no overall mapping of civil society in Tanzania that is available in a form which would generate useful
7
information out output level . The data available on the development of AcT CSO partners is however
increasing as a result of outcome mapping and this extends to the networks of smaller organisations and
associations linked through each CSO partner. The first indicator attempts to capture citizen action, where
individuals take action in relation to a governance issue. The second focuses on the level and type of activity
by AcT grantees and their partners, members, or associates.
The suggested indictors are:
•
•
•
Number (quantitative indicator) and description (qualitative indicator) of citizens raising complaints, or
seeking to influence policy or practice
Number (quantitative indicator) and description (qualitative indicator) of AcT Partners and their associates
raising complaints, or seeking to influence policy or practice
Indicators 3-5 are taken from performance measures of AcT partners maintained by the Programme
Management Team. These are explained in detail in the next section.
Output 3: Behaviour of elected representatives, officials at local and national level Influenced
Behaviour change by elected politicians at local and national level, civil servants and public officials and
members of the judiciary is not easily measurable at scale. The AcT CSO partners’ strategic interventions
include engagement with politicians and public officials to inform them through information and dialogue
events. Around specific campaigns CSO partners target individual politicians and public officials to make
decisions or take action. The numbers of national and local politicians and public officials informed and the
numbers targeted to make decisions are proxy indicators for those whose behaviour changes. Thus for this
output the following indictors are proposed:
•
•
•
7
Numbers (quantitative indicator) and description (qualitative indicator) of elected representatives at
national level directly engaged with through dialogue processes, targeted dissemination of information,
joint learning etc
Numbers of elected representatives at local level directly engaged with through dialogue processes,
targeted dissemination of information, joint learning etc
Numbers (quantitative indicator) and description (qualitative indicator) of appointed officials at national
level directly engaged with through dialogue processes, targeted dissemination of information, joint
learning etc
This Sustainability Index from USAID does include individual country profiles,
http://www.usaid.gov/our_work/democracy_and_governance/technical_areas/civil_society/angosi/
but we would argue not in a sufficiently nuanced way to be useful to understanding the dynamics
CSOs.
•
Numbers (quantitative indicator) and description (qualitative indicator) of appointed officials at local level
directly engaged with through dialogue processes, targeted dissemination of information, joint learning
etc
Again the data will become richer as Outcome Mapping extends across all partners. New examples and new
indicators may emerge and the narrative data to support these indicators will grow. Nevertheless, interesting
results are already there for 2011/12, and work is on-going to mine them further for what they reveal about
8
the dynamics of governance and change.
3.3 Additional Indicators of Strengthened Civil Society.
In addition to the indicators under output 2, we found it useful to make an assessment of whether
grantees could be seen to be getting stronger as CSOs. This section describes how we arrived at the
indicators and how they are measured.
At the outset of the AcT programme, OM was seen as a tool that would be used in a fairly limited
way to strengthen the way partners did their strategic planning. It wasn’t really envisaged as such a
major component of the whole programme. Greater familiarity with its scope and potential enabled
the programme management team (PMT) to see that the results of the AcT programme are in fact
results achieved by partner organisations, which we could only influence, not control. In this sense,
the grantees of the programme can be seen as the boundary partners of the AcT Programme.
During 2010, the second year of the programme, when we were supporting partners to adopt the
use of progress markers, and were observing the challenges they faced in terms of the discipline of
keeping and using journals, the PMT decided to try and use progress markers ourselves on the basis
that we shouldn’t be asking partners to undertake a challenge that we weren’t prepared to
undertake ourselves.
Hence we developed a set of descriptors at three different levels (expect to see, like to see, love to
see) for what we saw as the essential criteria for a successful CSO partner of the AcT Programme.
We called these our ‘progress markers for partners’, and they cover the following criteria:
1. Governance
2. Quality of strategic approach and planning
3. Organisational development
4. Approach to financial management
5. Operational Plans and Budgets
6. Financial and Operational Reporting
7. Value for money
8. Gender and Marginalised Groups - especially disabled
8. Narrative Reporting / Strength of results reporting
9. Technical use of monitoring, learning and outcome mapping
10. Approach to learning
11. Political Economy Analysis
8
www.accountability.or.tz/wp-content/uploads/2012/05/Summary-of-Partner-Results-in-Figures-.pdf
Full details of the ‘descriptors’ for these criteria are attached as annex 2, and in a slightly earlier draft
were shared with partner organisations for feedback.
The way we use them is as with an outcome journal – when we have any interaction with partners
(phone, email, meeting) and observe something that gives an indication of partner performance
against any of these criteria it is noted down on the spreadsheet for that partner. At the time of sixmonthly reporting from partners a judgement is made about what level they are working at.
During 2012 the decision was made to ‘upgrade’ the Progress Markers for Partners, and to use them
as the basis for making a judgement as to whether the AcT partners are improving in terms of
strength and effectiveness, which would enable us to include the changes in our log-frame
indicators. Undoubtedly there is a subjective element to this judgement, but by pinning it down
directly to clearly observable criteria, it is open to external validation. One of the goals for 2012 is to
use six monthly meetings with partners to get their own assessment of where they would place
themselves (as well as for them to give us feedback about our performance). Moreover, in the
absence of indicators of this sort, we would not be monitoring one of the key elements of the AcT
Programme which is the idea that by giving AcT partners carefully targeted individual support they
will become more effective in doing what they set out to do.
In terms of translating these into log-frame indicators we have selected the following, to supplement
the other indicators under output 2:
•
The approach and application of learning by AcT grantees, measured by the proportion
performing at ‘like to see’ level of the AcT “Progress Markers for Partners” number 10. This
reflects the fundamental importance we attach to learning and reflection as a central means
to improve the effectiveness of civil society – to contribute to an understanding of the ‘big
picture’ and to move away from ‘activism’ as an end in itself.
•
The application of Political Economy Analysis by AcT grantees, measured by the proportion
performing at the ‘like to see’ level of the AcT Progress Markers for Partners number 11.
This is because AcT is a governance programme, and being familiar with the theory and
practice of political economy analysis is argued to be central to effectiveness.
•
The median score, expressed as a percentage, of AcT grantees operating efficiently as
measured by a blend of Progress Markers for Partners criteria 2, 4, 5 and 6. These cover
quality of strategic approach and planning, approach to financial management and the
links made between plans, budgets and reporting. Each organisation is given a score of 1, 2
or 3 according to their performance against each criterion. The maximum possible score
across the four indicators would be 12. Actual scores are then presented as a percentage,
and the median established.
With DFID’s strengthened focus on VfM, we intend to develop an indicator around that, but need to
have further consultations with partners on how best to achieve it. It is not ideal to separate it out,
since our overall approach to VfM is programme wide, encompassing everything from strategic
planning, to internal governance and oversight, to financial management and reporting. However,
not separating it out may give the impression that it is not being taken seriously. This parallels the
arguments about whether or not to ‘mainstream’ any of the cross cutting issues, when the observed
experience of many programmes is that, through inadequate skill or emphasis at implementation, in
practice, mainstreaming has led to a losing sight of the issue in question.
4. Conclusion
AcT Programming, monitoring, learning and reporting has come a long way since these elements
were first designed at the outset of the programme in 2009. This reflects the strength of an
emergent design approach, rather than pinning down a baseline and indicators at the start of a six
year programme and continuing to monitor and report results in the same way, regardless of
changing context and dynamics. Emergent design is particularly relevant to governance
programming, given the changing political economy of Tanzania, and fundamental to a programme
in which so much stands or falls by the quality of learning.
We feel that we have a much more effective way of reporting partner results than the ones used
earlier in the programme. Rather than seeing OM results as something of a parallel reporting
process that would hopefully stimulate internal reflection by individual partners, we are now using
the results that partners are reporting to us in a systematic way that paints a broader picture beyond
individual organisations, and with the potential to shed light on the bigger dynamics of citizen
action, civil society and what drives change in government responsiveness and accountability. How
to do this would not have become apparent to us without making our theory of change explicit.
However, we are not complacent and are continually looking for how to improve the monitoring
work, and to learn from the results it generates.
Annex One: Theory of Change with Assumptions
With these INPUTS
If civil society organisations are carefully selected and respond to individual support tailored to their programming and internal systems, they will be able
to utilise grants
Assumptions from inputs to process:
•
the AcT teams selection process is successful in identifying organisations that are committed to bringing about change (as opposed to just
administering aid money with a governance spin on it)
•
•
the AcT team has the skills and judgement to provide appropriate support, manage risk, and pro-actively manage the portfolio
•
the CSOs have the relevant sectoral and district-specific knowledge and understanding
The CSOs can develop familiarity with, and confidence in, working with the changing dynamics of political economy and/or develop organisationally
to facilitate working with this understanding
and the following PROCESSES
to develop better targeted strategic interventions which are sensitive to changes over time and in the broader political economy, as well as their
geographic location, their sector, institutional mandate and values; and if these organisations also commit to systematic learning individually and
collectively the work they do will be more the effective.
Assumptions from process to outputs:
•
•
• Systematic learning enables CSOs to grow and move beyond:
‘business as usual’ – rolling out the same approaches and methodologies
‘copy cat approaches’ e.g. the spate of PETS training and studies that were carried out
‘chasing the money’ i.e. following donor priorities rather than their own analysis
CSOs become aware of the positive and negative lessons learned by others working in similar fields
•
CSOs monitor their own effectiveness and make changes as appropriate
The learning strategy, to which CSOs contribute and derive benefit, enables them to document ‘lessons learned’ in a way which will be useful to
themselves and others.
•
CSOs maintain ethics and integrity
We can expect the PROCESS OUTPUTS
CSOs implementing programmes will engage in a range of knowledge generating and information dissemination activities as well as developing the
capacity of other stakeholders to articulate their roles and responsibilities.
Assumptions from process output to programme output:
•
Citizens are stimulated to respond to the knowledge and information made available either by being directly engaged in the process, or in planning
action on the basis of findings.
•
Citizens see the value of taking action based on knowledge, information and capacity built.
•
Participatory approaches are empowering and can stimulate action
•
Citizens overcome fear and apathy and encourage others to join action
•
Decision makers recognise that they will not retain power unless they respond to the increasingly assertive citizen
•
Decision makers are open to citizen and civil society action.
That will result in the PROGRAMME OUTPUTS
Some participatory activities build directly into citizen action and civil society strengthening, whereas others focus on influencing the behaviour of elected
and appointed officials and of the judiciary – at local and national levels.
Influencing activities can be formal or informal, inside track or outside track, and CSOs become more adept at selecting which is going to be most
effective under what circumstances.
•
Assumptions from outcome to purpose:
Individual elected representatives, appointed officials and members of the judiciary are able to influence the politics and systems that frame their
actions.
•
Legislation, state systems and official processes are open to change.
That will ensure the PURPOSE WILL BE ACHIEVED
The result of the behaviour changes on the part of key stakeholders is the purpose level of the programme: ‘Increased responsiveness and
accountability of government through a strengthened civil society
13
Annex Two: Progress Markers for Partners
Category
Baseline
Expect to see / start to see
Like to see
Love to see
1. Governance
Basic
structures
in place
* Local NGOs: Meetings taking place and
documented * INGOs: basic technical support
eg to planning and reporting
* Local NGOs - meaningful engagement by
Board eg in setting strategic direction,
commissioning and receiving audits, Mid-Term
Review and Evaluations * INGOs demonstratively Tanzanian in orientation,
rather than simply following an international
agenda
* Local NGOs - demonstratively providing both
a challenge and a support function; ownership
for longer term goals; detailed oversight of
budget; INGOs - International level
demonstratively adding value - eg to
campaigning, to providing relevant experiences
from other countries
2. Quality of
strategic
approach and
planning
Little or no
engagemen
t with PEA
in the SPs
* Basic ownership of organisational strategies,
e.g. external consultants used to facilitate, not
write strategies; *Limited orientation to
results, eg clarity about outputs but weak links
to higher level results, or organisation's mission
and vision; 'missing middle' * Limited strategic
thinking - more roll out of long term
operational plans with little flexibility * Weak
link in current plan to lessons learned from
previous one * Incorporating a basic
understanding of PEA into work planning – eg
working with informal as well as formal ‘rules
of the game’
* Organisational strategies owned and
understood throughout the organisation, not
just senior staff *Strengthened focus on results
from output to outcome to impact.
* Basic application of learning to refine
strategic approach eg through reviews of
strategic plan, annual planning processes etc *
Initial reflections on reporting of progress
markers/conventional indicators - stimulating
thinking about strategies and theories of
change in relation to particular boundary
partners * Periodic internal assessments of
effectiveness of operations and strategies
* Robust internal challenge of strategies, work
plans and budgets both by Boards and internal
functions, i.e. AcT able to fund programmes
without significant consultation or changes. *
Strong results focus - clarity and accountability
for delivery of results at output level, but clarity
of thinking about how these connect to higher
level results
*
Strategic plans are genuinely documents which
provide strategic guidance, as opposed to being
5 year work plans. * Strategies started,
stopped and tweaked in response to lessons
learned and changes in the external
environment, in order to deliver planned
results. * Sophisticated reflection on picture
emerging from monitoring of progress markers
and of strategies - this leading to changes in
strategic approach and documentation of
lessons learned
14
3. Organisational
development
4. Approach to
financial
management
Have to
have
passed
FMA
* Basic understanding of the organisation's
strengths and weaknesses and possible
challenges/risks * Openness to discuss/ think
through programmatic and operational
challenges
* More clearly articulated understanding of
strengths and weaknesses, challenges/risks/
opportunities *Leadership plans and prioritises
how to address organisational weaknesses using participatory approaches where possible
and relevant * Systems of performance
appraisal in place and operational
* Leadership appreciate value of strong
financial management, e.g. timely and
appropriate response to FMCRA issues *
Strong internal controls in place
* Budget holders demonstrate ownership and
understanding of their own budgets, so there is
little variance from budget and reasonable
explanation where it does happen *
Willingness to share internal financial
management information rather than await
audit. * Demonstrated action to ensure internal
controls are operated upon * Adoption of good
practice from elsewhere to reduce risk of fraud
or misuse of funds
* Clear attempt to link higher level results,
strategies, work plans and budgets. * Lessons
learned in previous year about realism carried
into current plans * Assumptions made
transparent * Information readily available to
show how the budget is derived
5. Operational
Plans and
Budgets
* Basic link between higher level results,
strategies, work plans and budgets * Weak on
transparency of assumptions and/or
practicality and realism of plans (eg given
budget, size of organisation, newness of
approaches) * Mainly lump sum budgets
6. Financial and
Operational
Reporting
* single consolidated report which covers the
whole organisation * Clarity about quantitative
and qualitative results at output level, but not
presented in a way which links back to budget
* All documents received within one month of
end of reporting period. * Minimal to-and fro
to clarify financial reports and requests *
financial reports and requests on cycle with
agreed disbursement timings *Consistency
between management and audit financial
reports * Narrative and financial reporting
linked and relating back to plans, and forward
into outcome level results * presentation of
reports in a way which makes it easy to link
back to plans and explain variance
* Understanding within the organisation of how
OD bears on ability to deliver planned
strategies, and hence ability to change
organisational structures if required; openess
to experiment to improve performance *
Leadership progressively addresses
organisational weaknesses - shown in higher
performance - eg on time reporting, response
to queries, ultimately in achievement of results
* Robust internal challenge of strategies, work
plans and budgets; *systems and processes
regularly spot-checked internally and updated
to ensure effective reporting, VFM. * Actively
innovating to seek methods to reduce risk of
fraud or misuse of funds
* Plans and budgets fully aligned along
appropriate strategies with a clear link between
financial information, strategies / outputs and
outcomes * Reports demonstratively lead into
plans and budget for next cycle.
* Same report to all donors and governance
structures * concise reporting with appropriate
level of detail to different stakeholders *
Reporting makes decision making and advice
simple eg comparisons with previous years and
reporting periods
15
7. Value for
money
Lack of
familiarity
* Basic awareness, but implemented largely in
terms of cost management * Material budget
items based on costed inputs not ‘lump sums’
* Tendency to see VfM as largely a finance or
back office issue, not one for the whole
organisation
* Some knowledge of justification for a VfM
focus * Broadly positive attitude across the
organisation towards strengthening this *
Basic practice around VfM strengthening eg
ability to identify and monitor relevant unit
costs * Initial plans in place for how to
strengthen VfM
* Able to demonstrate value for money beyond
lowest cost * Familiarity across organisation
with issues of economy, efficiency and
effectiveness - especially at level of budget
holders. *Plans being implemented to improve
VfM eg Evidence of changing suppliers
/approach/strategies in order to improve VfM
8. Gender and
Marginalised
Groups especially
disabled
Gender
blind
interventio
ns;
'invisibility'
of disability
Sex disaggregation of data. Reference to
disability
Recognition in planning and reporting of
different strategic and practical needs
Achievement of results which respond to
strategic and practical needs
8. Narrative
Reporting /
Strength of
results reporting
* Able to report on progress against work plan
activities and predetermined output targets. *
Bias towards narrative reporting rather than
analytic, with focus on activities carried out not
results achieved/ or only results to output level.
* Largely qualitative reporting, with results
'embedded' in narrative text rather than being
clearly identifiable.
* Reporting provides a compelling narrative of
what has been achieved, why and how, which
feeds readily into broader understanding of
governance and accountability issues. * Able to
track through results at purpose level to impact
- and to quantify this where relevant or useful
9. Technical use
of monitoring,
learning and
outcome
mapping
* Overall understanding of value of ML&E to an
organisation * willingness to engage with AcT
in sharpening ML&E * If not using OM, then
own systems demonstrably supporting
organisational learning and strategy
development. *Unless deemed very strong,
showing willingness to take on OM; *Starting
to use outcome, strategy and organisational
journals, with governance results being clearly
identifiable.
* Reporting shows signs of being analytic and
concise - eg governance results clearly
identifiable and overall messages clear; *
monitoring (whether OM or RBM) used to
generate reflection on organisational efficiency
and effectiveness - and this is reported on; *
Able to report quantitative results as well as
descriptive ones * Qualitative results answer
questions Who? What? When? Where? Why? in a way which facilitates follow-up and
verification.
* Engaging with AcT staff on technical issues
related to monitoring and result reporting bot
AcT and to other stakeholders * Engaging in
constructive discussion with AcT staff about
technical issues arising with using OM;*
participating in e-learning group; * Innovating
with different ways to link OM with results
based reporting, especially where multiple
donors are involved
* Robust ML&E systems enable strong results
reporting. *Contributing to international
discussions, e.g. OM discussion group;
*Successfully using OM with own partners,
independently of support provided directly by
AcT; * doing advocacy to other donors to take
up OM
16
10. Approach to
learning
* Limited or formulaic documentation of
lessons learned but not in a way that influences
the approach or ways of working of the
organisation *Engagement in learning strategy
shows early attempts to engage with theories
of change such as understanding technical and
political dynamics * Limited learning going on
independently within the organisation (eg
learning events, reading circles) - not part of
organisational mainstream
* Active approach to learning within the
organisation * Active engagement in AcT's
learning strategy - eg sharing lessons learned
amongst AcT partners; * Evidence of reflection
on results driving thinking within the
organisation about what could or should be
done differently * Reflections on results
demonstrate thinking about how technical and
political dynamics effect achievement of results
11. Political
Economy
Analysis
* Basic understanding from OM or other
planning methodology, and beginning to show
influence in programming as move away from
'business as usual' * Basic thinking about the
connection between citizen action and NGO
action* Understanding of the limitations of
conventional information sharing / awareness
raising / capacity building approaches
* Making reasoned strategic choices over
working at national/policy level and with
community mobilisation; * Demonstratively
moving from information sharing to facilitating
action * Planning and reporting shows some
thinking in terms of PEA
* Shared and dynamic understanding about
what has worked in bringing about change in
accountability, based on lessons learned
internally and from other AcT partners; *
Sharing lessons learned outside the AcT family,
including contributions to international debates
* Qualitative and quantitative governance
results clearly identifiable in narrative or
progress marker annex * robust challenge
within organisation about what has actually
been learned. * Contributing understanding
oftechnical and political dynamics to support
learning of other AcT partners
* Pro-actively engaged in current events,
successfully taking calculated risks *
Documenting and communicating practical and
theoretical understanding of PEA through
reporting and other publications
17
18