Who we are

» Purpose
˃ Shortly share the evaluation framework again to ensure everyone
knows why we are spending time on this
˃ Discuss some ideas about how we will be measuring Impacts (By the
May Meeting)
˃ Share some of the preliminary evaluation findings relating to the
budget / cost analysis and get inputs on a membership typology in
terms of cost
» Key to its sustainability, long-term growth and
continued relevance to the learning community
served by the Coalition.
» Will essentially address three key questions:
How is the
Coalition
performing?
How is the
Coalition planning
for sustainability?
How can the
Coalition be
strengthened?
How is the
Coalition
performing?
How can the
Coalition be
strengthened?
How is the
Coalition
planning for
sustainability
?
» Learning: It is anticipated that the M&E will be used by
the coalition’s project manager and key decision
makers to inform its future work, and improve its
practice.
» Accountability: To prove to its donors that the
resources provided to the organization is spent well
and is delivering good results
» Building the Knowledge base: To supplement the
management of Bridge’s knowledge base about how
communities of practice like this one can function
optimally.
How is the
Coalition
performing?
How can the
Coalition be
strengthened?
How is the
Coalition
planning for
sustainability
?
» It is important to consider the relevance,
effectiveness, impact, budget and timeline
compliance, and sustainability of the
Coalition
(Organization for Economic Cooperation and Development –
Development Assistance Committee Criteria)
» If the coalition is successful we would expect
˃ People to react positively
˃ People to learn something (Skills, knowledge, values &
attitudes)
˃ People to change their individual behaviours
˃ Organizations to start transforming
Donald Kirkpatrick, the Strategic Training Model
Relevance is the
extent to which
the coalition
and its activities
are suited to
the priorities
and policies of
the target
group, recipient
and donor.




How has the coalition’s objectives and activities changed since inception,
and why?
To what extent are the objectives of the coalition still valid?
Are the key activities and outputs of the coalition still relevant to the
overall goals of the coalition?
Are the key activities and outputs of the coalition still relevant to the
intended
o impacts and effects influencing policy and practice, (finding a
solution to the education crisis)?
o impacts in schools?
POSSIBLE EVALAUTION METHODS:
Interviews / survey with key stakeholders and thematic analysis.
Review of documented objective statements over time.
Effectiveness is a
measure of the
extent to which
the coalition
attains its
objectives.
 To what degree did the coalition support its membership base
to achieve and maintain the coalition’s values / minimum
criteria?
 To what extent were the planned objectives achieved / are
likely to be achieved?
 What were the major factors influencing the achievement or
non-achievement of the objectives?
POSSIBLE EVALUATION METHODS: Outlined in relation to each of
the aspects contained in the logical framework
ACTIVITIES
OUTPUTS
4 x National Seventeen participating
workshops
schools attend 4 x
per year
national workshops (30
where each – 50 participants)
of the five
work
2011 – 2012 All
streams
participating schools
send at least one
meet
participant to each of
the meetings
OUTOMES
Satisfaction: Members report that
the coalition provides a safe space
for difficult conversations (trust)
(Members Survey)
IMPACTS
More coalition
schools can show
strong evidence of
maintaining criteria
for membership and
Attitudes: Members feel that the implementation of
coalition members have a
principles of
common purpose (Analyse
excellent practice
meeting notes, Members survey) (Annual reports
from schools on
Knowledge: Individual participants criteria, Stories /
2013: All participating
and Member schools as a whole Case studies for
schools send at least
shared and learnt new working
principles)
two participants (Leader practices from one another
and an instructional
(Analyse meeting notes for
Schools improve
practitioner) to all
evidence of sharing, survey asking performance on
national workshops
for concrete examples rated on a reported peer
rubric)
review results over
By August 2012:
time (approximately
Involvement of
Behaviour: New working practices 3 year cycle)
education department adopted by participating
Peer review reports
stakeholders, and
individuals and schools as a whole where at least two
education agencies in
(Survey asking for concrete
are available for
The positive and • What were the intended / unintended changes, that the coalition
negative
contributed to at different levels?
changes
• How have individuals and member organizations used the gains from the
produced by the
coalition meetings?
coalition,
• How many people have been affected?
directly or
o Participation in meetings / peer review / social media
indirectly,
o Number of children in Coalition schools, number of success cases in
intended or
Coalition schools
unintended.
POSSIBLE EVALUATION METHODS:
Monitoring records on participation and school enrolment figures
Social media monitoring (Twitter, Facebook, Bridge online)
Interviews / Survey questions
A consideration of
the actual costs
required to run
the coalition in
addition to the
budgeted grant
amounts, and an
indication of the
extent to which
the coalition met
timelines




What financial and non-financial contributions did the coalition
leverage for its work over time? (e.g. core donor funding versus
inputs from schools vs. funding from other donors)
In which ways did the coalition contribute towards using the financial
and other contributions optimally (i.e. did the cost of participation go
up or down over time?)
Did the coalition meet key project milestones timeously and if not,
why not?
Were the meeting outputs made available to all participants
timeously, and if not, why not?
POSSIBLE EVALUATION METHODS:
Cost analysis, Verification of milestone analysis
Sustainability is
concerned with
measuring whether
the benefits the
coalition are likely to
continue after donor
funding has been
withdrawn.
• Is the coalition still valuable enough to keep participants involved?
• Are the networks that were established of significance to sustain
future maintenance of the coalition?
• Does the coalition have a concrete plan to sustain its activities
should current funding cease? (E.g. Are Network champions
developed?)
• Is the model of the coalition scalable / replicable?
• What are the risks to sustainability?
POSSIBLE EVALUATION METHODS:
Stories / Survey, Key informant interviews
» Questions or Inputs?
The positive and
• What were the intended / unintended changes,
negative
that the coalition contributed to at different
changes
produced by the
levels?
coalition,
directly or
• How have individuals and member organizations
indirectly,
used the gains from the coalition meetings?
intended or
• How many people have been affected?
unintended.
o Participation in meetings / peer review / social media
o Number of children in Coalition schools, number of success cases in
Coalition schools
POSSIBLE EVALUATION METHODS:
Monitoring records on participation and school enrolment figures
Social media monitoring (Twitter, Facebook, Bridge online)
Interviews / Survey questions
» At the previous meeting, John asked for some
examples of changes / impacts that the
coalition members have noticed as a result of
participating in the Coalition
• The HODs that participated in the school peer review have come back
with a direct leadership impact in instructional practice
• Looking at the young leaders and young interns – what is possible in
the future –intern programme and gift of transgenerational sharing
• Follow-up on instructional leadership and personal management by
two of the school leaders following the SPR
• Learnt from the timetable structuring at Lebone (SPR)
•Talking about classroom practice – practice of classroom writing – first
10 minutes in the classroom
• Comment on student uniforms – looking at this as a change
• Staff development
To make sense of these stories, we need to know a few
more things!
» Curriculum change – we are embarking in 2013 on an
integrated curriculum approach in gr 8. A team has
met and redesigned our offering of 6 subjects. This is
a direct result of seeing this operative at Lebone 11.
» Curriculum change – we are embarking in 2013 on an
integrated curriculum approach in gr 8. A team has
met and redesigned our offering of 6 subjects. This is
a direct result of seeing this operative at Lebone 11.
» Curriculum change – we are embarking in 2013 on an
integrated curriculum approach in gr 8. A team has
met and redesigned our offering of 6 subjects. This is
a direct result of seeing this operative at Lebone 11.
» Curriculum change – we are embarking in 2013 on an
integrated curriculum approach in gr 8. A team has
met and redesigned our offering of 6 subjects. This is
a direct result of seeing this operative at Lebone 11.
» Impact: 1-5 Individuals, 6 – 30 Individuals, 31 –
100 individuals; Organizational; System Wide
» Impact on Learners: Significant, Tenuous, Not
Applicable
» Impact on Teachers: Significant, Tenuous, Not
Applicable
» Change: Value; Attitude; Skill; Knowledge,
Individual Behaviour, Organizational Behaviour
» Verified: Yes / Partly / No / Not Yet
» Does the template work?
» Any suggestions?
» Can you think of any evidence that relates to you,
your colleagues or your organization that could
help us to document examples of how the
Coalition helped to:
˃ Build and improve the reflective practice capacity of individuals, schools
and organizations
˃ Foster the adoption of new working practices by participating individuals,
schools and organizations?
˃ Reduce the duplication of effort?
˃ Establish effective partnerships?
˃ Expand reach?
˃ Influence broad policy change?
˃ Support Coalition schools to maintain criteria for membership?
˃ Support Coalition schools to implementation of principles of excellent
practice ?
A consideration of
the actual costs
required to run
the coalition in
addition to the
budgeted grant
amounts, and an
indication of the
extent to which
the coalition met
timelines
 What financial and non-financial contributions did
the coalition leverage for its work over time? (e.g.
core donor funding versus inputs from schools vs.
funding from other donors)
 In which ways did the coalition contribute towards
using the financial and other contributions optimally
(i.e. did the cost of participation go up or down over
time?)


Did the coalition meet key project milestones timeously and if not,
why not?
Were the meeting outputs made available to all participants
timeously, and if not, why not?
» Two Sources of Data –
˃ From the Coalition’s Financial Records
+ Good for calculating actual financial contributions
/ expenses
+ Not good for calculating in-kind contributions /
expenses
˃ From the Coalition’s self-reported information
+ May be difficult to reconstruct for each event
after the fact
+ Might be possible to come up with a model of
“typical” cost contributions
Meeting Expenses
Venue
Catering
Office Supplies
Travel
Accommodation
Flights
Cost Estimates
R 85 800
R 30 000
R 55 800
R
R
R 341 300
R 39 900
R 180 000
Claims (MSDF
Contribution)
R
30 645
In Kind
Contribution
R 55 155
R
R
523
121 174
R
R 220 126
R
863 679
R
R
98 005
Other Donor
Contributions (In
Kind)
Ground Transport R 121 400
Project Manager & Bridge Staff
Other Professional Fees
Overheads
Special Projects
R
R
R
R
-
SUBTOTAL
R
For every One Rand the coalition contributed
For Every One Rand the MSDF contributed
-
R 266 470
R 1 114 026
R
2.06
R
1
R 266 470
R 541 751
R
1.00
R
0.49
R
320 000
R
R
R
320 000
0.59
0.29
» Leverage (per year)
» Cost per workshop participant per day (per
year)
» National Meetings
˃
˃
˃
˃
˃
Venue
Facilitation
Catering (x number of participants attending)
Participant Accommodation (x number of staff attending)
Participant Flights & Ground Travel (x number of staff attending)
» Peer Review
˃
˃
˃
˃
Reviewers Fees
Catering
Reviewer Accommodation
Reviewer Flights & Ground Travel
» Steering Committee
˃ Participant Fees
» Typology
˃
˃
˃
Travel and accommodation at National Meetings (Might differ for meetings in own province / other
province)
+ Level 1: Claims for most costs for one person, do not send more staff at own costs
+ Level 2: Claims for most costs for one person, sends one or more staff at own costs
+ Level 3: Do not claim any costs, one participant
+ Level 4: Do not claim any costs, sends 2 – 3 participants
+ Level 5: Do not claim any costs, sends more than 3 participants
Hosts a meeting (National Meeting / Peer Review) and provides for most of the catering costs
+ Level 1: Provides Venue free of charge
+ Level 2: Provides Catering and Venue free of charge
+ Level 3: Provides Catering, Venue and Accommodation
Member that actively contributes time to facilitate National Meetings / Participate as reviewers / lead
reviewer
+ Level 1: Spends time facilitating at least one session at a National meeting
+ Level 2: Spends time participating in steering committee meetings and decision making
+ Level 3: Spends time participating as a reviewer in at least one peer review
+ Level 4: Spends time participating as a lead reviewer in at least one peer review
+ Level 5: Multiple engagement in various of the aforementioned activities
» Is this a credible approach?
» Any suggestions?
1
•FRAMEWORK
2
•IMPACT STORIES
3
•COST DATA
4
•OTHER?
Benita Williams
Feedback Research & Analytics
1102 Burnette Str. (Corner of Hilda), Hatfield, Pretoria
Fax/Phone: (012) 430 2009
E-mail: [email protected]
www.feedbackra.co.za