» Purpose ˃ Shortly share the evaluation framework again to ensure everyone knows why we are spending time on this ˃ Discuss some ideas about how we will be measuring Impacts (By the May Meeting) ˃ Share some of the preliminary evaluation findings relating to the budget / cost analysis and get inputs on a membership typology in terms of cost » Key to its sustainability, long-term growth and continued relevance to the learning community served by the Coalition. » Will essentially address three key questions: How is the Coalition performing? How is the Coalition planning for sustainability? How can the Coalition be strengthened? How is the Coalition performing? How can the Coalition be strengthened? How is the Coalition planning for sustainability ? » Learning: It is anticipated that the M&E will be used by the coalition’s project manager and key decision makers to inform its future work, and improve its practice. » Accountability: To prove to its donors that the resources provided to the organization is spent well and is delivering good results » Building the Knowledge base: To supplement the management of Bridge’s knowledge base about how communities of practice like this one can function optimally. How is the Coalition performing? How can the Coalition be strengthened? How is the Coalition planning for sustainability ? » It is important to consider the relevance, effectiveness, impact, budget and timeline compliance, and sustainability of the Coalition (Organization for Economic Cooperation and Development – Development Assistance Committee Criteria) » If the coalition is successful we would expect ˃ People to react positively ˃ People to learn something (Skills, knowledge, values & attitudes) ˃ People to change their individual behaviours ˃ Organizations to start transforming Donald Kirkpatrick, the Strategic Training Model Relevance is the extent to which the coalition and its activities are suited to the priorities and policies of the target group, recipient and donor. How has the coalition’s objectives and activities changed since inception, and why? To what extent are the objectives of the coalition still valid? Are the key activities and outputs of the coalition still relevant to the overall goals of the coalition? Are the key activities and outputs of the coalition still relevant to the intended o impacts and effects influencing policy and practice, (finding a solution to the education crisis)? o impacts in schools? POSSIBLE EVALAUTION METHODS: Interviews / survey with key stakeholders and thematic analysis. Review of documented objective statements over time. Effectiveness is a measure of the extent to which the coalition attains its objectives. To what degree did the coalition support its membership base to achieve and maintain the coalition’s values / minimum criteria? To what extent were the planned objectives achieved / are likely to be achieved? What were the major factors influencing the achievement or non-achievement of the objectives? POSSIBLE EVALUATION METHODS: Outlined in relation to each of the aspects contained in the logical framework ACTIVITIES OUTPUTS 4 x National Seventeen participating workshops schools attend 4 x per year national workshops (30 where each – 50 participants) of the five work 2011 – 2012 All streams participating schools send at least one meet participant to each of the meetings OUTOMES Satisfaction: Members report that the coalition provides a safe space for difficult conversations (trust) (Members Survey) IMPACTS More coalition schools can show strong evidence of maintaining criteria for membership and Attitudes: Members feel that the implementation of coalition members have a principles of common purpose (Analyse excellent practice meeting notes, Members survey) (Annual reports from schools on Knowledge: Individual participants criteria, Stories / 2013: All participating and Member schools as a whole Case studies for schools send at least shared and learnt new working principles) two participants (Leader practices from one another and an instructional (Analyse meeting notes for Schools improve practitioner) to all evidence of sharing, survey asking performance on national workshops for concrete examples rated on a reported peer rubric) review results over By August 2012: time (approximately Involvement of Behaviour: New working practices 3 year cycle) education department adopted by participating Peer review reports stakeholders, and individuals and schools as a whole where at least two education agencies in (Survey asking for concrete are available for The positive and • What were the intended / unintended changes, that the coalition negative contributed to at different levels? changes • How have individuals and member organizations used the gains from the produced by the coalition meetings? coalition, • How many people have been affected? directly or o Participation in meetings / peer review / social media indirectly, o Number of children in Coalition schools, number of success cases in intended or Coalition schools unintended. POSSIBLE EVALUATION METHODS: Monitoring records on participation and school enrolment figures Social media monitoring (Twitter, Facebook, Bridge online) Interviews / Survey questions A consideration of the actual costs required to run the coalition in addition to the budgeted grant amounts, and an indication of the extent to which the coalition met timelines What financial and non-financial contributions did the coalition leverage for its work over time? (e.g. core donor funding versus inputs from schools vs. funding from other donors) In which ways did the coalition contribute towards using the financial and other contributions optimally (i.e. did the cost of participation go up or down over time?) Did the coalition meet key project milestones timeously and if not, why not? Were the meeting outputs made available to all participants timeously, and if not, why not? POSSIBLE EVALUATION METHODS: Cost analysis, Verification of milestone analysis Sustainability is concerned with measuring whether the benefits the coalition are likely to continue after donor funding has been withdrawn. • Is the coalition still valuable enough to keep participants involved? • Are the networks that were established of significance to sustain future maintenance of the coalition? • Does the coalition have a concrete plan to sustain its activities should current funding cease? (E.g. Are Network champions developed?) • Is the model of the coalition scalable / replicable? • What are the risks to sustainability? POSSIBLE EVALUATION METHODS: Stories / Survey, Key informant interviews » Questions or Inputs? The positive and • What were the intended / unintended changes, negative that the coalition contributed to at different changes produced by the levels? coalition, directly or • How have individuals and member organizations indirectly, used the gains from the coalition meetings? intended or • How many people have been affected? unintended. o Participation in meetings / peer review / social media o Number of children in Coalition schools, number of success cases in Coalition schools POSSIBLE EVALUATION METHODS: Monitoring records on participation and school enrolment figures Social media monitoring (Twitter, Facebook, Bridge online) Interviews / Survey questions » At the previous meeting, John asked for some examples of changes / impacts that the coalition members have noticed as a result of participating in the Coalition • The HODs that participated in the school peer review have come back with a direct leadership impact in instructional practice • Looking at the young leaders and young interns – what is possible in the future –intern programme and gift of transgenerational sharing • Follow-up on instructional leadership and personal management by two of the school leaders following the SPR • Learnt from the timetable structuring at Lebone (SPR) •Talking about classroom practice – practice of classroom writing – first 10 minutes in the classroom • Comment on student uniforms – looking at this as a change • Staff development To make sense of these stories, we need to know a few more things! » Curriculum change – we are embarking in 2013 on an integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11. » Curriculum change – we are embarking in 2013 on an integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11. » Curriculum change – we are embarking in 2013 on an integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11. » Curriculum change – we are embarking in 2013 on an integrated curriculum approach in gr 8. A team has met and redesigned our offering of 6 subjects. This is a direct result of seeing this operative at Lebone 11. » Impact: 1-5 Individuals, 6 – 30 Individuals, 31 – 100 individuals; Organizational; System Wide » Impact on Learners: Significant, Tenuous, Not Applicable » Impact on Teachers: Significant, Tenuous, Not Applicable » Change: Value; Attitude; Skill; Knowledge, Individual Behaviour, Organizational Behaviour » Verified: Yes / Partly / No / Not Yet » Does the template work? » Any suggestions? » Can you think of any evidence that relates to you, your colleagues or your organization that could help us to document examples of how the Coalition helped to: ˃ Build and improve the reflective practice capacity of individuals, schools and organizations ˃ Foster the adoption of new working practices by participating individuals, schools and organizations? ˃ Reduce the duplication of effort? ˃ Establish effective partnerships? ˃ Expand reach? ˃ Influence broad policy change? ˃ Support Coalition schools to maintain criteria for membership? ˃ Support Coalition schools to implementation of principles of excellent practice ? A consideration of the actual costs required to run the coalition in addition to the budgeted grant amounts, and an indication of the extent to which the coalition met timelines What financial and non-financial contributions did the coalition leverage for its work over time? (e.g. core donor funding versus inputs from schools vs. funding from other donors) In which ways did the coalition contribute towards using the financial and other contributions optimally (i.e. did the cost of participation go up or down over time?) Did the coalition meet key project milestones timeously and if not, why not? Were the meeting outputs made available to all participants timeously, and if not, why not? » Two Sources of Data – ˃ From the Coalition’s Financial Records + Good for calculating actual financial contributions / expenses + Not good for calculating in-kind contributions / expenses ˃ From the Coalition’s self-reported information + May be difficult to reconstruct for each event after the fact + Might be possible to come up with a model of “typical” cost contributions Meeting Expenses Venue Catering Office Supplies Travel Accommodation Flights Cost Estimates R 85 800 R 30 000 R 55 800 R R R 341 300 R 39 900 R 180 000 Claims (MSDF Contribution) R 30 645 In Kind Contribution R 55 155 R R 523 121 174 R R 220 126 R 863 679 R R 98 005 Other Donor Contributions (In Kind) Ground Transport R 121 400 Project Manager & Bridge Staff Other Professional Fees Overheads Special Projects R R R R - SUBTOTAL R For every One Rand the coalition contributed For Every One Rand the MSDF contributed - R 266 470 R 1 114 026 R 2.06 R 1 R 266 470 R 541 751 R 1.00 R 0.49 R 320 000 R R R 320 000 0.59 0.29 » Leverage (per year) » Cost per workshop participant per day (per year) » National Meetings ˃ ˃ ˃ ˃ ˃ Venue Facilitation Catering (x number of participants attending) Participant Accommodation (x number of staff attending) Participant Flights & Ground Travel (x number of staff attending) » Peer Review ˃ ˃ ˃ ˃ Reviewers Fees Catering Reviewer Accommodation Reviewer Flights & Ground Travel » Steering Committee ˃ Participant Fees » Typology ˃ ˃ ˃ Travel and accommodation at National Meetings (Might differ for meetings in own province / other province) + Level 1: Claims for most costs for one person, do not send more staff at own costs + Level 2: Claims for most costs for one person, sends one or more staff at own costs + Level 3: Do not claim any costs, one participant + Level 4: Do not claim any costs, sends 2 – 3 participants + Level 5: Do not claim any costs, sends more than 3 participants Hosts a meeting (National Meeting / Peer Review) and provides for most of the catering costs + Level 1: Provides Venue free of charge + Level 2: Provides Catering and Venue free of charge + Level 3: Provides Catering, Venue and Accommodation Member that actively contributes time to facilitate National Meetings / Participate as reviewers / lead reviewer + Level 1: Spends time facilitating at least one session at a National meeting + Level 2: Spends time participating in steering committee meetings and decision making + Level 3: Spends time participating as a reviewer in at least one peer review + Level 4: Spends time participating as a lead reviewer in at least one peer review + Level 5: Multiple engagement in various of the aforementioned activities » Is this a credible approach? » Any suggestions? 1 •FRAMEWORK 2 •IMPACT STORIES 3 •COST DATA 4 •OTHER? Benita Williams Feedback Research & Analytics 1102 Burnette Str. (Corner of Hilda), Hatfield, Pretoria Fax/Phone: (012) 430 2009 E-mail: [email protected] www.feedbackra.co.za
© Copyright 2026 Paperzz