Evaluation of the Save the
Children Response to
Cyclone Nargis
Andy Featherstone (Team leader, consultant), Kate Hart (Food & Livelihoods,
BRCS), Susan Thurstans (Nutrition, SCUK), Salai Isaac Khen (Civil Society,
Thingaha), Terry Durnian (Education, SC Sweden), U Khin Maung Lwin
(Accountability, Care) Dominique Porteaud (WASH, SCUK), Dipendra Shahi
(Health, MERLIN), Brigitte Goby (Child Protection, EMDH), Apple
Chaimontree (Child Participation, consultant)
April 2009
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | ii
Acknowledgements
The evaluation team would like to express their appreciation to all those interviewees who gave
their time to answer questions and participate in discussions, particularly the community members
living in the Ayeyarwady delta.
Thanks are also due to the Area Managers who put considerable effort into hosting the evaluation
team despite their own heavy workloads. The team are grateful to the many other Save the Children
staff that assisted with project visits, logistics and translation.
Dan Collison and Dave Hockaday championed the concept of opening up the Save the Children
programme to an unprecedented level of scrutiny and it’s the team’s sincere hope that the results
presented herein will justify the efforts it took to realise this vision.
Cover Photograph
Participants of the Save the Children child friendly space in Shwe Bo village, Mwa’gyun township,
February 2009
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | iii
Part 1: Evaluation of the Save the Children Response to Cyclone Nargis
Andy Featherstone, 04 April 2009
Executive Summary
Introduction
Cyclone Nargis struck Myanmar on 2 and 3 May 2008 with wind speeds up to 200Kph affecting 50
townships in Ayeyarwady and Yangon Divisions. The scale of human loss and suffering was vast with
nearly 140,000 people dead or missing and 2.4m people thought to be severely affected. UN
estimates suggest that the number of people displaced by the cyclone may have been as high as
800,000 with some 260,000 seeking refuge in temporary settlements in the immediate aftermath.
Methodology
The main objectives of the evaluation were, (i) to assess the quality and effectiveness and technical
strength of the response; (ii) to evaluate the extent to which the response has been accountable to
local communities; and (iii) to recommend improvements for the future of the programme focusing
on programme and management quality and accountability.
The evaluation team comprised 10 members spanning a diverse range of internal, peer agency and
consultant staff with specialism’s across all the sectors of the Save the Children response and the
cross-cutting themes of child participation, accountability and civil society. The team spent 16-days
in Myanmar and 11 of these in the delta visiting programmes, interviewing staff and conducting
participatory exercises with communities.
The Structure of the Report
This report is split into three parts; PART 1 comprises the main findings including the
recommendations. PART 2 of the report is a separate PDF document which contains the annexes.
PART 3 is an illustrated booklet of the results of the evaluation and will be shared with children and
villagers who participated in the Save the Children programme.
The main report has been divided into ten sections: SECTION 1 provides an introduction to the
evaluation and a contextual background. SECTION 2 introduces the objectives of the evaluation and
provides an overview of the methodology and limitations of the exercise. SECTIONs 3–9 provide
details of the main findings and recommendations for each of the evaluation criteria including a
section on critical decision-making. SECTION 10 concludes the evaluation report. Four stand-alone
‘case studies’ have been developed (on children’s feedback, accountability, civil society and village
committees) which are presented in text boxes in the narrative.
Summary of Recommendations
Timeliness
Save the Children scaled up most parts of the programme extremely quickly which is impressive
given the complex operating environment, the scale of the programme and the commitment to
delivering a multi-sectoral programme. That the response wasn’t informed by a signed-off
preparedness plan is testament to the industry and ambition of the team, but also highlights an
important area for action.
The large size of the programme is an important qualifying factor in any discussion about timeliness of
the programme, however Save the Children could have avoided some of the delays experienced across
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | iv
some of the sectors through better preparedness planning including a pre-agreed ‘package’ of possible
emergency interventions for each sector;
Specifically for nutrition, Save the Children should consider the development and implementation for key
staff of a preparedness package related to nutrition activities in emergencies inclusive of internationally
developed assessment tools. The development of this package could also include development of tools
for predicting caseloads of the carers of infants requiring support for breastfeeding or safe artificial
feeding;
Sectoral activities should be categorised by the impact they have on lives at the outset of a crisis and a
phased strategy of implementation agreed to ensure that the greatest possible number of lives are saved.
This is particularly pertinent for the WASH and nutrition sectors;
In such a rapid scale-up over such a large area, there are inevitable trade-offs between responding to
humanitarian need and ensuring that skills gaps are bridged. In making a judgment about the size and
scale of response, it will be important to be clear about the minimum necessary skills required to deliver
quality programmes and to phase programmes accordingly. Where a skills gap is anticipated or
discovered, it is recommended that either creative ways are quickly found to fill this or that programme
ambitions are reduced.
Coverage
Given the limitations in both the government and NGO response capacity, the decision to go to scale
was the right one but required the organisation to undertake an unprecedented recruitment drive
and work in sectors where it had very little prior experience. While strong co-ordination between
Alliance members greatly facilitated the swift scale-up, the lack of internal co-ordination between
different sectors of the response at a country level meant that opportunities for coherent
programming were missed.
The key challenge for the Save the Children Alliance is to seek to institutionalise as many of the elements
which facilitated the successful Nargis scale-up as possible to ensure a level of predictability across global
emergency responses in the future. While this won’t be possible for all aspects of the response, many can
be standardised through rigorous recruitment practices and step-aside policies, a strengthened
emergency architecture, a centralised standby capacity, a pool of funds set aside for under-writing
emergency programmes, and rigorous and feasible emergency preparedness plans. An organisational
culture which permits and supports risk-taking is also a prerequisite. Without doubt, the Nargis response
should be cause for optimism, but if predictability is the goal, then it will be important for the Alliance to
1
build on this successful model of collective action to further develop its humanitarian capacity .
To date there has been very limited integration of sectoral programmes. Looking to the transition
programme, it will be important consolidate/rationalise locations supported by Save the Children in
coordination with other agencies. It is suggested that a GIS mapping exercise across sectors may help to
support this process (if it has not already been done by OCHA). A Geographic Information System (GIS)
will visually identify the overlap/gaps between Save the Children sectors working in each region and
village and will enable a coordinated, integrated approach in villages selected for the next phase. It will
also help analyse Save the Children’s added value from working multi-sectorally;
A basic building block for the transition phase should be a holistic/cross-sectoral village-level assessment
which should form the basis of all future targeting and interventions;
Save the Children should review its approach to targeting villages for child protection and take a decision
on whether it should seek to support family tracing and reunification in all villages it has a presence in. If
not, at a minimum it should seek to conduct a rapid assessment and raise the issue in the appropriate
cluster group meeting.
Relevance
The evaluation found the programme to be relevant across all of the sectors. Over time,
programmes developed a sound understanding of the context and were effective in meeting the
basic needs of communities. The vibrant and informed discussion about the complexities of
targeting went a long way to ensuring the appropriateness of programme interventions.
1
ELT Operational Review of the SC Alliance Response to Cyclone Nargis, Andy Featherstone, October 2008.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | v
In order to ensure the best outcomes from its programmes, it is recommended that Save the Children
documents its experience of targeting to inform future programmes. A close working relationship with
the Accountability & Learning Working Group in Yangon will undoubtedly benefit the Save the Children
programmes as they are further refined in the transitional phase. The creation of a strategic oversight
role, responsible for working across sectors to identify common approaches and practices to community
engagement and targeting and the decentralisation of decision-making would likely also strengthen Save
the Children’s understanding of community dynamics and facilitate improved practice across the
programme;
The situation of extremely poor community members living in villages which are no longer receiving food
aid should be monitored (as a follow-up to the October/November needs assessment) lest there is a
significant drop in self-sufficiency of the caseload. This action should be co-ordinated at cluster-level;
A strong analysis and understanding of social networks and structures will be vital as the Livelihoods
programme moves in to the transitional phase to ensure that Save the Children reinforces traditional
networks and does not do harm. An understanding and identification of the small number of
marginalised groups that fall outside of existing social networks will also be important;
Save the Children should review the design of the latrines to ensure that they are accessible to children.
Where they are elevated off the ground, and a step has not been provided, teams should seek to provide
one or advocate for communities to make appropriate modifications;
Save the Children should seek to standardise their NFI distributions in the future – where it is not possible
to distribute the full package due to lack of stock, distribution teams should return to villages at an
appropriate time to ensure beneficiaries receive their full entitlement of NFIs;
The use of Waterguard as a means of providing potable water should be evaluated in more detail by Save
the Children with a view to understanding issues of relevance and uptake in order to inform future
practice;
The distribution of ceramic water filters in some locations was highly relevant but would benefit from
follow-up both by public health promotion teams (for upkeep of the filter) but also at a Yangon level with
WASH stakeholders, including the government to advocate for spare parts to be made available;
The education team should critically assess the relevance of the various components included in the
training programme offered to teaching staff to inform future practice;
It will be important in the future for Save the Children to reflect on ways that it can reach some of the
most vulnerable children, particularly those working full time who have been unable to attend the CFS.
As part of this, it should incorporate sensitisation on issues of child labour into ongoing CP programmes;
Further assessment and mapping of malnutrition and infant and young child feeding practices in Save the
Children catchment areas is recommended to guide appropriate programmes in addressing both the
consequences and underlying causes.
Technical Merit
While it’s difficult to make an accurate judgement on the quality of the programmes due to a lack of
baseline data and gaps in monitoring information, the programme is judged as being technically
sound and was largely successful in meeting SPHERE standards. In scaling up the programme to such
an extent, often in areas outside of the teams core competencies, programme size has at times been
given priority over programme quality, and there is an important need to ensure that the skills gaps
that remain are filled as the programme moves into the transition phase.
It is understandable that staff capacity is limited given the large number of people recruited into Save the
Children and the rapid scale-up, however, there is an important responsibility to find creative ways to fill
this gap so that all members are able to perform in their jobs and deliver quality programmes. From
November 2008 the expectation has been that 3% of every budget is put towards staff development. A
needs- based training and mentoring scheme should now be developed, with the support role of incountry advisors agreed across sectors;
For the reverse osmosis water treatment system, it is recommended that the design is reviewed and that
a non-continuous flow used in addition to treatment with aluminium sulphate (see WASH section for
explanation);
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | vi
The WASH team should ensure that a systematic water quality monitoring system is in place at each of
the different sources of water used by the community;
Save the Children should change mid upper-arm circumference (MUAC) cut offs to 115cm for SAM and
ensure WFH screening for at risk children.
Accountability
Against an accountability framework of (i) provision of information (ii) participation (iii) monitoring,
and (iv) feedback/complaints, the Save the Children programme scored mixed results. Programme
Staff worked hard to build links with communities and the development of village committees has
played a central role in the provision of information and the delivery of the programmes, albeit with
some limitations in the participation of the members. Monitoring systems do exist, but these have
historically prioritised the collection of quantitative data over qualitative data, have suffered from a
lack of co-ordination, and have often lacked sufficient resources to permit regular data collection.
That is not to say that good practice does not exist, and Save the Children’s willingness to host the
HAP/Sphere post and the establishment of an innovative accountability mechanism for the food
distribution programme demonstrates a commitment to strengthening accountability. It will be in
the ability of the team to continue to rationalise and strengthen systems and roll out good practice
to the broader programme that will see accountability ambitions achieved in the future.
As Save the Children moves away from emergency programming through transition to development
work, it will be essential that children are more involved in its programmes, not only by consulting with
them, but also through empowerment for children and advocacy to communities on meaningful
children’s participation;
An important step in this will be to ensure that children understand that Save the Children mandate and
mission. The translation of the Save the Children brand would be a good start to this;
A single monitoring system that can provide quantitative information for donors and senior managers in
addition to qualitative information for sector teams would be a significant achievement. A way to
accomplish this may be to develop standard operating procedures for monitoring systems and activities
across field offices and to agree monitoring principles (each team to monitor its own work or monitor its
peers in rotation - e.g. a monitoring team of 12 and each week 3 of the 12 staff undertakes monitoring
2
activities ). However, given the complexities of achieving this, a mapping exercise of the different
monitoring systems (sectoral and M&E team) linked to improved documentation and reporting
(particularly for the sectors) would be a good first step;
While the outcome matrix represents a significant step in the right direction and will greatly facilitate the
collection of qualitative data, it will be important to ensure it benefits from the full engagement of the
sector teams;
Full advantage should be taken of Save the Children’s hosting of the HAP/Sphere post to ensure that
knowledge of both initiatives goes beyond senior staff members and the M&E team to project level staff
tasked with implementation;
It is recommended that Save the Children review how they are calculating their beneficiary numbers and
refine their monitoring systems to ensure that a distinction is made between direct and indirect
beneficiaries;
For child protection particularly where the issues are complex and often new to communities, in addition
to working through established committees, Save the Children should seek to engage the community
more broadly (to the extent that local authorities permit this), to verify their understanding of the issues
and participation in the programme. The use of participatory tools would significantly strengthen this;
Approaches should be adopted by the nutrition programme which empower communities to adopt good
nutrition practices and prevent malnutrition
Effectiveness
2
Currently there is one M&E officer in each township, who, because of time pressures tends to collect ‘success
stories at the village level’.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | vii
In responding to the destruction wrought by cyclone Nargis on the lives and livelihoods of
communities living in the delta, Save the Children were successful in using in-country and
organisational capacity to deliver a proportionate response (meeting the needs of over 500,000
people) in an extremely challenging environment. In evaluating the achievements of the
programme, it is important to recognise the extraordinary vision and ambition that the team
demonstrated in going far beyond their comfort zone in establishing a geographically dispersed,
multi-sectoral programme. It is the heritage of these same decisions that has provided some of the
greatest challenges to the team; that of ensuring sound strategic planning, adequate quantity and
quality of staff, ensuring internal co-ordination and having sufficiently robust systems in place to
adequately monitor programme quality.
Given that livelihoods work to date has been conducted by teams from other sectors, if the decision is
made for the livelihoods programme to be progressed into longer term recovery, Save the Children will
need to carefully analyse and plan strategically the resources needed, including skill-sets of staff;
Ongoing development of staff capacity will be vital in the coming months as the livelihoods programme
moves away from distributions to a more ‘social development’/ community mobilisation role. Given that
teams are relatively inexperienced, ongoing training programmes/ mentoring of staff in the field should
be a priority and be needs-led. Rotation of staff between field offices to share knowledge and learning
and support from technical specialists is recommended;
Integration of (i) community management of acute malnutrition (CMAM) and infant and young child
feeding (IYCF), and (ii) integration of health and nutrition to form a comprehensive package of care that
addresses primary health care, treatment of acute malnutrition and infant and young child feeding needs
which can be linked with the work of other sectors in the overall country plan is recommended;
Design and implementation of an advocacy campaign, focusing on policies that affect nutritional status.
This should include promotion of optimal infant and young child feeding practices and incorporation of
community therapeutic care (CTC) protocols into integrated management of childhood illness (IMCI)
protocols. This campaign should also highlight the need for country wide nutrition surveillance to guide
nutrition and other sector interventions;
Save the Children should consider assessment of (if not already covered in survey) and expansion of CTC
programming to non CTC villages with high levels of Global Acute Malnutrition (GAM) in light of the
hunger season;
Save the Children should consider further assessment and mapping of malnutrition and infant and young
child feeing practices in Save the Children catchment areas to guide appropriate programmes in
addressing both the consequences and underlying causes of malnutrition;
Save the Children should advocate for and if possible gap fill for the treatment of malnutrition in
pregnant and lactating women;
As the emergency phase comes to an end, it is recommended that Save the Children develop phased
‘exit’ plans across all sectors for relevant villages and inform villages well in advance accordingly.
Critical Decision-Making
The context of decision-making, particularly in the first 3-months, was one of a dynamic
environment, where NGOs were working with very limited information in a catastrophe of
unprecedented proportions. Such an environment necessarily required risks to be taken and for
decisions to be made based on imperfect knowledge. 9-months on, there is far greater stability, and
while significant needs remain, Save the Children now have an opportunity to reflect on and make
changes to systems and structures in order to ensure they are fit for purpose for the future of the
programme. In particular, it is recommended that management structures are reviewed and
strenuous efforts are made to ensure internal cross-sectoral co-ordination.
While there was a draft emergency preparedness plan for Myanmar prior to the cyclone, this was
considered of limited practical value in guiding the humanitarian response. A document that provides
appropriate guidance to operational emergency response and which builds into it the changed scope and
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
3
Page | viii
nature of the Save Alliance programme and contextual changes as a result of improved humanitarian
3
access should be developed and signed-off .
While documentation about the transition programme suggests there has been some adjustment in the
focus of the programme to areas that are most in need, the relative fast pace of recovery in the east
compared to that of the west provides a strong justification to make more significant changes to the
geography of the Save the Children response to ensure that unmet needs in Middle Island and West Delta
are prioritised. This would also allow an opportunity for rationalising what is a geographically diverse
programme and by providing greater focus for the programme, could facilitate stronger sectoral coordination.
It is recommended that a skills audit is undertaken and that this coupled with a clearly articulated
transition programme strategy forms the basis of decision-making about staff numbers and about the
location for sectoral managers and advisory staff. The decision in December to create a small number of
field-based livelihoods advisors was a good one and it is recommended that this is replicated for other
sectors if the skills audit justifies this course of action.
The transition programme provides an opportunity to rationalise the management structure and it is the
view of the evaluation team that a structure that places line management, human resource management
and budget management in the hands of the area/regional managers would provide a much stronger
basis for effective programming;
One of the reasons given for maintaining the status quo was limited staff experience and capacity to
manage multi-million pound budgets. If capacity is considered to be a constraint, then this should be
dealt with through ensuring that sufficiently experienced international/national staffs are recruited into
field-based management posts;
If the status quo is maintained, it is strongly recommended that greater decision-making authority is
delegated down to field teams and that an explicit shift in the balance of power is made with
area/regional managers empowered to drive strategy. In order to facilitate this, it is recommended that
decision-making, communication, M&E and reporting mechanisms should be refined so that programme
planning, design and revision is routinely informed by the field and systems are established that ensure
two-way flow of information between Yangon and the area offices;
It will be important in the transition period and in future long-term interventions to develop integrated
programming. Management systems and mechanisms should be established to ensure cross-sectoral
analysis and planning (e.g. joint assessments, joint proposal writing, reporting formats across sectors to
encourage staff to work between and across sectors etc). Area/regional managers should be given
greater responsibility to facilitate/enforce cross-sectoral programming in the Areas under their
responsibility;
Save the Children should seek to develop a strong evidence-base for its future programme by carrying out
cross-sectoral assessments to inform future programming. It may wish to consider using a vulnerability
capacity assessment to frame the multi-sectoral assessments around disaster risk reduction (DRR)
(hazard mapping, vulnerabilities and capacities of communities, social organisation) as well as a
stakeholder analyses to identify key actors in need, of influence, and as potential support (civil society
groups, local authorities, other NGOs). Seasonal calendars could be developed for target areas to ensure
activities are implemented at the most appropriate time;
In moving forward it will be important to update the existing emergency planning document and develop
a coherent strategy that has a cross-sectoral vision and which plots a path through transition to longerterm development (where this is appropriate). It will be important to ensure this plan is coordinated in
particular with the development of Save the Children’s DRR strategy and plan;
In progressing the proposed disaster risk reduction programme, it will be important to ensure that
lessons are learnt from the Nargis response and that the basic concepts of DRR are understood by the
Save the Children staff and that the proposed programme cuts across sectors and builds on the strong
community links that have been established.
ELT Operational Review of the SC Alliance Response to Cyclone Nargis, Andy Featherstone, October 2008.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | ix
Table of Contents
PART 1: The Report
Acknowledgements........................................................................................................................... ii
Cover Photograph.............................................................................................................................. ii
Executive Summary............................................................................................................................iii
Table of Contents.............................................................................................................................. ix
Acronyms........................................................................................................................................... xi
Myanmar: Cyclone Nargis – Humanitarian Profile (July 2008).......................................................... xii
1. Introduction.................................................................................................................................. 1
1.1 The Impact of the Cyclone on Life, Livelihoods and Vulnerability .............................................. 1
1.2 The Humanitarian Response to the Cyclone............................................................................... 1
1.3 The Save the Children Response.................................................................................................2
2. Scope of the Review & Methodology............................................................................................ 2
2.1 The Scope of the Review............................................................................................................. 3
2.2 Methodology............................................................................................................................... 3
2.3 The Voice of Project Participants................................................................................................. 3
2.4 Limitations................................................................................................................................... 4
CASE STUDY 1: Children’s Feedback..................................................................................................5
3. Timeliness ..................................................................................................................................... 8
3.1 Main Findings.............................................................................................................................. 9
3.2 Recommendations....................................................................................................................... 12
4. Coverage....................................................................................................................................... 12
4.1 Main Findings.............................................................................................................................. 12
4.2 Recommendations...................................................................................................................... 16
CASE STUDY 2: Save the Children engagement with civil society organisations............................... 16
5. Relevance..................................................................................................................................... 18
5.1 Main Findings.............................................................................................................................. 18
5.2 Recommendations...................................................................................................................... 23
CASE STUDY 3: Save the Children and village committees................................................................ 24
6. Technical Merit............................................................................................................................. 25
6.1 Main Findings.............................................................................................................................. 25
6.2 Recommendations...................................................................................................................... 29
7. Accountability............................................................................................................................... 29
7.1 Main Findings.............................................................................................................................. 30
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | x
7.2 Recommendations....................................................................................................................... 35
CASE STUDY 4: Evaluation Against HAP Accountability Principles....................................................35
8. Effectiveness................................................................................................................................. 36
8.1 Main Findings.............................................................................................................................. 37
8.2 Recommendations...................................................................................................................... 42
9. Critical Decision-making................................................................................................................ 42
9.1 Main Findings.............................................................................................................................. 43
9.2 Recommendations...................................................................................................................... 45
10. Concluding Comments.................................................................................................................46
PART 2: The Annexes (separate PDF document)
Annex 1 Evaluation Participants........................................................................................................47
Annex 2 Terms of Reference for the Evaluation................................................................................ 49
Annex 3 Methodology Matrix........................................................................................................... 56
Annex 4 Children’s Suggestions.........................................................................................................60
Annex 5: Timeline of Significant Events that Shaped the Nargis Response.......................................62
PART 3: Children’s Report (separate booklet)
This booklet is part of the evaluation of Save the Children's cyclone Nargis emergency response and
will be shared with children and villagers who are beneficiaries of Save the Children's programmes to
inform them about the results of the evaluation.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Acronyms
ACF
CBO
CHANT
CLDRR
CMAM
CFW
CP
CFS
CTC
DoH
DRR
ECCD
EMDH
FTR
GAM
GIS
HAP
IFE
IMCI
INEE
IYCF
M&E
MERLIN
MUAC
NFI
NGO
OCHA
PONJA
PONREPP
PRA
QIP
RO
SAM
SCUK
SC Sweden
TCG
UN
Unicef
WASH
WFH
WHO
WFP
WTP
Action Against Hunger
Community Based Organisation
Community Health and Nutrition Team
Child Led Disaster Risk Reduction
Community Management of Acute Malnutrition
Cash for Work
Child Protection
Child Friendly Space
Community Therapeutic Care
Department of Health
Disaster Risk Reduction
Early Childhood Care & Development Centre
Children of the World
Family Tracing & Reunification
Global Acute Malnutrition
Geographic Information System
Humanitarian Accountability Project
Infant Feeding in Emergencies
Integrated Management of Childhood Illness
Inter-agency Network for Education in Emergencies
Infant & Young Child Feeding
Monitoring & Evaluation
Medical Emergency Relief International
Mid-upper Arm Circumference
Non Food Item
Non Governmental Organisation
United Nations Office for the Co-ordination of Humanitarian Affairs
Post-Nargis Joint Assessment
Post Nargis Recovery & Preparedness Plan
Participatory Rural Appraisal
Quick Impact Project
Reverse Osmosis
Severe Acute Malnutrition
Save the Children UK
Save the Children Sweden
Tripartite Core Group
United Nations
United Nations Children’s Fund
Water, Sanitation, Hygiene Promotion
Weight for Height
World health Organisation
World Food Programme
Water Treatment Plant
Page | xi
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Attribution: Map provided courtesy of the ReliefWeb Map Centre, UN Office for the Coordination of Humanitarian Affairs
Disclaimer: The boundaries and names shown and the designations used on this map do not imply official endorsement or acceptance by the United Nations
Page | xii
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 1
1. Introduction
1.1 The Impact of the Cyclone on Life, Livelihoods and Vulnerability
Cyclone Nargis struck Myanmar on 2-3 May 2008, causing devastation in Ayeyarwady and Yangon
Divisions. The scale of human loss and suffering was vast with nearly 140,000 people dead or missing
and 2.4m people thought to be severely affected. Nargis is considered to be the worst natural
disaster in the history of Myanmar with the area affected being remote, heavily populated and
difficult to access. UN estimates suggest that the number of people displaced by the cyclone may
have been as high as 800,000 with some 260,000 people having resided in camps or settlements
throughout the delta in the initial weeks following the cyclone.
Damage to Infrastructure: The Post Nargis Joint Needs Assessment (PoNJA)4 indicated that more
than 500,000 homes were severely damaged or destroyed by the cyclone; 99% of which were made
of bamboo and wood. Vital community-level infrastructure was severely damaged or destroyed:
72% of village’s required urgent rehabilitation of schools and educational facilities, and 32% of
villages required restoration of health facilities. Households also lost vital assets such as cooking
equipment, clothing, and animals. Ponds and wells were flooded by salt water and were unusable
with 60% of people relying on rainwater for drinking purposes.
Damage to Food Stocks: Over 780,000 ha of paddy fields were flooded by the cyclone, and 75 to 80%
of seed stocks from the cyclone-affected area were destroyed by flooding or subsequent rain. In
addition, the loss of draught animals and power tillers means that an estimated 52,000 farmers were
unable to plant monsoon rice between June and August 2008.
The Impact on Livelihoods: The majority of the population in the disaster-affected area was already
poor pre-cyclone Nargis with many facing serious threats to their livelihoods and having limited
access to basic services. The cyclone severely compounded these vulnerabilities. Affected
populations in the rural areas of southern townships of Yangon and the entire affected areas in the
Delta sustained severe damage and destruction to their agricultural livelihoods5. Data from the
PoNJA indicates that 42% of villages have identified the loss of livelihoods as their primary early
recovery challenge.
The Social Impact: Social cohesion within the Delta region is considered to be high and the
immediate impact of the cyclone appeared to strengthen rather than diminish this. There were no
visible tensions between ethnic and religious groups noted and the outpouring of assistance from all
faiths was considered to be a unifying factor. On this issue of cohesion, the PoNJA does provide a
warning to aid providers - while the response to the cyclone has the potential to change life in Delta
villages for the better, it noted that it also has the potential to result in negative consequences and
highlighted the need to build in measures to address low absorptive capacity, in particular through
community capacity building, as well as progress monitoring mechanisms.
The Impact on Vulnerability: The PoNJA also showed that the scope and devastation of the cyclone
had rendered women and children particularly vulnerable to abuse, exploitation, violence and
neglect. Children separated from their families, child-headed households, children with disabilities,
and other vulnerable children were considered to be in particular need of special support.
1.2 The Humanitarian Response to the Cyclone
4
The Post-Nargis Joint Needs Assessment, Tripartite Core Group, July 2008.
Early Recovery Strategy, A Community Driven Approach to Start Restoring Life and Livelihoods in Myanmar
Post Cyclone-Nargis, The Early Recovery Cluster and Network, Myanmar, June 2008.
5
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 2
The national response started quickly with assessments in some parts of Yangon occurring within 24hours by national Non-governmental Agencies (NGOs) and by International Non-governmental
Agencies (INGOs) and the Myanmar Red Cross Society (MRCS) using in-country capacity. Agencies
with established programmes in the delta launched small-scale responses using contingency stocks
built up as part of preparedness measures, however the scale of the disaster meant that needs went
far beyond what national capacity could provide.
The authorities had indicated informally as early as 04 May that they would be willing to receive
international assistance and this was formalised in a briefing to the United Nations (UN) and
Diplomats on 05 May, but the focus was on supporting national capacity and there were no
significant steps taken to facilitate international entry of humanitarian staff to Myanmar, or to relax
travel restrictions for international staff until much later in the month.
The impact of this was to place the response squarely in the hands of Myanmar nationals who didn’t
face the same restrictions. The lack of such key staff led to ‘critical delays in getting reliable data on
needs, in strengthening coordination and response capacity and in establishing the logistical systems
necessary to move large quantities of relief supplies quickly and efficiently to affected areas’6. It also
meant that local staff on the ground had to work for extended periods without rotation and often
with insufficient organisational support.
While access did improve following the visit of the UN Secretary General on 23 May and the
establishment of the Tripartite Core Group (TCG), it took time to bolster the capacity of international
organisations responding to the crisis and to provide much-needed support and technical expertise.
The view of the OCHA-commissioned Inter-Agency Real Time Evaluation of the Nargis Response, is
that ‘while this played a part in decreasing coverage of affected areas and probably prolonged
suffering in some communities...the team saw no evidence from agency reports/surveys or
interviews that there has been any epidemic or increase in acute malnutrition from pre-cyclone
levels’7 which may be considered testament both to the resilience of communities in the delta but
also of the tremendous efforts made by pre-existing agency staff, civil society groups and generous
individuals.
1.3 The Save the Children Response8
Save the Children has been operational in Myanmar for the last 13 years and prior to the cyclone
had a sizeable country presence (although not in the Delta area) employing around 500 staff with a
projected budget of £4.25m for the financial year 08/09. In January 2009, eight months after the
cyclone, Save the Children employed over 1400 staff, working in 14 out of the 15 most affected
townships in the delta with an emergency response budget of approximately £26m spanning a seven
sector response including shelter, food security, livelihoods, WASH, nutrition, child protection and
education. The scale up to the response was massive and the size, scale and programmatic scope of
the emergency programme that emerged had not been seen in Save the Children since the response
to the Boxing Day Indian Ocean tsunami of 2005.
2. The Scope of the Evaluation & Methodology
6
Negotiating Humanitarian Access to Cyclone-affected areas of Myanmar: A Review, J Belanger & R Horsey in
Humanitarian Exchange, Issue #41, Humanitarian Practice Network, ODI, December 2008.
7
Inter-Agency Real Time Evaluation of the Response to Cyclone Nargis, R Turner, J Baker, Dr. Zaw Myo Oo,
naing Soe Aye, UN OCHA, December 2008.
8
See Annex 5 for a timeline of significant events that shaped the Save the Children response to the Nargis
Cyclone.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 3
2.1 The Scope of the Evaluation
The objectives of the evaluation were as follows;
Assessing the quality and effectiveness and technical strength of Save the Children in
Myanmar’s Emergency Response to cyclone Nargis between May 2008 and January 2009;
Measure the extent to which the response has been accountable to the local needs (of children
and their families);
Recommend improvements for the longer-term strategies of the response programmes
focusing on programme and management quality and accountability.
These objectives were broken down into 7 key questions which are detailed in the ToR which is
reproduced in Annex 2.
2.2 Methodology
In order to frame the evaluation and to guide the work of the evaluation team, a methodology matrix
was developed9 which assigned a benchmark or standard which placed each of the criteria in the
organisational or sectoral context (see below), and provided a list of questions, a menu of methods
and a guide to possible sources of information for each criterion.
Criterion
Effectiveness
Timeliness
Coverage
Relevance
Description
The extent to which the activities achieved their purpose or whether this can be expected to
happen on the basis of the outputs. Implicit within this criterion is timeliness
The extent to which the programme was delivered according to its timetable and the extent
to which this appropriate to the needs of beneficiaries
The overall size of the programme relative to needs taking into account other agency
programmes and gaps in coverage
The extent to which the programme activities are in line with local needs and priorities (as
well as donor policy)
Strengths and weaknesses of response across all technical areas
Technical
Merit
Connectedness The extent to which the activities of the emergency programme are carried out in a context
that takes longer-term and issues of programme integration into account
Accountability Upwards & downwards accountability: provision of information, communication,
participation, reporting, decision-making
In the case of Save the Children, the most appropriate standards/benchmarks were considered to
include the Code of conduct for international RCRC Movement and NGOs in disaster response
programmes10; the Early Recovery Cluster & Network in Myanmar, in which Save the Children
participated, developed a strategy in June 2008 to guide the work of the humanitarian community
which included some broad principles which could also be considered as helpful benchmarks11; the
evaluation ToR makes specific reference to Sphere minimum standards and the HAP accountability
principles; and internal standards such as the Save the Children Alliance Child Participation
standards.
9
See annex 4 for the Methodology Matrix.
For a fuller discussion of how the DAC criteria relate to the RCRC CoC please refer to ALNAP/Channel
Research, An Introduction to Evaluation of Humanitarian Action: A Course Manual, Margie Buchanan-Smith &
John Telford.
11
See Early Recovery strategy: A Community-driven approach to begin restoring lives and livelihoods in
Myanmar post-Cyclone Nargis, The Early Recovery Cluster and Network, Myanmar, June 2008.
10
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 4
In approaching the evaluation, Save the Children sought to mandate a broad-based evaluation team
composed of Save the Children Alliance and peer agencies which included specialism’s in food
security and livelihoods, health & nutrition, WASH, child protection, child participation, education,
accountability and civil society. The team included members with international and national
experience and drew from a broad base of agencies (Care, British Red Cross, SC UK, SC Sweden,
MERLIN, Thingaha, and EMDH).
The schedule for the review was as follows12;
Preparation in Yangon for the evaluation leader and child participation member (3-days);
Interviews of Yangon-based staff (1.5-days);
Field trip to Middle Island, East Delta and West Delta (11-days);
Write-up and debrief in Yangon (1-day);
The review team were given access to several evaluative reports; the ELT Operational Review of the
Save the Children Alliance Nargis response, the 2 Lessons Learnt documents (Yangon and London
Emergencies team), the review of the IFE programme, and the post-distribution monitoring report
for the NFI and shelter distributions. This report will use appropriate findings from these although
will seek to avoid unnecessary duplication.
2.3 The Voice of Project Participants
A strong focus of the evaluation was to elicit feedback on the quality and effectiveness of the
programme from project participants and the schedule for the itinerary underlined this, with a
significant proportion of the time spent conducting field work in Save the Children programme areas.
A range of methods were used to capture the views of project participants which included key
informant interviews, focus group discussions, and PRA techniques.
Picture 1: Body map exercise with school children, Middle Island
12
See annex 1 for a list of persons interviewed and locations visited.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 5
Children were identified as primary stakeholders for the evaluation and a child participation
consultant was included in the team specifically to support design of appropriate methodology and
to undertake 2 stand-alone children’s feedback workshops. Techniques used to capture the views of
children included the ‘H’ assessment and body mapping. During the evaluation, efforts were taken to
ensure that different sectoral evaluators (child protection, WASH, Food and livelihoods, nutrition,
education, health) and thematic members of the team (civil society, child participation,
accountability) worked together to ensure cross-sectoral linkages were made. Informal daily debriefs
and more structured weekly feedback sessions ensured information was captured, gaps were
identified and that learning was shared amongst the team.
2.4 Limitations
While the focus on field work was an important one, it meant that time spent with staff in Yangon
was limited to just over a day for the majority of the team which allowed for limited engagement on
strategic issues. There was a general lack of programme reports and strategies made available to the
team which complicated the task of establishing programme goals. In particular, documents about
the transition programme were incomplete and obtained late in the evaluation which complicated
the task of making recommendations for the future of the programme. Documents about the longerterm programme were similarly received late in the evaluation and for this reason it has not been
possible to fully evaluate connectedness as a discrete criterion.
As a result of the large size of the programme, the throughput of staff, and the lack of written
information, it has not been possible for the team to evaluate some aspects of the programme,
specifically the health intervention with Mercy Malaysia for which there was an absence of data; the
information centre in Middle Island was not included in the itinerary and hence information about
this was limited to a discussion with advisors in Yangon; and there was no time scheduled for
discussions with external agency staff involved in cluster groups and hence this was also omitted
from the evaluation.
That said, the team met a wide range of field staff, visited a representative sample of area offices and
had unprecedented access to communities who had participated in Save the Children programme
work during the Nargis response which provided a strong foundation for observation of programme
work and community discussions about its effectiveness.
Case Study 1: Children’s Feedback
While children’s participation was mainstreamed across the evaluation, two stand-alone children’s
feedback workshops were held to encourage creative expression by children using child friendly
participatory tools. The preparation and training for these was undertaken by a Save the Children
staff member with the sessions being overseen by the child participation evaluator. The findings
below reflect the substantive feedback from children on the Save the Children programme and these
also form the basis of a children’s feedback report13 which will be distributed to children and adults
who have been involved in the programme.
In general, children are happy and feel safer with the Save the Children presence in their village.
Many of Save the Children’s responses brought new facilities, knowledge, and attitudes to the
affected communities, which they had never had even before the cyclone. Amongst the different
activities, the CFS is recognised and praised the most by children. Children said the singing and
playing in the CFS helped them forget about what happened during Nargis.
13
Children’s report, Apple Chaimontree, April 2008
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 6
Child Protection Programme
When asked to identify Save the Children activities in their communities, children always came up
with the CFS first. It is a ‘space’ that they never had before in their communities, which they can
meet up with other children and CFS teachers (whom they call ‘animators’). Parents also gave
positive feedback about the CFS saying that they were less worried about the whereabouts of their
children as there were CFS teachers who took care of them.
Children said they enjoy singing songs and poems, playing, and listening to stories told by CFS
teachers. They would always come to the CFS whenever they were free from classes and/or work.
They said they learned new knowledge from songs, poems, and stories they heard from the CFS.
They learned how to keep themselves clean and prevent diseases by washing their hands and feet
before eating and going to bed and after using toilets; and by using mosquito nets when they sleep.
They learned about children’s rights and how to keep themselves safe and away from dangers,
although the majority could not explain about their understanding on rights. Only a 14-year-old boy
from a village in Middle Island could give a simple example about children rights and protection.
“ … giving an example, if my mum is cooking and my younger sibling who is a baby is
sleeping, then my mum call for me to help her and in the same time the baby cries,
before I would have gone to help my mum. But now, I’ll run to the baby because I
know that the baby can’t help himself and need more help than my mum.”
The issue of violence against children came up when asking children about their relationship with
adults, as well as in the children’s feedback workshops. Feedback on violence against children varied
in different locations. In a village in Middle Island, children said that parents and teachers did not hit
them anymore because they were now aware of child protection issues. Yet, in a group discussion
with 10 children aged between 6-10 years old in a village in Labutta, children said their teachers hit
them with bamboo sticks and one of their friends still had a wound from being beaten. Children in
other communities did not raise issues of violence but said that their relationship with adults was no
different from the time before Save the Children entered their communities. Some children said they
still run to their parents and teachers when they had problems, and they could talk and the adults
would listen to them. Children participating in the feedback workshops said adults talked to them
more kindly after they learnt about child rights and child protection.
When asked to give suggestions about the activities, children said that they wanted the CFS to
continue14 and since there closure they had missed the CFS. Several commented on the structure of
several CFS’, that they were not strong and safe. Children said that the rain came through the roof of
the CFS which was made from thatch. Some children said the floor of the CFS, which was made of
bamboo, got damaged after sometime because there were too many children playing there. A 6year-old boy in a village in Labutta was quite badly injured because his leg fell through one of the
holes in the floor and got stuck. He said his friends helped him out and he didn’t report the accident
to the animators.
Children also said there were children who didn’t come to the CFS because they had to work, and
not every child received toys distributed by Save the Children.
Education Programme
While the CFS gave a new space and new experiences to children, recovery of schools helped
children feel that their lives had returned to normal. Most of children said several stakeholders,
14
Save the Children phased out the CFS between December 2008 – January 2009. Only some communities
were able to carry on the CFS by mobilising local resources.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 7
including local private donors, Save the Children and other organizations helped rebuild their schools
soon after Nargis. In a village in Mawlamyinegyun where there were only 35-37 children in school,
they said they went to study at their teacher’s house while their school was being rebuilt.
Children said they were happy to have received school kits (e.g. school bags and stationery) and
blackboards. They were also happy about the new schools they had15 as they were bigger than the
previous ones. However, some children were concerned that the new schools would not be strong
enough to withstand another cyclone and would be destroyed like the previous ones had been.
Some said that rains came through the tarpaulin roof and classroom windows, which were not fixed
properly. The classroom floors in several schools also needed to be improved because a floor hadn’t
been fitted and the classrooms were built on the dusty earth.
Children said that their teachers helped comfort and make them feel better after the Nargis. Yet,
corporal punishment was still reported in schools. They also gave positive feedback and were
supportive about the establishment of the ECCD centres. Nevertheless, in places where the CFS was
planned to be transformed into the ECCD centre and the CFS was to be closed, older children were
questioning whether they would be allowed to play in the ECCD’s as they were in the CFS’.
Food, Shelter, and Livelihood Programmes
It is clear from group discussions and the feedback workshops that for many children their family’s
livelihoods changed after the Nargis. However, it was not always easy to know whether the changes
were as a result of Save the Children interventions or other agency/donor action. From discussions
with out-of-school children who helped their parents to work, the feedback varied. A group of
working children aged between 11-16 years in a fishing village in Labutta who bought their new
fishing nets with Save the Children’s QIP grant said they worked less hours after Nargis because
there were less people going back to fish. As a result, they could catch more fish and did not have to
spend such a long time in the delta. Another group of school children in a village in Labutta said their
parents had more difficulty working because of the emotional affect Nargis had on them. They said
people were still sad and didn’t want to go to work. In another village, again in Labutta, four working
children said, despite the cash grant, they had to work more after the cyclone because, not only did
they need to earn for normal household expenses, but they also needed to save money to repair
their houses before the arrival of the rainy season.
Nevertheless, there are a few cases where Save the Children livelihoods interventions have had a
direct positive impact on the lives of the children and their families. A 12-year-old girl from a village
in Mawlamyinegyun explained how her family used Save the Children cash grant. She said she lived
with her grandparents because her mother remarried and lived with her new family nearby. Her
grandparents received the 50,000-Kyat cash grant and so did the family of her mother. Therefore,
altogether the two families received 100,000 Kyats. She said her grandparents and her mother put
the money together to buy a big boat. They rent the boat out and share the benefits between them.
The boat rent earned them more than the farm work that they used to do before Nargis. Later on,
they used the money they earned to buy a pig and twenty ducks. Another child from a focus group
discussion in a village in Labutta said
“Distribution of money is very useful because we can buy oil, salt, chili, onions, and
house materials. My elder brother can go (migrate for work) to Yangon by this
money. My father can repay our debt. This money is very useful to our family.”
15
In some communities, children still study in temporary classrooms with tarpaulin-sheet roof, while waiting
for the permanent school to be built. In some communities where schools were partially damaged, children
said Save the Children and other organisations provided them with construction materials for the roof and
windows to fix their schools.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 8
Children also said that the distribution of food and non-food items helped their families save some
money. Children reported they have received rice, oil, salts, peanuts, chili etc. regularly from local
private donors, Save the Children, and other organizations. The frequency of distribution was
different from one community to another based on the community’s needs and the donors’
resources. However, as mentioned earlier, some families still need to earn more money to fix their
houses, which are still partly covered with tarpaulin sheets 9-months after the cyclone. Children said
that after Nargis prices of housing materials increased which made it difficult for some households
to afford to fix their houses.
Health, Nutrition, and WASH Programmes
The impact of the health, nutrition, and WASH programmes are related. The main impact is
children’s awareness on how to keep themselves and other children healthy and avoid sickness.
Children said they learned to wash their hands and feet before eating and going to bed and after
using toilets, and to use mosquito nets when they sleep. From both of the feedback workshops,
participating children expressed their concerns about babies who may have lost their mothers in the
cyclone as there would be no one to breast-feed them. They also mentioned that babies should be
weighed regularly and not be given supplementary food if they are under 6-months.
Children mentioned specifically about how useful the Save the Children-built latrines in their
communities were. They said there were very few latrines before Nargis but they now have more
and better latrines some of which are fly-proofed and equipped with nets to prevent flies and insects
going into the latrines. In a few communities, children also reported that there have been no
diarrhea cases since the new latrines were built.
“There is no diarrhea case because of the latrine construction. Those latrines are
useful for every family. Before, we went for toilet on the bank of the river. Now we
use fly-proof latrines. We pack baby’s waste with paper and throw it into the flyproof latrine. Thank you so much for building them. We are very happy.”
- 14-year-old girl from a village in Pyapon
Children reported a few problems with the community latrines that because they were not built for
every household16; they were sometimes yelled at when using the latrine by the household on
whose land it was constructed. For this reason, they said they wanted their own latrine for their
family. According to the children, the household nearest to the latrine is responsible for taking care
of it. Despite the distribution of soap and water containers and the hygiene promotion, children
reported that some people in their communities still do not use the latrines properly.
In communities where Save the Children helped to rehabilitate local ponds and build water pumps
children also gave very positive feedback. Children said they appreciated having clean water for
drinking and cooking. Some even said that the water tasted better. Where ponds had been
protected, they were grateful that no animals could get into them and make them dirty anymore.
With the water pumps, they no longer had to go into the ponds to fetch water.
3. Timeliness
16
Community latrines were built based on the ratio of 20 people to one latrine.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 9
Summary: Save the Children scaled up most parts of the programme extremely quickly which is
impressive given the complex operating environment, the scale of the programme and the
commitment to delivering a multi-sectoral programme. That the response wasn’t informed by a
signed-off preparedness plan is testament to the industry and ambition of the team, but also
highlights an important area for action.
Picture 2: Consolidated Save the Children Nargis response timeline, Yangon
3.1 Main Findings
Food & Livelihoods: The speed of Save the Children’s scale up in its food aid programme is
impressive. Despite the availability of limited documentation to compare planned and actual food
distribution times, food was distributed from the second day after the cyclone and food distributions
commenced from June onwards. Save the Children recognised that Food and Livelihoods was a key
priority from an organisational perspective so it re-organised resources, transferring staff who
usually worked in other sectors to work on Livelihoods for varying periods of time which means the
team was relatively inexperienced. Given this, the initial food and livelihoods response is a
remarkable testament to the dedication and endeavour of all staff, and national staff in particular.
The Quick Impact Project (QIP) was proposed by visiting senior staff in July 2008 after they identified
the urgent need to support early livelihood recovery. Some staff felt this represented a failure of
timeliness, however in the immediate 2-month period post- Nargis local markets were disrupted,
with many productive assets not in stock in local shops; local stocks of fishing nets had been
destroyed during the cyclone and it took some time to re-stock from Thailand17; livestock markets in
the region were also destroyed and hence Implementation of the QIP may be considered to have
been timely. Middle Island represented the quickest response where Save the Children took the
courageous and high risk decision to distribute cash over a 4-day period in mid August18.
17
Livelihoods Rapid Needs Assessment in Post-Nargis, Myanmar, November 2008.
Myanmar authorities prohibited cash distributions initially but now accept them if they are for a specific
purpose/ asset replacement e.g. shelter.
18
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 10
Implementation of the QIP in other areas was delayed as a result of a decision not to distribute cash
but to pursue a facilitated procurement process which took time to agree, but the programme was
still well-received. In both Labutta and Maw’Gyung teams from across sectors implemented the
project as part of their ongoing jobs, which might have contributed to the delay.
The impact of other agencies’ working practices on the outcomes of Save the Children’s
interventions is important to note. In Middle Island, where there are much fewer actors because of
its remote location, Save the Children is often the only organisation present in villages and so
appears to be generally taking a multi-sectoral approach. It is responsible for both food and
livelihood programmes in most villages and has ensured that food is generally delivered on time. In
other areas where more organisations are present, Save the Children has been coordinating its
interventions with them. The evaluation team noted that a delay in food distributions of another
agency sometimes adversely impacted on the Save the Children livelihood interventions (e.g. some
cash from the QIP being used by households to cover food needs).
The cash for work (CFW) project has only commenced in Middle Island and was suspended
temporarily in December due to procedural irregularities. In Middle Island there have been delays in
procurement of up to 6-weeks but careful guidance has now been established by the logistics team
which should ensure that planned projects in Labutta can commence without such delays.
Nutrition: The initial consolidated rapid assessments made on May 18 followed by the arrival of
technical support and activation of clusters and working groups, development and translation of
statements and guidance, and training of trainers indicate a timely response given the context. It
must be noted however that the absence of emergency preparedness for Infant Feeding in
Emergencies (IFE) programme and lack of pre-designed programming models or tools available, may
have impacted negatively on the speed of the response. Additionally some activities such as staff
sensitisation, procurement, formula distribution and achievement of advocacy outputs were not
effectively carried out or were relatively slow to be achieved and in some areas initiation of projects
were later than others. In the future it will be important to recognise the immediate life saving
nature of the programme and to ensure that all parts of the programme can be delivered in the
quickest possible time.
The Community Therapeutic Care (CTC) response was slower than IFE in some areas due to several
factors which include difficulties in recruitment, the length of time needed for training and decision
to deploy staff to other offices. Once again the lack of preparedness (previous nutrition activities
had focused largely on positive deviance approaches) may have contributed to the slower response
time. Given the pre-existing high levels of malnutrition, staffing of CTC should have been the priority.
Community mobilisation did not start until September due to delays in the recruitment of
community health promoters by the WASH team and the prioritization of WASH-related activities
over health and nutrition. This will have had serious impacts on the timely delivery of nutritional
care19 to malnourished children given communities’ lack of recognition and understanding of
malnutrition pre Nargis20.
Health: The mobile medical team was started 2-weeks after the cyclone which may be considered
timely given the context. During the cyclone, many of the pre-existing health facilities (both public
and private) were destroyed and essential drugs were in short supply and hence there was initially a
huge need for health care services. The use of mobile teams visiting villages on a weekly basis meant
19
A Review of Save the Children’s Cyclone Nargis (Myanmar) Infant Feeding in Emergencies response:
September 15th-26th 2008, Victoria Sibson (Save the Children Emergency Nutrition Adviser) and Bienfait
M’mbakwa (Save the Children Emergency Response Nutritionist)
20
Nutrition Survey, Save the Children, 2007
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 11
that access to services was generally good. The fixed health facilities managed and run by the
Department of Health (DoH) started to become operational two weeks after the cyclone and these
were used by communities when mobile teams were elsewhere.
WASH/NFI: While early distributions of bottled water and jerry cans and the establishment of the
first water treatment plant (WTP) on June 13 may be considered to be relatively timely, considering
the life-saving nature of potable water provision, Save the Children was relatively slow to then scale
up its programme to meet needs. In most of the areas visited, watsan activities didn’t start until 3months after the cyclone and as such cannot be considered to be life-saving. It is important to note
that there were 2 water crises, the initial one which followed the cyclone and a second which was
more predictable and which followed the onset of the dry season when access to potable water for
many communities living in the delta declines significantly. For this, Save the Children were far
better prepared, however, the technical solution chosen, the reverse osmosis plant was being
installed at the time of the evaluation and as such is late. The distribution of tarpaulin to use as
guttering and water collection containers may be considered an appropriate method of rainwater
collection, however, timeliness was patchy – Save the Children project documents show that 2,721
community and household rainwater tanks were constructed before the end of the rains, however
some of the villages visited reported that they received the items after the rains had stopped
(Labutta and Middle Island in particular). Similar to water, the sanitation programme was also
relatively slow to start with the first latrine construction occurring in July and programme activities
still ongoing.
The villages visited in Middle Island received WASH non food items (NFI) in October which is late; for
Labutta and Maw’gyun distributions took place between August 2008 and January 2009. Distribution
of essential items like water containers, mosquito nets and soap should have taken place earlier,
particularly given that some communities visited received water containers after the rains had
finished, significantly reducing their usefulness. Conversely, those that did receive them on time
commented on their great usefulness.
Education: Overall Save the Children’s interventions were timely within the context of the area that
they were working. In most places education interventions began in either June or July with the
exception of Labutta. Discussions with education committee members suggested that communities
prioritised education and the general pattern which was observed was that once life-saving
interventions such as food and shelter had begun, education activities were initiated. Labutta was a
special case as it was assumed that community needs were being catered by the large number of
NGOs operating in the townships and hence Save the Children prioritized under-served areas (such
as Middle Island). It was only later that it was agreed to support education activities as a gap-fill.
It is noteworthy that once Save the Children Education teams had established a presence, start-up of
interventions was often ‘delayed’. For example in Labutta, the education programme took 2-months
to get underway. One of the reasons for this is that it took a long time to get permission from the
relevant education authorities for the proposed activities and it took some time to build relations.
Child Protection: According to proposals and Yangon office instructions, child friendly spaces (CFS)
were to be established before the end of August 2008 and in the majority of the villages visited most
had been established by this time. Given the difficulty in accessing villages due to the poor weather
conditions and the huge area covered, this should be considered a real success. The challenge such a
rapid scale-up bought was the depth to which the teams could go to in order to meet targets. One of
the casualties of this was that teams were often unable to conduct proper assessments which were
mostly carried out with the participation of local authorities and village leaders rather than a more
representative sample of the community. It also meant that there was very limited time for training
which risked compromising the effectiveness of the programme. CFS construction training was
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 12
provided in June and it wasn’t until September when training related to the objectives and the
activities of the programme was carried out. The implication of this was that some team members
went to villages without having a clear understanding of their mission.
With respect to the family tracing and reunification programme, a large number of children were
found to have been registered in each township and for most, a solution, either temporary or longterm has been found. Training of the team was timelier, with a July and October programme
although some members of the team still felt they had a lack of knowledge to undertake the work
effectively.
3.2 Recommendations
The large size of the programme is an important qualifying factor in any discussion about
timeliness of the programme, however Save the Children could have avoided some of the
delays experienced across some of the sectors through better preparedness planning including
a pre-agreed ‘package’ of possible emergency interventions for each sector;
Specifically for nutrition, Save the Children should consider the development and
implementation for key staff of a preparedness package related to nutrition activities in
emergencies inclusive of internationally developed assessment tools. The development of this
package could also include development of tools for predicting caseloads of the carers of
infants requiring support for breastfeeding or safe artificial feeding;
Sectoral activities should be categorised by the impact they have on lives at the outset of a crisis
and a phased strategy of implementation agreed to ensure that the greatest possible number of
lives are saved. This is particularly pertinent for the WASH and nutrition sectors;
In such a rapid scale-up over such a large area, there are inevitable trade-offs between
responding to humanitarian need and ensuring that skills gaps are bridged. In making a
judgment about the size and scale of response, it will be important to be clear about the
minimum necessary skills required to deliver quality programmes and to phase programmes
accordingly. Where a skills gap is anticipated or discovered, it is recommended that either
creative ways are quickly found to fill this or that programme ambitions are reduced.
4. Coverage (inc. Co-ordination)
Summary: Given the limitations in both the government and NGO response capacity, the decision to
go to scale was the right one but required the organisation to undertake an unprecedented
recruitment drive and work in sectors where it had very little prior experience. While strong coordination between Alliance members greatly facilitated the swift scale-up, the lack of internal coordination between different sectors of the response at a country level meant that opportunities for
coherent programming were missed.
4.1 Main Findings
Cross-cutting Issue – Internal Co-ordination: The Save the Children Unified Presence laid a
foundation for strong inter-Alliance co-operation which saw global resources focused on the needs
and opportunities presented by the Nargis response. It is noteworthy that without exception the
Myanmar staff considers itself a Save Alliance team and hence there was no requirement to consider
or negotiate as the immediate assumption was that any response would use the same model of
collective action. This is, without doubt, a tremendous strength of the programme and permitted a
fluency of decision-making which greatly facilitated early response to the crisis. While it was difficult
to determine the impact on programming that such a fluency had, it is a fair assumption to make
that a single entity, working collectively and drawing on the combined resources of Alliance
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 13
members was able to make a swifter response, and develop a more joined-up programme than a
collection of Alliance members would have been able to. The fact that the beneficiary of the
Alliance’s considerable resources was a single Save Alliance entity with a unified voice permitted a
high level of efficiency and co-ordination.
Food & Livelihoods: The scale of Save the Children’s food distribution has been vast. It distributed
food to 80% of the population in areas where it was the lead food agency. In October 2008 Save the
Children was distributing food aid to 275,000 beneficiaries each month as an implementing partner
of WFP as well as filling gaps in very vulnerable areas with its own food distributions. This is
remarkable, given the variety of operational constraints in Myanmar in general and the delta in
particular during the rainy season.
The QIP was also large scale as well as being high risk, given that the authorities initially prohibited
cash distributions. Save the Children distributed cash/in-kind grants to 41,000 households in 300
villages across 10 area offices in the eastern and western regions of the delta. This is again
impressive, given that there was no pre-existing livelihoods programme or staff prior to the cyclone.
It has been difficult to ascertain the extent of Save the Children’s cross-sectoral planning. A GIS
mapping of sectors present in villages would be extremely helpful to support an analysis of the Save
the Children’s added value, as well as assisting with transition planning.
Nutrition: The coverage of Save the Children nutrition programme is immense, reaching 585 villages
in 9 townships. This with a limited number of staff considering the vast areas reached. Numbers of
beneficiaries reached have far exceeded set targets. A total 8298 children aged between 6 and 59
months have been screened for MUAC of which 105 children have been treated for severe acute
malnutrition, 1158 for moderate acute malnutrition and 464 pregnant or lactating women have
been treated for acute malnutrition. In terms of IFE, 17 819 carers of children <2 years have
attended IFE sessions of which 984 have had skilled IFE assessment and 441 have received skilled IFE
support. In total 26558 people have benefited from the nutrition programming 8739 of whom are
children.
Picture 3: IFE information posters, Middle Island
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 14
The nutrition survey conducted in October 2008 presented rates using the direct method to assess
coverage for the two target townships. In Mawlamyainegyun coverage was calculated to be 71.7%
falling well above international standards of 50% in rural areas. In Ngapudaw however coverage was
calculated at 17.8% which is considerably lower. While the screening criteria in use which was not
sensitive to a large proportion of GAM means coverage calculations may not be truly representative
of actual coverage of GAM (see technical merit), this does not explain the big difference in the two
regions. Logistical constraints and the isolated nature of Ngapudaw may be contributory factors to
low levels of coverage.
Village selection for IFE was dependent upon the presence of child protection teams and the
presence of a CFS. At the assessment stage however, rather than working only from CFS catchment
villages, the breastfeeding counsellors gradually assessed communities outwards from the office and
villages partly because of the slower roll out of CFS’ and partly due to the limited catchment
populations. This can be considered appropriate as the organisation of the CFS is not always
conducive to the skilled assessment or counselling that may be needed by some women and in
reality these women are often unable to travel to a CFS. Government areas were not targeted in
order to avoid duplication.
Coverage of stabilisation centres for children with complicated SAM was difficult to quantify during
the evaluation but is believed to be very low and of poor quality based on field discussions. In
addition to treating complicated malnutrition these centres typically address the needs of children
under 6-months who as with children with complicated malnutrition, are at risk of death and require
specialise support. Without the SC IFE programme there would be no other appropriate nutrition
support for these children.
Health: In most cases, the health cluster meetings determined areas of intervention for health
programmes and ensured a level of co-ordination between NGOs and the DoH. The decision to base
the mobile teams at area offices and to travel daily to the villages perhaps meant that those living in
remote areas missed out and served to reduce the service time to the beneficiaries and increased
dependence on a weekly service, however, given the absence of government services in the
immediate aftermath of the cyclone, the mobility of the Save the Children service was certainly
beneficial. The delivery of free treatment and drugs up until December also ensured that services
were available to all who required them. Coverage and use of the clean delivery kits and new born
kits was expected to be 75 to 80% but there was no documentary evidence to confirm this. Similarly,
the distribution of mosquito net for malaria prevention and condom distribution for prevention of
STIs including HIV was done but there was no evidence to see use and coverage during the
evaluation.
Levels of coordination with health service provides at village level was found to be patchy. In some
villages, parallel medical services were being offered by Save the Children mobile service and DoH
health centres and local health staff were quick to voice concern that better collaboration would
have meant that more remote villages could have been served.
WASH: The WASH programme straddled a huge geographic area and managed to reach an
impressive number of people. This was complemented by a huge WASH NFI distribution from which
over 480,000 people reportedly benefitted in addition to the distribution of buckets to an additional
70,000 households. As well as being a tremendous strength, the large geographical area also posed
some challenges to Save the Children. Given the rapid scale-up and the limited watsan experience in
the team, there was a large skills gap which took time to make up, particularly given the decision not
to have field-based sectoral advisors. Furthermore, with the team working across such a large area,
there were insufficient staff numbers to visit each village more than once a month, and on some
occasions, visits happened only every 2-months. Give the fragility of the situation and the absence of
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 15
robust monitoring mechanisms, this meant that it was not possible to provide adequate oversight or
monitoring of programme quality. The lack of a co-ordinated approach between health, nutrition
and WASH is also considered a missed opportunity and served to cement the fractured nature of the
programme which often saw different sectoral teams work in silo’s rather than seeking to exploit
cross-sectoral opportunities. There were some efforts to overcome this such as joint monitoring of
school and CFS latrines by WASH, CP and Education teams and sharing of data on water borne
diseases between the health and WASH teams, but these appear to have been exceptions to the rule
rather than the norm. It is understood that the issue of inter-sectoral co-ordination will potentially
be addressed through changes to the management structure.
Education: Save the Children did an amazing job covering the targeted areas considering the tough
geography, significant constraints to travel and (initial) inclement weather. In terms of coverage Save
the Children adopted different approaches in different areas based on the context and need. In
areas such as Middle Island where Save the Children was the main provider of support to education,
there appeared to be good coverage of most schools although documentary evidence to support this
is limited. In other areas such as Labutta, a ‘gap fill’ approach was used as there were many other
agencies supporting education, albeit incompletely in terms of coverage.
There doesn’t appear to have been a consistent approach to targeting of communities at the
township level. In areas such as Labutta decisions appeared to have been driven by what the
township education officer suggested whereas in other areas, referrals from other sector teams
provided a cross-check. Given the lack of documentation, it’s not possible to make a judgment on
the extent to which Save the Children targeted the most vulnerable areas or communities.
From interviews held during the evaluation and project reports, Save the Children worked closely
with the Education Cluster nationally and locally and through this had strong linkages with township
education officers. From discussions with staff, it was evident that they were aware of other agency
interventions such as UNICEF, Malteser, and private companies. Save the Children support did not
appear to duplicate support provided by other agencies. In a couple of schools, stationary supplies
were provided to students who also received student kits from UNICEF but this appeared to be in
cases where such support was considered to be insufficient for the needs of the students – which is
a reasonable explanation.
At the local level, Save the Children coordinated its activities with local authorities such as the Peace
and Development Committees and 100 Households. Usually members from these established groups
were part of the Save the Children education committee who were generally considered to have a
good understanding of their role and played an important part in the implementation of the
programme.
Child Protection: Child Protection teams assessed villages for CFS activities during the food
distribution. There was no formal assessment tool to guide this process and so it’s difficult to
establish the extent to which the programme responded to the greatest needs (villages which had
the greatest number of vulnerable children) – although it did permit quick decisions to be made and
for the programme to scale-up relatively quickly.
There appears to be a lack of agreement on the criteria for selection of villages used by CP teams.
The view from the field was as follows: (i) agreement of the community to have a CFS; (ii) the
number of children in the village and; (iii) availability of a suitable site for the CFS to be constructed
on. There were some additional criteria (which appeared to depend on the team) such as the
availability of community volunteers (animators), the extent of the impact of the cyclone on the
village. According to the team in Yangon, the criteria also included the number of children in need of
psychosocial support/extremely vulnerable children but this was no evidence of this during field
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 16
discussions. On several occasions, it was discovered that some of the most vulnerable children were
not part of the programme.
Regarding the FTR work, there was clearly a huge task to trace families after the registration of
children as there was often a lack of information provided by them. It was noted that some of the
villages with a significant number of orphans are not yet covered by CP teams though there are Save
the Children education activities ongoing.
4.2 Recommendations
The key challenge for the Save the Children Alliance is to seek to institutionalise as many of the
elements which facilitated the successful Nargis scale-up as possible to ensure a level of
predictability across global emergency responses in the future. While this won’t be possible for
all aspects of the response, many can be standardised through rigorous recruitment practices
and step-aside policies, a strengthened emergency architecture, a centralised standby capacity,
a pool of funds set aside for under-writing emergency programmes, and rigorous and feasible
emergency preparedness plans. An organisational culture which permits and supports risktaking is also a prerequisite. Without doubt, the Nargis response should be cause for optimism,
but if predictability is the goal, then it will be important for the Alliance to build on this
successful model of collective action to further develop its humanitarian capacity21.
To date there has been very limited integration of sectoral programmes. Looking to the
transition programme, it will be important consolidate/rationalise areas supported by Save the
Children in coordination with other agencies. It is suggested that a GIS mapping exercise across
sectors may help to support this process (if it has not already been done by OCHA). A GIS will
visually identify the overlap/gaps between Save the Children sectors working in each region and
village and will enable a coordinated, integrated approach in villages selected for the next
phase. It will also help analyse Save the Children’s added value from working multi-sectorally;
A basic building block for the transition phase should be a holistic/cross-sectoral village-level
assessment which should form the basis of all future targeting and interventions;
Save the Children should review its approach to targeting villages for child protection and take a
decision on whether it should seek to support family tracing and reunification in all villages it
has a presence in. If not, at a minimum it should seek to conduct a rapid assessment and raise
the issue in the appropriate cluster group meeting.
CASE STUDY 2: Save the Children knowledge of and engagement with civil society organisations
The inclusion in the evaluation of a team member dedicated to exploring civil society linkages
represents an important strategic investment for future collaboration between Save the Children and
civil society. The case study below provides an overview of Save the Children’s engagement at the
tail-end of an extraordinary period of humanitarian activity in the delta. Recommendations are made
for action that could be taken during the transition programme to build on the foundations already
laid by the Save the Children team.
A variety of definitions of the words “civil society” were given by the Save the Children staff but very
few staff were able to provide a comprehensive explanation. Over half of the respondents were able
to list the names of Myanmar NGOs and faith-based groups who were implementing programmes in
the respective townships, but only a few staff were able to describe what activities they were
carrying out and their location. Senior staff generally had a better knowledge of civil society
21
ELT Operational Review of the SC Alliance Response to Cyclone Nargis, Andy Featherstone, October 2008.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 17
organisations and their area of operations although their knowledge was still limited.
Discussions provided 2 potential explanations for the limited knowledge; (i) The civil society
organizations in Myanmar are usually working at a low-profile for security reasons, and many of the
faith based organisations rarely involve themselves in the coordination meetings held in the
townships so it’s difficult to get information about their activities and capacities, (ii) as Save the
Children is involved in direct implementation of emergency programmes, the Save the Children
teams are often too busy to collect information about civil society organizations some of whom may
be working in close proximity to them.
Over half those interviewed expressed a conceptual acknowledgement of the need for local
organizations such as CBOs, self help groups and village committees in the township and villages as
well as an appreciation of the achievements of these local groups and organizations. In one
township, Save the Children WASH team is working alongside a local NGO but as they have limited
knowledge of such organisations they don’t necessarily appreciate the commitment and dedication
of these groups. Despite this over 80% of the respondents said local groups and organizations such
as this can become potential partners of Save the Children in future but, they said this will depend
on the guidelines of the Save the Children head office.
The evaluator observed an “institution-compound mentality” by the Save the Children staff. They
are far more keenly aware of and place far greater emphasis on their own internal environment than
that of external organisations and this leads to the risk of them becoming “institution staff”, with
extensive knowledge of the internal and external workings of Save the Children rather than
“humanitarian staff” having a broader understanding of the principles and institutions that make up
the humanitarian community in the delta more broadly.
Through the township level cluster meetings, Save the Children is able to build relationships with
different types of civil society organizations, albeit mostly at a level of coordination and information
sharing. In some sectors however, it has been possible to go beyond this and collaborative activities
with civil society organizations have been initiated. For instance, the Save the Children IFE team in
one township has started working with a range of civil society including local groups and
organizations. Similarly, the WASH team in a different township is working closely with one of the
local NGOs in the emergency response programme. What is apparent from this is that where
opportunities present themselves for collaborative working, then Save the Children staff are willing
to take advantage of these; however it is clearly not the case that this is part of a broader strategy of
engagement with civil society groups.
At a strategic level, collaboration with civil society groups, particularly faith-based NGOs and local
NGOs is weak despite the efforts made as part of the Paungku Project in the delta. The Save the
Children programme team has clearly (and perhaps understandably) focused its energies on the
emergency response which has left very limited time for investment in civil society. A potential
obstacle to future collaboration is attitudinal as the majority of the staff interviewed were
unconvinced of the ability of such groups to achieve quality outcomes.
The following recommendations are made for the transition phase of the programme during which it
is anticipated that the operating context and objectives of the programme will permit stronger
collaboration between Save the Children and civil society;
The role of civil society in the transition programme, particularly in DRR activities should be
discussed and agreed at a strategic level and appropriate structures established within
communities from the outset of programmes. This will require the establishment of an
appropriate community mobilisation approach and training of staff in mobilisation and
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 18
facilitation methodologies;
A CBO capacity building programme or capacity building framework should be introduced to the
potential villages for longer-term implementation and sustainability;
The situation of women’s participation in village committees should be accompanied by a
relevant women’s empowerment framework to improve quality participation of both women
and men;
Save the Children should develop an active action-reflection circle to facilitate learning from
and sharing with layers of civil society especially with relevant groups such as CBOs and
Myanmar NGOs;
Network and collaborations should be strengthened among and between villages and peer
learning systems should be developed.
5. Relevance
Summary: The evaluation found the programme to be relevant across all of the sectors. Over time,
programmes developed a sound understanding of the context and were effective in meeting the
basic needs of communities. The vibrant and informed discussion about the complexities of
targeting went a long way to ensuring the appropriateness of programme interventions.
5.1 Main Findings
Cross-cutting Issue – The Challenge of Targeting: Vulnerability and the concept of targeting are
understood differently in Myanmar. Most agencies have reported that the community sense of
‘equity’ is extremely strong, with assistance almost always re-distributed by communities so that all
members benefit a little. Communities have a very strongly developed social cohesion and
traditional safety nets. Experience from Save the Children’s programmes has led teams to be
understandably critical of the viability of targeting within villages since the relief items are ultimately
redistributed in any case. Such informed pragmatism which builds on field experience has been a
real strength of the programme and should be fostered. The use of relevant external co-ordination
mechanisms to share knowledge will be important to inform practice of the wider humanitarian
community. There is also scope here for ensuring that Save the Children learns from its experience
internally. The Accountability team, which was established in December, has an important role to
play in this, but there is also scope for a senior, centrally-based staff member mandated with a
strategic, analytical oversight role to work across sectors to identify common approaches and
practices to community engagement and targeting (see food & livelihoods, section 7). A move to
decentralise decision-making and allow field teams’ greater reach into programme strategy and
planning would likely also strengthen analysis of community dynamics that may lead to improved
practice (see management structure, section 9).
Food & Livelihoods: Save the Children’s food and livelihoods programmes were extremely relevant.
Rapid assessments undertaken by Save the Children by July 08 indicated that although some food
was available in local markets, prices of basic commodities had increased by more than 50% in some
cases and the cost of transport by as much as 1000%22 The short-term distribution of food aid was
thus vital to protect vulnerable households’ health and livelihoods23, with household income then
being available to support early recovery activities. Save the Children did not undertake a market
analysis until October/ November, however, ongoing market analysis is vital to monitor factors that
may affect vulnerable households’ well-being.
22
23
th
Summary of Save the Children Cyclone Nargis Response Strategy, 20 July 2008.
Food aid should prevent households entering into harmful livelihood coping strategies.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 19
Save the Children should be commended for covering the urgent food needs of households not
included in WFP distributions24. Save the Children argues that the objective of its distributions was as
income support to protect livelihoods although little analysis to monitor the impact of the
distributions on beneficiaries’ well-being appears to have occurred25. Some post-distribution
monitoring was undertaken from August to early October but reports were not produced until the
end of November, towards the completion of food distributions in some areas. Limited crosssectoral analysis with the nutrition team has taken place.
WFP’s selection criteria, however, targeted specific households in 100% of villages using criteria
including households with more than 50% damage to their shelter. Although there were other
criteria, such as targeting households with pregnant and lactating women and very poor and elderly
headed households, ‘damage to shelter’ was the main criteria used. In practice, no wealth rankings
were undertaken because of time pressures and relatively inexperienced newly deployed staff who
had not undertaken targeting in communities previously. Well-off households were thus often
recipients of food aid, with poor households omitted from lists when they had sustained limited
damage to their shelter. Save the Children staff, although uncomfortable with the process,
distributed food to beneficiaries and communities then re-distributed food amongst themselves as
Save the Children staff departed so that all households received something.
Save the Children has demonstrated its ability to adapt programmes over time to better support the
extremely vulnerable and prevent harm being done to communities’ self-sufficiency. In December
2008 it revised its food distribution practices in Middle Island, the last remaining location it is
continuing to distribute food. Criteria used for beneficiary selection is now based on a wealth
ranking. Food distributions were stopped in other areas following the Save the Children food needs
assessment in October/ November 2008 because they were found to be self-sufficient following the
harvest. This strategy should be monitored carefully with the nutrition and health teams to ensure
the well-being of the extremely poor.
The QIP has been very successful. All key informants and villagers met stressed the beneficial impact
it had had in supporting the early livelihoods recovery of the cyclone affected population. It was
reported by villagers to have had a particularly beneficial effect on the very poor. The ‘better off’ in
communities also expressed their appreciation as they were able to combine this money with their
own income to buy more expensive livelihood assets lost in the cyclone, such as boats, buffalo and
larger fishing nets. Save the Children’s decision not to target QIP support within villages is
reasonable, given that around 85% of the population in the delta is functionally very poor26. All
villagers met expressed their appreciation that all villagers had benefited from this initiative.
“Those that had a lot before Nargis lost a lot. Those that had little lost a
little…… We are all now in the same situation”.
Save the Children argues that the QIPs project began the early recover process but more
comprehensive livelihoods recovery is needed at the household and market level. By introducing an
additional intervention of CFW at the completion of QIPs, it believed it could build on the impact of
QIPs on the recovery of vulnerable households’ livelihoods.
24
WFP capped the amount of food it distributed.
SPHERE standard.
26
The landless represent 60-65% of the total population in the delta, with small land owners having less than 5
acres of paddy land representing 15-20% (Livelihoods Rapid Needs Assessment in Post-Nargis Myanmar,
November 2008).
25
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 20
Save the Children has transferred significant responsibility for the CFW project implementation to
committees established (see ‘accountability’) but it now has more time to engage with communities,
unlike in the early months of the response. Although there is an enormous potential to integrate
village level CFW projects with other sector initiatives, such as WASH/Health and Education27, this
does not seem to have taken place as yet.
Nutrition: Pre Nargis government data (though believed to be under-reported) in Myanmar shows
that 2.4m (32%) of children under five are either moderately or severely underweight, and in some
areas, levels of acute malnutrition were at emergency levels. The village tract assessment indicated
close to half (48%) of surveyed households had lost all food stocks during the cyclone. On the day of
the survey, almost 28% reported having no food stocks remaining, and a further 43% reported
having stocks sufficient to last only 1 to 7 days28. To further compound the desperate situation, May
and June are peak hunger months.
After the cyclone an initial rapid screening in 24 camps in MyuangMya showed 17% GAM (2% SAM)
indicative of a necessary response. The nutrition survey conducted in November further confirmed
high levels of GAM, 14.5% GAM and 1.9% SAM in Mawlamyainegyun and 11.5% GAM and 1.1% SAM
in Ngapudaw. Levels of GAM and SAM guided by international standards are all indicative of the
relevance of this response. In areas with reported lower levels of malnutrition such as Pyapon
where a rapid assessment indicated 5% GAM29, response has focused largely on prevention through
IFE.
An initial rapid assessment conducted by Save the Children staff in temporary settlements in
Myaung Mya Township noted that many infants <6months old were being fed whatever foods were
available in the settlements and confirmed inappropriate, and sometimes dangerous, feeding
practices for young children. ACF also reported very high rates of inappropriate complementary
feeding, noting that some 50% of the mothers and carers they interviewed described feeding
complementary foods. These foods are traditionally spoon or finger fed but some use of infant
feeding bottles was also observed.
Pregnant and lactating women are also a vulnerable group. Breastfeeding mothers who do not have
access to adequate food may themselves deplete their own nutritional and energy reserves quickly
and show signs of malnutrition and begin to feel fatigued as a consequence. Save the Children data
shows a number of women identified as malnourished. Mothers interviewed during the evaluation
reported feeling ‘weak’ due to lack of food and had previously believed this would affect the
quantity and quantity of their milk supply and had therefore sought additional foods for their
children, a practice which they report to have changed since being in contact with SC breast feeding
counselors and trained community volunteers.
The predisposing suboptimal nutrition practices and high malnutrition rates coupled with increased
vulnerability risks as a result of cyclone Nargis indicate a sound rationale for the comprehensive
approach taken in both IFE and CTC components within the nutrition response as a means of
addressing child survival in the wake of such an event
Health: The use of National and international health advisors in Yangon and experienced health
coordinators ensured that Save the Children provided relevant medical care to people. Guidelines
and protocols were followed for emergency health care. External professionals were approached to
train local and front line workers. The initial assessment conducted by area staff formed the basis for
27
Protecting water sources, adding water sources, repairing schools and latrine construction.
Post-Nargis Joint Assessment, Food & Nutrition Clusters Village tract assessment
29
Rapid Nutrition Assessment, Unicef, 2007
28
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 21
the establishment of the mobile team services. It was then reviewed in October when it was decided
to partially change the programme; several of the static clinics in Middle Island were closed, and the
mobile medical teams were incorporated into CHANT (community based health and nutrition team).
This team commenced work in December and was considered to be more relevant to the needs,
being more community-focused and using a better strategy of targeting.
The community mechanism for health services was ineffective before Nargis and was almost nonexistent after the cyclone hence the programme approach of community mobilization through
volunteers and health teams was relevant and valid. Community engagement with and acceptance
of the service was found to be high in all programme areas.
In addition to suffering wounds from the cyclone and more common diseases, those that survived
Nargis also suffered from psychological trauma. Save the Children’s efforts to train some of its staff
in what is considered by the OCHA-sponsored Interagency real time evaluation30 to be one of the 2
main gaps in relief provision (the other being a response to dry season water shortages) to deal with
such cases and provide necessary services including referrals was highly appropriate.
WASH/NFI: There is no doubt that Save the
Children provided relevant assistance to
communities. By focusing on the basics of
access to potable water and sanitation
services and by adhering to SPHERE
standards, some of the most basic needs of
communities were met, albeit not in the
initial emergency phase. The temporary
rainwater tanks provided important access
to water for community members, despite
some being built towards the end of the
rains.
The sanitation programme was very
successful
and
embraced
by
the
communities involved. The latrines visited
were being properly maintained and
cleaned and a bucket of water and soap
were present. Despite the beneficiaries not
being involved in the design, interviews
revealed that there was considerable Picture 4: Save the Children temporary latrine
appreciation about the structure which was
felt to be more appropriate than previous ‘traditional’ designs. In community discussions there was
much praise for the solid design, safety and absence of flies and odour in the Save the Children
latrines. It is important to note that some of the latrines were difficult for children to access due to a
high step and it will be important for ease of access to be considered in the future.
The key messages discussed with communities as part of the hygiene promotion programme were
well received and understood by community members (both adults and children). Due to the lack of
a baseline survey31 it is still difficult to make a judgement on whether this is as a result of the work of
30
Inter-Agency Real Time Evaluation of the Response to Cyclone Nargis, R Turner, J Baker, Dr. Zaw Myo Oo,
naing Soe Aye, UN OCHA, December 2008
31
The WASH team is planning to conduct a KAP survey in March 2009 which will be used to construct a
baseline and will facilitate the monitoring of progress.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 22
Save the Children or due to pre-existing knowledge but there was clearly a high level of satisfaction
with the programme.
The contents of the WASH NFI was not standardised and differed in each geographic location with
the main issue being quantity of items received (water buckets in particular, but other items also). In
future, a standardised NFI package is recommended.
Community discussions highlighted the fact that the mosquito nets weren’t pre-treated which
reduced the effectiveness of the nets and represents a missed opportunity. The use of Waterguard is
also questionable – both in terms of its effectiveness and also for issues of community uptake. In
many of the villages visited, the small bottle that was originally distributed 6-months previously was
‘still being used’ suggesting non-usage. Some villagers complained about the taste of the water after
treatment with Waterguard. Usage levels were not monitored by the Save the Children team.
Ceramic filters were distributed as part of the package and were also included in other agency
distributions. It would be useful to evaluate the usage of this item more extensively as it certainly
appeared to be preferable to Waterguard due to the absence of taste. While elsewhere in Asia,
filters are extensively used, this is not the case in the delta and the ceramic filters may be difficult
and expensive to replace. It is recommended that the use of filters is reinforced through the public
health promotion messaging in addition to working with other WASH stakeholders, including the
government to make the filter and spare parts more accessible. If this is not found to be practical or
tenable, then the appropriateness of filter distribution is bought into question and other methods of
household treatment should be explored.
Education: Save the Children assistance was relevant to the context and community requirements –
school infrastructure and materials. Education programmes were focused on the school, not the
child which meant that the interventions were relevant to the needs of the community generally but
not necessarily for the needs of specific vulnerable groups such as orphaned and/or working
children, or children from very poor households. In Labutta and Mawlameigngyun, primary aged
children were not in schools due to cost and parents not valuing education. Some parents felt that
working was more beneficial for their children than an education. The latter sentiment was
especially true for post-primary aged children.
On the training component, the teachers interviewed found the child-centred approach and painting
the most useful items from the training programme and disaster preparedness and games the least
useful. The teacher who stated that games was the least useful taught Middle School and said games
where boys and girls touch, etc was not appropriate at this age and more appropriate for primary
age children. Regarding the topic of preparedness, a teacher felt that it was not useful as the
message was too simplistic, simply saying “go to high ground” in case of a cyclone or disaster and
there was no high ground in the location which he was based.
Child Protection: The establishment of CFS activities was considered to be a real benefit by all
communities interviewed and the programme clearly responded to their needs; parents expressed
their gratitude for having a safe place for their children to play and expressed how participation in
the CFS had helped their children recover from the trauma of Nargis through being able to
participate in different activities. It was evident also that many parents had a greater awareness of
the importance of their children’s well-being. Parents felt that children gained in self-confidence and
in their ability to express themselves in the CFS.
For children, the CFS was the place where they could learn new games, play with new toys, and gain
knowledge on various topics (including hygiene messages). Although the children sometimes found
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 23
it difficult to express themselves, it was evident from discussions that the CFS’ had become “their”
place where they were able to play in a safe environment.
The Child protection strategy adopted by Save the Children was to target vulnerable children and in
all villages visited, most of the children attended the CFS. As a result of Nargis (although to some
extent pre-dating it also), a lot of village children, many of them girls, are out of school and stay
home to take care of their siblings while their parents are working. As they were able to take their
siblings to the CFS with them, it was a great opportunity for them to socialize with other friends and
participate in activities.
However, there were also specific vulnerable groups who weren’t able to access the CFS and
effectively were passed over by the Save the Children programme. The most notable of these were
children working full time (in salt processing, fishing, farming activities) who didn’t have the
opportunity to attend the CFS. When Save the Children team members were asked about this group,
the responses given showed a gap in knowledge and ideas about how such children could be
incorporated into the programme. This reflects an understandable skills gap amongst some of the
staff – many of whom have had very limited opportunity to be trained and have no experience of CP
outside of their work for Save the Children. As a result there is limited understanding and knowledge
of key issues such as rights, protection, vulnerability and mobilisation. Much of the training that has
been provided has been targeted at the programme officer level and individuals have not always
been able to effectively transfer knowledge to their colleagues.
There is regular monitoring of activities, with includes collection of figures regarding activities,
targets, follow-up etc, however, there is very limited qualitative monitoring which could
demonstrate the impact of training provided or allow for feedback which would allow programme
staff to adjust the activities and address issues of concern. The same can be said of FTR for which
there are quantitative data sets (actions taken, services provided, caseload analysis, follow-up
undertaken), but an absence of qualitative data that would allow measurement of processes such as
progress of psychologically affected children or follow-up for children at risk of institutionalization.
5.2 Recommendations
In order to ensure the best outcomes from its programmes, it is recommended that Save the
Children documents its experience of targeting to inform future programmes. A close working
relationship with the Accountability & Learning Working Group in Yangon will undoubtedly
benefit the Save the Children programmes as they are further refined in the transitional phase.
The creation of a strategic oversight role, responsible for working across sectors to identify
common approaches and practices to community engagement and targeting and the
decentralisation of decision-making would likely also strengthen Save the Children’s
understanding of community dynamics and facilitate improved practice across the programme;
The situation of extremely poor community members living in villages which are no longer
receiving food aid should be monitored (as a follow-up to the October/November needs
assessment) lest there is a significant drop in self-sufficiency of the caseload. This action should
be co-ordinated at cluster-level;
A strong analysis and understanding of social networks and structures will be vital as the
Livelihoods programme moves in to the transitional phase to ensure that Save the Children
reinforces traditional networks and does not do harm. An understanding and identification of
the small number of marginalised groups that fall outside of existing social networks will also be
important;
Save the Children should review the design of the latrines to ensure that they are accessible to
children. Where they are elevated off the ground, and a step has not been provided, teams
should seek to provide one or advocate for communities to make appropriate modifications;
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 24
Save the Children should seek to standardise their NFI distributions in the future – where it is
not possible to distribute the full package due to lack of stock, distribution teams should return
to villages at an appropriate time to ensure beneficiaries receive their full entitlement of NFIs;
The use of Waterguard as a means of providing potable water should be evaluated in more
detail by Save the Children with a view to understanding issues of relevance and uptake in
order to inform future practice;
The distribution of ceramic water filters in some locations was highly relevant but would benefit
from follow-up both by public health promotion teams (for upkeep of the filter) but also at a
Yangon level with WASH stakeholders, including the government to advocate for spare parts to
be made available;
The education team should critically assess the relevance of the various components included in
the training programme offered to teaching staff to inform future practice;
It will be important in the future for Save the Children to reflect on ways that it can reach some
of the most vulnerable children, particularly those working full time who have been unable to
attend the CFS. As part of this, it should incorporate sensitisation on issues of child labour into
ongoing CP programmes;
Further assessment and mapping of malnutrition and infant and young child feeding practices in
Save the Children catchment areas is recommended to guide appropriate programmes in
addressing both the consequences and underlying causes.
CASE STUDY 3: Save the Children and Village Committees
As part of its humanitarian response, Save the Children has initiated and convened village
committees across all of its sectors. The committees are usually organised and formed by the open
consent of the majority of villagers at the proposal of village authorities or village elders. In the
committees, women are often requested to participate and over 90% of village committees
interviewed by the evaluator had at least 30% women committee members. However, the reason
for women’s participation is rarely explained and there is no evidence of Save the Children
monitoring the contribution they make or the performance of the committee more generally.
In terms of male participation, It is evident that the male committee members are usually village
elders and those who have influence in the community and hence are rarely representative of the
wider community. Village authorities are usually involved in the committees in the role of patrons,
and the form a link between the Save the Children committees and the local authoritative
committee.
Over 90% of the village committees that are working with SC were initiated and convened by the
Save the Children staff in order to facilitate the process of relief provision. In severely affected
villages, it was observed that there is a very weak connection between the newly formed
committees and those that pre-existed and it was only in the less-affected villages that pre-existing
committee members were used in the newly established committees.
What is clearly evident is the proliferation of committees as a result of Save the Children. For
instance, in one village, community members testified that they have eight committees initiated by
Save the Children only. On average, however, there are two-to-three committees which have been
established by Save the Children in each village and each of these committees is entirely sector
focused. It was noted that the livelihood sector committee is still in place despite the fact that the
programme has ended and there are no plans for work in the future.
In the relief programme these committees do not have any wider scope or longer-term strategy –
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 25
they are purely for the purposed of the administration of the sectoral assistance programme.
Participation of the committee members on the committee is highly appreciated, but their activities
are wholly guided and directed by Save the Children, with very limited evidence of efforts to build
capacity. This might be relevant for the short-term nature of the relief programme, but could be
considered a missed opportunity should Save the Children choose to work in the delta longer-term.
As part of the transition programme, it is understood that Save the Children is planning to integrate
all the sector committees to become a single village development committee, but planning for this is
still in process.
6. Technical Merit
Summary: While it’s difficult to make an accurate judgement on the quality of the programmes due
to a lack of baseline data and gaps in monitoring information, the programme is judged as being
technically sound and was largely successful in meeting SPHERE standards. In scaling up the
programme to such an extent, often in areas outside of the teams core competencies, programme
size has at times been given priority over programme quality, and there is an important need to
ensure that the skills gaps that remain are filled as the programme moves into the transition phase.
6.1 Main Findings
Food & Livelihoods: In general, Save the Children has met the SPHERE minimum standards regarding
implementation of its activities but it should arguably have strived to undertake more situational
analysis and monitoring after the initial few weeks of the emergency. The evaluation team
recognises, however, the extremely challenging conditions under which the Save the Children
response was operating where access to the delta was severely restricted by the authorities until
July which means that needs assessments were being undertaken by national staff teams who were
often relatively inexperienced in emergency response32.
Baseline data is limited in Myanmar and particularly for the delta however, Save the Children had
undertaken a household economy assessment in Kangyidaunt Township, Delta, in February 2007,
which SCUK’s London office then shared with other organisations in June 2008 to assist with
humanitarian efforts. It thus had “an understanding of [how the cyclone affected population]
normally accesses food, the impact of the disaster on current and future food security, and hence the
most appropriate response” (SPHERE). The extent, however, to which Save the Children “analysed
the impact of food insecurity on the population’s nutritional status”, is not clear (SPHERE)33. It was
not until October/ November 2008 respectively that Save the Children was able to undertake rapid
post-Nargis food security and livelihoods assessments, which were used to inform its next phase of
programming.
Save the Children provided large- scale food distributions to meet cyclone- affected communities’
immediate food needs, later followed by cash or in-kind cash grants to support early livelihood
recovery. The assumption was made in its own food distributions, where just rice was distributed,
that households had access to other food items by purchasing them in markets or undertaking
livelihood activities, such as fishing. This assumption does not appear to have been fully monitored.
32
Technical expertise is required across sectors (SPHERE).
Access to food and the maintenance of adequate nutritional status are critical determinants of people’s
survival in a disaster. Malnutrition can be the most serious public health problem and may be a leading cause
of death, whether directly or indirectly. …Food aid can be important in protecting and providing for food
security and nutrition, as part of a combination of measures (SPHERE).
33
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 26
Although some post-distribution monitoring was undertaken from August to early October, reports
were not available until the end of November 2008. It did not undertake a market analysis until
October/ November. This should have been undertaken on a regular basis from programme set-up
to monitor the potential impact of its large scale food distribution on market prices as well as the
populations’ access to markets, availability of goods and any changes in vulnerable households’
purchasing power.
The initial Save the Children food aid response was vast in its scale and scope and took place in the
months immediately post-Nargis, as did the implementation of the QIP. Establishment of systematic
monitoring systems would have been difficult at this time because of the pressure to implement
programmes. Save the Children should by now have qualitative food and livelihoods monitoring
systems in place to inform programme implementation34. Systems should be established to collate
information regarding SPHERE indicators related to food and livelihoods. This includes the effects of
Save the Children’s responses on the local economy, social networks, livelihoods and the
environment (SPHERE).
Limited information was available regarding the extent to which the food aid programme met
SPHERE standards in terms of its planning and management35. Villagers reported they were satisfied
with its availability and quality, except for the quality of the rice (see ‘accountability’).
Nutrition: International standards to which the nutrition component of the response should follow
are The Sphere minimum standards and the operational guidance on infant feeding in emergencies.
In terms of the infant feeding component Save the Children have played a significant role in the
promotion and implementation of the operational guidance on infant feeding in emergencies.
In terms of the CTC component data collected show very good results. To date numbers of children
cured fall well within and above Sphere minimum standards as do defaulters. Mortality rates are
also very low and fall well within Sphere standards. It was however noted that due to screening
exclusively with MUAC, not all children with GAM may have been accurately identified. The survey
conducted in November confirmed this and demonstrated large numbers of children with WFH <-2
Z-score that were not detected with MUAC alone. Additionally some of the children enrolled in the
programme may have been diagnosed as MAM but are in fact SAM.
Positive steps have been made to initiate WFH measurements for all children identified as at risk
(MUAC 125-130mm), but MUAC screening for SAM remains at <110mm. The introduction of the
WHO Growth Standards in April 2006 represented a significant move forward in the understanding
and measurement of global child malnutrition (WHO, 2006). Following Nargis WHO standards have
been adopted and applied in the identification of acute malnutrition in Myanmar. A meeting held in
Geneva by the World Health organisation (June 2008)36 on transitioning to WHO growth standards,
concluded that the new standards would identify children at higher risk for dying from severe acute
malnutrition at an earlier stage, making them easier to treat at a lower individual cost. Based on
evidence presented they concluded that current cut-offs should be reviewed and revised as follows:
SAM - MUAC < 115mm; MAM - MUAC ≥ 115 mm & < 125 mm, for children from 6 months of age.
Key people interviewed during the evaluation have highlighted the ongoing research led by the
34
Post-distribution monitoring data had been sent from areas offices to Yangon from August to early October
for compilation and analysis but area teams knew little about what this analysis had revealed or how it had
been used.
35
The teams have now been disbanded and documentation and key informants were not available.
36
Transitioning to the WHO Growth Standards: Implications for Emergency Nutrition Programmes IASC
Nutrition Cluster Informal Consultation, Geneva, 25-27 June 2008, Meeting report.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 27
nutrition cluster to review anthropometric cut offs but in the mean time, these international
recommendations could be applied
The nutrition survey in November was an important tool to assess the nutrition situation. The results
have been analysed and presented in line with internationally recognised SMART methodologies.
International guidelines on the treatment of acute malnutrition indicate systematic provision of
vitamin A, measles vaccinations, antibiotics and antihelminthics. Within the CTC programme vitamin
A and measles vaccines were not provided due to government restrictions. Similarly Save the
Children has not been able to distribute vitamin and mineral supplements as originally proposed due
to restriction. The MOH and UNICEF have reportedly distributed micronutrient supplementation
including B1 distribution in 13 townships though no documentation was seen for this.
Health: Significant training/orientations were provided to the staffs (including doctors, nurses, and
health promoters). Broad programme objectives and operational plans were provided to the staff at
the outset of the response. Orientation on communicable disease surveillance and health promotion
was provided (WHO standard format and reporting), alongside targeted support on health
promotion, appreciative community mobilization, safe motherhood, newborn care, universal
precautions for doctors and training on psychological care. Such training helped provide
professionally sound services to the communities.
The beneficiaries were found to be satisfied with consultation, treatment, and health education
sessions. There appeared to be good community engagement and information sharing about the
visiting schedule for the mobile team in the villages and there was generally good feedback on the
location and organisation of the clinics.
Communicable diseases surveillance and reporting to WHO was done on a weekly basis which is
satisfactory.
Community mobilization on providing health care including referral during the absence of mobile
medical team was a gap which was not explicitly addressed throughout the programme period. At
times when the Save the Children service was not available, for emergency cases, villagers either
used the local health facility (where this existed) or private providers although there were significant
cost and time implications to this. Non-emergency cases waited for the mobile team to return to the
village.
A pharmacy was established in the area offices to supply the drugs to the mobile medial teams.
Though the drugs were separated and temperature was maintained using fan, limited procedures for
drug storage were maintained.
WASH: One of the first activities of the WASH team was to pump out ponds which had been
inundated by saltwater. While this was an appropriate course of action to take, in the villages
visited, success was very limited even after 2 or 3 attempts. Despite this, it still provided water for
hygienic purposes. The issue of access to potable water was then addressed through the
construction of temporary water catchments which was both technically sound and much
appreciated by communities. The only issue that arose from this was the decision to vary the size of
the tanks rather than to focus on the provision of large tanks which stored significant amounts of
water. One of the reasons given for the small size was lack of catchment areas which could have
been overcome by using tarpaulins to create catchments. The use of water catchments was
bolstered in some areas by the establishment of water treatment plants from which water was
distributed to villages by boat. This was a technically sound response although alternative technical
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 28
solutions37 might have yielded greater quantities of water (although they might have had
implications for timeliness). In response to the water shortages as a result of the dry season, the
decision to use a reverse osmosis system
was a technically appropriate one although
there are issues with the sedimentation
tank which should be addressed to ensure
that it is effective.38 In light of these it is
recommended that the design is reviewed
and that a non-continuous flow used in
addition to treatment with aluminium
sulphate.
Water quality analysis is a major gap in the
response. One of the objectives of the
programme is to ‘improve access to safer
water...’ but it is impossible for the team to
prove this given the paucity of monitoring
trips. An assumption can be made based on
the lack of major water-related disease
outbreaks but there is no direct qualitative
evidence that can prove it. In Labutta, it was
reported that there was no culture media
(to verify the level of faecal coliform) due to
the difficulty of stocking it and in Maw’gyun
a similar problem was reported in one of
Picture 5: Save Children Water Collection Tank
the water treatment plants which had not
had culture media for the 4-weeks prior to the visit. In the locations visited, bacteriological analysis
was not carried out at household level or at the borehole constructed by Save the Children although
the team say that these have been carried out in the past. It is strongly recommended that spot
checks are carried out in addition to more systematic monitoring at each of the different sources of
water used by the community (from source to household).
In the villages visited, the ratio of numbers of latrines per person is above the SPHERE standard39
which was achieved with the participation of the community. Importantly, in some locations there
was evidence of strong collaboration between Save the Children and other agencies to reach the
SPHERE standard.
Education: Save the Children is fortunate to have a committed and dynamic education team in the
field and staff were found to have a sound knowledge and experience of key educational concepts
bought from the pre-existing development programme, however, Save the Children has struggled to
provide sufficient training to these staff during the emergency and there was found t o be
insufficient time given for capacity development in areas such as ‘education in emergencies’ and
working with communities. At the beginning of the emergency, training was conducted on the INEE
37
An alternative option to provide clean water (but which would not have solved the problem where there
was salinity) was to use Oxfam tanks of 95 m3 and undertake the flocculation/decantation directly in the tank
which would have doubled or even tripled the production of the WTP unit provide by the Swedish team.
38
Based on the description of the system, a continuous flow sedimentation tank is being used which,
considering the turbidity of the stream and the design of the tank will not allow an adequate reduction of the
turbidity which will result in turbid water entering the RO and will reduce the output.
39
SPHERE standard: People have adequate numbers of toilets, sufficiently close to their dwellings to allow
them rapid, safe and acceptable access at all times of the day and night. One of the indicators suggests a
maximum of 20-persons per toilet.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 29
standards which programme officers attended but it was not always clear how they were expected
to apply them in their work without support or clear expectations and many were understandably
unable to apply the learning in their programmes as the issues were new to them.
The emergency provided an opportunity for staff to advance themselves, so for example an outreach
worker in the development programme become a PO in the emergency programme which sent an
important message about valuing skills but capacity development should have also been carried out
to ensure staff could perform in their jobs. A capacity assessment could have been carried out by
month 4 or 5 post-Nargis to develop simple targeted training. Relevant development needs that
education field staff identified included management skills, accounting skills, basic knowledge of
ECCD, and skills to work with community.
Regarding the technical merit of the teacher training, the evaluation is unable to make a judgment
as this activity was not ongoing at the time of the field visits. In schools where the teachers already
had some capacity, the training appeared to have affected positive changes at least in the
appearance of the classroom. Where teachers lacked capacity prior to the training, it’s difficult to
judge what impact there had been. Follow-up from the initial training appeared to be patchy: In
Labutta and Middle Island interviews with teachers revealed that there had been no follow-up; in
Mawlameigngyun, there had been one instance of follow-up immediately after the training.
The Operational Guidelines was initially an important project document for the Educational
Programme and provided an excellent gap fill at the beginning of the emergency but it was not
subsequently updated. As a result of this, the targets, data and budget are out of date and
inaccurate; for example in Thingagone the project document showed that no schools were targeted
for repair and that there were no plans to build temporary schools whereas 20 were eventually
supported. Field teams appear to follow Yangon’s instructions with very limited space for inputting
local-level analysis which is part of a broader failure to decentralize decision-making. With
appropriate support and training, it is recommended that more responsibility should be delegated
down to field level and that teams should be given a greater role in supporting analysis and be more
involved in decision making.
6.2 Recommendations
It is understandable that staff capacity is limited given the large number of people recruited into
Save the Children and the rapid scale-up, however, there is an important responsibility to find
creative ways to fill this gap so that all members are able to perform in their jobs and deliver
quality programmes. From November 2008 the expectation has been that 3% of every budget is
put towards staff development. A needs- based training and mentoring scheme should now be
developed, with the support role of in-country advisors agreed across sectors;
For the reverse osmosis water treatment system, it is recommended that the design is reviewed
and that a non-continuous flow used in addition to treatment with aluminium sulphate (see
footnotes for explanation);
The WASH team should ensure that a systematic water quality monitoring system is in place at
each of the different sources of water used by the community;
Save the Children should change MUAC cut offs to 115cm for SAM and ensure WFH screening
for at risk children.
7. Accountability
Summary: Against an accountability framework of (i) provision of information (ii) participation (iii)
monitoring, and (iv) feedback/complaints, the Save the Children programme scored mixed results.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 30
Programme Staff worked hard to build links with communities and the development of village
committees has played a central role in the provision of information and the delivery of the
programmes, albeit with some limitations in the participation of the members. Monitoring systems
do exist, but these have historically prioritised the collection of quantitative data over qualitative
data, have suffered from a lack of co-ordination, and have often lacked sufficient resources to
permit regular data collection. That is not to say that good practice does not exist, and Save the
Children’s willingness to host the HAP/Sphere post and the establishment of an innovative
accountability mechanism for the food distribution programme demonstrates a commitment to
strengthening accountability. It will be in the ability of the team to continue to rationalise and
strengthen systems and roll out good practice to the broader programme that will see accountability
ambitions achieved in the future.
7.1 Main Findings
Cross-cutting Issue - Monitoring and accountability: After an understandably slow start, there has
been significant time and human resource investment in monitoring which has seen the
development of an output tracker and more recently, the trialling of an outcomes matrix. While this
has met the requirement of upwards accountability to donors, its contribution to programme quality
and learning is more problematic.
There has been a general lack of acceptance of this system by many of the sectors which has seen
significant duplication of effort. The difficulty which many members of the evaluation team
experienced in obtaining monitoring data suggests that there is a level of confusion and possibly a
lack of robust data. The lack of qualitative data, although partly addressed by the outcomes matrix,
is still an area that will need prioritisation if programmes are going to start building on the strong
relations they have with communities.
Several of the sectoral evaluation team members noted a lack of staff to adequately carry out basic
monitoring functions40. In saying this, the evaluation team is mindful that Save the Children has a
large team, but it also has a huge multi-sectoral programme and from a programme quality
perspective, the inability of field staff to regularly visit programmes makes it extremely difficult to
gather sufficient evidence to demonstrate that programmes have achieved their desired outcomes –
that’s not to say that the programmes haven’t done so, it’s just that monitoring systems have failed
to capture this. With an eye on the future, it’s important to note that with planned decreased in
programme size, and with the launch of the outcomes matrix there is a good opportunity for
monitoring to catch up with programming.
Cross-cutting Issue - Child Participation: Children have been involved in Save the Children activities
at a low level. The main involvement is through consultations for specific purposes including needs
assessments for various programmes and identification of violence and abuses in communities for
child protection programme, for example. The evaluation found no specific involvement of children
in decision-making process, planning, or implementation of activities. Children said they voluntarily
helped with school recovery (e.g. drying their books, cleaning classrooms, and carrying construction
materials), building and taking care of community latrines, and take care of younger children in the
CFS when their teachers are away or take lunch break. Children said they were consulted on the
rules of the CFS (e.g. no fighting, keep toys in order, and etc.) and on what they wanted to do in the
CFS (e.g. what games they want to play, what stories they want to hear, etc.). In the communities
visited where CFS’ did not continue after Save the Children phased out, none of the children were
informed or consulted about the phase-out and/or closure of the CFS. In a few communities, there
40
In the CP sector, a PO explained that in 4-months she could only make a single visit to 12 villages out of a
total of 20 for which she had oversight.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 31
are children participating in the CFS committee but they are unclear about their roles. In a village in
Mawlamyinegyun, there is a group of twelve children sitting in the CFS committee, yet the group
members and their leader were selected by adults. According to them, their roles are to monitor CFS
rules. They are not involved in planning or initiating CFS activities.
Having said that, in the eyes of the children, they already see themselves as participating in Save the
Children programmes by carrying out the tasks allocated to them. They are eager to take up the
responsibilities and be part of the programme. In a village in Labutta where the CFS has been closed
down because the community could not afford to run it after Save the Children left, children said
they wanted to fill the teacher’s roles themselves41 if they could be trained to look after smaller
children.
Discussions with children also highlighted their lack of understanding about Save the Children's
mandate. When asked who Save the Children sought to target in its work and what was important to
Save the Children, children reeled off hygiene messages, but were unaware that they constituted the
most important set of stakeholders. In discussion with the Area team, it was suggested that the
reason for this was that the Save the Children brand had not been translated into the Myanmar
language.
Food & Livelihoods: In all areas visited committees had been established to support the
implementation of Save the Children food and livelihood programmes. These committees had been
appointed in most villages42 and usually consist of equal numbers of men and women. Committees
did not appear to have a planning, decision-making or management role, or in most cases, a role
agreed after the distribution itself. They had generally been formed to enable a quick, effective
distribution of goods at distribution points. Processes for establishing committees and agreeing
roles and responsibilities of Save the Children and committees were not consistent across areas
visited and were often inconsistent within areas.43 This is perhaps unsurprising given the changing
composition of food and livelihood teams and the speed of programme implementation, with
constraints of access. That said there clearly has been a level of confusion about the role of
committees and a lack of co-ordination in their creation and proliferation - other Save the Children
sectors had also established committees in the same villages, as had other organisations. There was
a lack of coordination within Save the Children regarding committee formation as well as with other
agencies44. With this in mind, it is strongly recommended that a senior, centrally-based staff member
is given a strategic, analytical, over-sight role across sectors to identify common approaches and
practices to community engagement and other related issues45.
It would appear that the pressure for ongoing distribution of food, cash/ in-kind materials for the
QIP and materials and cash in CFW projects has prevented Save the Children staff to date from
spending more time disseminating information, supporting committees or monitoring programme
implementation. There were no real feedback or complaints mechanisms in place (but see paragraph
below on ‘monitoring’). Committee members and villagers did not know who to contact if they had a
41
The main reason given by several communities for closing down the CFS after Save the Children phased out
there support was because they were not able to mobilize local resources to cover the cost of the CFS
especially salaries for the CFS teachers.
42
In Middle Island staff reported that if the team had concerns about the integrity of the village leader they
called for their own community meeting to elect the committee.
43
Some committees had signed agreements with Save the Children regarding their roles and responsibilities,
including responsibility for post-distribution monitoring, whereas others had no concept of their role beyond
following Save the Children instructions during distributions.
44
In some villages 4-5 committees existed. Individual villagers sometimes reported that they were members
of several committees, which has potential workload implications for them.
45
Staffs at field level do not have an oversight across regions.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 32
concern. In all villages visited, there was a resistance to the concept of ‘complaining’, an issue also
identified by the Accountability Learning Working Group and HAP representatives in Yangon.
In several villages the committees reported that a few very poor households (from 3- 5 households
in a village) had been unable to use the $50 they had received to support their livelihood recovery
because of other vital needs, such as health care for the sick. When asked if they had informed QIP
Save the Children staff they stated “We did not tell them and they did not ask us”. There was not an
understanding that concerns could be fed back through other Save the Children teams who
subsequently visited villages. For example, some committees stated that the livelihood team had not
returned to the village since the QIP and they did not inform other Save the Children teams that
visited of issues arising because it was not felt to be their concern. Information should be
disseminated about Save the Children as a ‘whole team’/ organisation, with feedback mechanisms
discussed and agreed with beneficiaries.
Save the Children has begun to establish information centres at distribution points as a means of
enabling beneficiaries to feed back their comments and issues. These were rolled out in Middle
Island from January. Unfortunately the evaluation team did not visit one of the locations where
these has been established and so the evaluation team member responsible for evaluating
accountability held discussions with key staff in Yangon when the team returned there at the end of
the evaluation.
Children’s ‘voices’ or feedback have not been sought throughout the food and livelihoods
programme management cycle.
Nutrition: Programme monitoring is in place for both infant feeding and CTC programmes and is
updated monthly. Indicators strive to achieve and are compared to Sphere minimum standards and
operational guidance on infant and young child feeding in emergencies. The team also feeds into the
output tracker though in terms of nutrition, the indicators are more indicative of inputs rather than
outputs and do not represent the quality of the programmes, only the scale. The nutrition survey
was a positive step in terms of accountability to the objectives of the programme, though in the
absence of cross analysis of sectoral programming, accurate coverage, consideration of seasonal
differences and accurate base line data, the evaluation of the full effectiveness and attribution of
these nutrition programmes on population level is difficult.
Beneficiary participation has been through the recruitment of community volunteers of which there
are 110. Initial plans were to have volunteers, but the necessity for recovery activities to be
undertaken has made this difficult. Each volunteer has terms of reference and is responsible for
screening of children, provision of support to mothers and community sensitisation, in partnership
with Save the Children mobile teams. Feedback from the volunteers indicates they are happy with
their responsibilities, though it seems they have had limited input into the programme. Some have
ideas they would like to be able to do more in the way of activities such as cookery demonstrations
and many involved in the health side would like their own community clinic.
The recruitment of these volunteers has undoubtedly contributed to increased coverage and
continuity in follow up for some women needing skilled breastfeeding support. The future of these
volunteers is unclear as they receive a nominal salary for their activities. Some volunteers report
they would like to continue regardless of salary, others are dependent upon it whilst recovery
continues.
Health: Most of the programme activities were based on the rapid needs assessments conducted by
the Save the Children team shortly after the cyclone. These assessments included the participation
of village leaders and Township Medical Officers. Design of health programme activities is far beyond
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 33
the capacity of the beneficiaries however their participation during the implementation of the
programme was found to be relatively high. Community members assisted in the selection of the
participants for health education sessions, provided space for the mobile teams, communicated to
nearby villages on services offered by the mobile clinics, helped identify people requiring medical
assistance and supported co-ordination with local authorities.
Community members were found to have a good knowledge of the programme activities including
the phase-out plan. Interviews showed widespread approval of the programme (except for the
closure) but it must be borne in mind that this is in a culture where complaints are rarely voiced,
particularly to outsiders. There was no evidence that complaints had been sought by Save the
Children staff from the users of the medical services.
WASH: The WASH Committees were considered key to the successful implementation of the
programme and have been used appropriately. Based on the model of community participation in
the development programme, such committees were established to be the main focal point for
activities in the villages. Latrines and water collection tanks were built with the materials and cash
provided by Save the Children and this was overseen and monitored by the committee. The trust
established between the WASH teams and the committees played an important part in the success
of the programme. Despite the strong linkages this created between the WASH team and
community members, it is noteworthy that there was no participation of community members in the
design of the water collection tank or latrines although there was a high level of satisfaction with the
results.
The number of staff per location was considered insufficient for the ambition and geographic reach
of the programme. For WASH specifically, there are currently 68 staff in 7 offices. For each location
visited by the evaluation team, 6-7 staff members were covering up to 40 villages. Considering the
difficult access and time that travel can take, it is not possible for staff to visit each village more than
once each month (once every two-months in some cases). While there appears to be an increase in
staff numbers and a reduction in the number of villages targeted in the transition phase, it is
recommended that staffing plans reflect the needs of monitoring in addition to programme
implementation.
An issue highlighted by the WASH evaluation but which might potentially have relevance across the
programme was that of calculating beneficiary numbers. There did not appear to have been a
distinction made between direct and indirect beneficiaries. This was highlighted in the sanitation
programme; when 3 community latrines were constructed, it was considered that all the villagers
benefited through the reduction in the spread of excreta and therefore disease. It is recommended
that the team define clearly who is truly benefitting from the projects and that this data is
disaggregated direct and indirect beneficiaries.
Education: Education interventions were implemented through an education committee. In some
villages Save the Children established a new committee for their purposes while in others they
worked through a pre-existing committee. Save the Children's interaction with the community was
through this committee. In communities with pre-existing committees, there seemed to be more
community participation in the process as opposed to locations where committees were established
anew where they were often less organized and knowledgeable about the programme. A good
example of this is a school in a village in Middle Island where a large number of community members
in a meeting did not know the CFS was transitioning to an ECCD centre; they only knew that it had
been closed. There was no evidence that children were part of discussions in any of the
communities.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 34
Nevertheless, the committees were very active in the decision-making, implementation and
monitoring of activities. All committees felt they had responsibilities, particularly in relation to
school construction. School design was discussed with the committee and decisions were arrived at
jointly. Though Save the Children uses the committees for their purpose, there is no process of
facilitating problem solving or planning with them so that the committees can fill the gaps between
what they have and what they need. This is not an issue of capacity building but more of an issue of
supporting them to identify ways to meet their needs. The exception is with ECCD committees which
are linked into the SC longer term strategy post-emergency.
Regarding monitoring and evaluation, the teams were good at collecting numbers or data in the
output tracker but there was a real lack of analysis especially at the local level. This information
appeared not to be fed back into programme. Qualitative outcome data is not collected and
constitutes a gap in monitoring.
Child Protection: In order to implement the different steps of the project, the staff needs to have a
clear understanding of the strategy as well as the activities which requires sharing information in
advance and participation in and explanation of decisions taken by the Yangon office. The prevailing
method of decision-making in the programme is top-down which tends to stifle bottom-up planning
and creativity. The lack of joined-up programme planning is best illustrated by an example of the
CFS closure – in one village visited the children learnt about this only one day before the CFS was
closed. There was no evidence that the views and concerns of children had been elicited in the
villages visited by the evaluation team.
The involvement of communities through the establishment of CFS committees provided an
important foundation for the implementation of the programme. Given that a key objective of the
programme was to enable communities to better protect children, particularly vulnerable children,
participation was a prerequisite and while this did happen for purposes of information sharing, it
rarely went beyond this and the reliance of the team on dissemination through the committee was
often flawed as interviews with village members during the evaluation suggested that information
often didn’t go beyond the committee and was rarely shared widely.
The CP training did have positive impact on the knowledge of participants about vulnerable children
- in a number of villages, committees were aware of the existence of particularly vulnerable groups
such as working children and orphans, however they didn’t feel able to take action.
7.2 Recommendations
As Save the Children moves away from emergency programming through transition to
development work, it will be essential that children are more involved in its programmes, not
only by consulting with them, but also through empowerment for children and advocacy to
communities on meaningful children’s participation;
An important step in this will be to ensure that children understand that Save the Children
mandate and mission. The translation of the Save the Children brand would be a good start to
this;
A single monitoring system that can provide quantitative information for donors and senior
managers in addition to qualitative information for sector teams would be a significant
achievement. A way to accomplish this may be to develop standard operating procedures for
monitoring systems and activities across field offices and to agree monitoring principles (each
team to monitor its own work or monitor its peers in rotation - e.g. a monitoring team of 12 and
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 35
each week 3 of the 12 staff members undertakes monitoring activities46). However, given the
complexities of achieving this, a mapping exercise of the different monitoring systems (sectoral
and M&E team) linked to improved documentation and reporting (particularly for the sectors)
would be a good first step;
While the outcome matrix represents a significant step in the right direction and will greatly
facilitate the collection of qualitative data, it will be important to ensure it benefits from the full
engagement of the sector teams;
Full advantage should be taken of Save the Children’s hosting of the HAP/Sphere post to ensure
that knowledge of both initiatives goes beyond senior staff members and the M&E team to
project level staff tasked with implementation;
It is recommended that Save the Children review how they are calculating their beneficiary
numbers and refine their monitoring systems to ensure that a distinction is made between
direct and indirect beneficiaries;
For CP particularly where the issues are complex and often new to communities, in addition to
working through established committees, Save the Children should seek to engage the
community more broadly (to the extent that local authorities permit this), to verify their
understanding of the issues and participation in the programme. The use of participatory tools
would significantly strengthen this;
Approaches should be adopted by the nutrition programme which empower communities to
adopt good nutrition practices and prevent malnutrition.
CASE STUDY 4: Evaluation against HAP Accountability Principles
The inclusion in the evaluation ToR of a proposal to evaluate against the HAP accountability
principles reflects a commendable commitment of Save the Children to pushing the boundaries of
their practice. While the current context in the delta makes an accountability framework a realistic
proposition, it is important to note that given the scale of the programme, the relative inexperience
of many of the staff, and a relatively sensitive operating environment the attainment of the principles
represents a significant challenge for the humanitarian sector as a whole.
Commitment to humanitarian standards and rights: Being a member of SPHERE and HAP, Save the
Children staffs are committed to meet the SPHERE standards and rights during emergency response.
However, it was observed that while senior staff and M&E staff had received training, information
and knowledge of those standards and rights had not been effectively transferred to the field staff
due to the large size of the team and the urgency of implementation. Consequently, it’s unlikely
that these staff can fulfill the obligations in their work.
Setting standards and building capacity: Save the Children set up an output tracking system
relatively early in the emergency response programme, however, it did not effectively establish an
accountability framework and despite hosting the HAP/SPHERE initiative, was relatively weak in
disseminating the standards amongst its field staff. Interviews with the new emergency staff suggest
a lack of awareness of standards and principles (e.g. most staff does not know Red Cross Codes of
Conduct, which was often confused with SCs own internal codes of conduct and principles).
Communication: Community members were often not well informed enough about their
programmes (i.e. they just knew when the staff arrived in their villages and asked them to make the
necessary arrangements for the activities). In general, communication was top-down from Save the
46
Currently there is one M&E officer in each township, who, because of time pressures tends to collect
‘success stories at the village level’.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 36
Children with little space or opportunity for bottom-up feedback from the community.
Participation in programme: It was observed that community participation in carrying out activities
was quite limited. Although management committees were formed with community members, Save
the Children field staff carried out most of the activities and made the majority of the decisions
without consultation with the committees. Interviews with Save the Children staff revealed concerns
about the capacity of the committees to adequately engage in the programmes given their limited
capacity and time constraints.
Monitoring compliance: Save the Children appointed the Monitoring Officers at each township to
monitor the programme who were involved in output tracking (i.e. numbers of beneficiaries and
activities). Due to the urgency of the emergency response, they were unable to conduct regular
monitoring trips and instead focused on spot-checks and field visits where they collected case
studies (which were mostly success stories). Overall, while efforts were made to seek to monitor the
programme, the system used was relatively weak in monitoring effectiveness and there was no
system established for acting on the findings to improve programmes. Some concerns about doublecounting of beneficiary numbers were reported to the Save the Children team.
Addressing complaints: Save the Children has started collecting community feedback and complaints
as part of a pilot programme in Yangon Division in July 2008 but it was not successful due to a lack of
resources. However, with the provision of extra staff, this mechanism was successfully resumed in
Middle Island in January 2009 and there are plans to replicate it in other areas in the future.
Recently another attempt has been started at Myaungmya to establish a children’s feedback and
complaints system by CFS and ECCD.
The following recommendations are made to strengthen accountability mechanism in the transition
phase of the programme;
A stronger inter-sectoral coordination mechanism should be developed between the
programme teams, M&E unit and accountability team to share information, develop holistic
monitoring systems and improve programme quality and accountability. This could be
facilitated through the creation of a strategic oversight role, responsible for working across
sectors to identify common approaches and practices to community engagement and targeting;
A mechanism that systematically captures lessons and channels these back into programme
teams for follow-up should be developed and trialled by the accountability team;
Efforts should be made to further develop staff knowledge and understanding of quality and
accountability by cascading the knowledge down to field teams. Those who attended the
Sphere ToT should be tasked with training internal staff;
The scale-up of the Information Centre activities is welcomed and it is recommended that these
are rolled out to other sectors with a view to making a step change in the accountability of the
programme to project participants across all sectors.
8. Effectiveness
Summary: In responding to the destruction wrought by cyclone Nargis on the lives and livelihoods
of communities living in the delta, Save the Children was successful in using in-country and
organisational capacity to deliver a proportionate response (meeting the needs of over 500,000
people) in an extremely challenging environment. In evaluating the achievements of the
programme, it is important to recognise the extraordinary vision and ambition that the team
demonstrated in going far beyond their comfort zone in establishing a geographically dispersed,
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 37
multi-sectoral programme. It is the heritage of these same decisions that has provided some of the
greatest challenges to the team; that of ensuring sound strategic planning, adequate quantity and
quality of staff, ensuring internal co-ordination and having sufficiently robust systems in place to
adequately monitor programme quality.
8.1 Main Findings
Food & Livelihoods: Save the Children’s rapid response to the cyclone-affected population’s food
needs is impressive, particularly given the many challenging operational constraints in Myanmar in
general, and the cyclone affected area in particular. These constraints include the size of area
affected47, difficulties with access (both the initial negotiations undertaken to obtain humanitarian
space with the authorities, the initial access to the delta by international humanitarian staff, the
geographical remoteness of programme locations with limited infra-structure (roads,
telecommunications) and the lack of reliable secondary data48.
All villagers met expressed their appreciation for the food delivered, although some had concerns
about the quality of rice provided. The food distributions had ensured their household food security
so that income earned could be used to cover other essential needs in the months immediately postNargis.
Post- distribution food monitoring was undertaken from August to early October but reports were
not produced until the end of November towards the end of the large scale food distributions.
Information collated could not thus have been used to feed back into programme implementation.
Some information is available regarding levels of malnutrition in the areas covered49 but it is
uncertain whether any cross-sectoral (ongoing) analysis has been undertaken regarding the impact
of the food distribution on the nutritional status of beneficiaries or, indeed, what impact breaks in
the WFP pipeline may have had.
Save the Children’s Livelihood QIP was designed as a rapid impact project to support vulnerable
people affected by Cyclone Nargis “to begin to fulfill their right to livelihoods and to reduce
dependence on food assistance through the rapid replacement of essential livelihoods assets50.” The
rapid distribution of cash or in-kind grants was reported by all key informants met as remarkably
successful in promoting vulnerable households’ early livelihood recovery. The extent to which the
project has enabled an understanding of vulnerable communities’ coping strategies is as yet unclear
(See section ‘relevance’) given that most of the activities undertaken to date have been distributions
of cash/productive assets, with relatively little analysis or monitoring of community livelihood or
coping strategies.
Discussions are now ongoing in the Save the Children Emergency Response team about the
transition from emergency response to recovery and beyond. To date the emergency livelihoods
programme has consisted of rapid distributions of cash or in-kind materials. In most areas visited
these programmes were implemented by teams deployed from other sectors, after an initial rapid
47
The cyclone affected area is 23,500 square kilometers of flat land, mangrove swamps and tidal estuaries
divided into islands and peninsulas by many large rivers and smaller water ways.
48
There are great variations between official and other reported economic and social indicators which mean
that there is generally a poor understanding of whom and where poor children are. For the cyclone- affected
areas in particular there has been limited information from the field because of a lack of access and very poor
communications.
49
Save the Children has information where it has CTCs regarding the number of women and children
presenting with malnutrition. Nutritional surveys have recently been undertaken in Mawlamyainegyun and
Ngapudaw in November/ December 2008.
50
Emergency Cash Transfer Programme, Save the Children, January 2009
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 38
training, who were then disbanded. If the decision is made for the livelihoods programme to be
progressed into longer term recovery and beyond, the Livelihoods team will need to carefully
analyse and plan strategically the resources needed, including skills sets of staff51.
Nutrition: In developing countries (non emergency situations) a non breastfed infant <6 months old
is 14 times more likely to die than an exclusively breasted child52. Counselling on breastfeeding has
been identified as one of the interventions that has the greatest potential to reduce the burden of
child morbidity and mortality53. The Save the Children IFE programme should be regarded as one of
the strengths of the emergency response and lessons should be learnt and applied for future
interventions both in terms of emergency, transitional and development programming. The
breakdown of activities within the IFE programme whereby both general and targeted skilled
support are given has resulted in a comprehensive model package of interventions adherent to the
operational guidance for IFE. At overall response coordination level Save the Children has been
effective and valued in leading the working group on IFE and a key activity from this process has
been promotion and translation of a joint statement and operational and successful sensitisation of
the logistics cluster.
At community level in all villages visited, there is a visible presence of posters with the 4 basic
messages for IFE covering both CFS and the wider community (though it was noted though that
these are not being used across sectors and are therefore only effective in IFE programme areas),
and all mothers and caretakers spoken to were able to demonstrate knowledge on best practices for
infant and young child feeding.
Undoubtedly the effectiveness of IFE interventions is dependent upon other sector interventions, in
particular other nutrition activities, health, food and livelihoods and WASH. The initial planning for
the IFE programme planned to link mothers with general food distributions. This was not observed in
the field and some problems have been identified such as low coverage, poor quality and irregular
supplies. A challenge identified by many field teams and some community volunteers was the
absence of regular referral and availability of treatment for malnourished mothers and support in
complimentary feeding in some areas which has implications for the effectiveness of IFE
programmes. Save the Children has detected cases of malnourished pregnant and lactating women
but following a decision by UNICEF to not treat these women, have not been admitting
malnourished PLW to date and there doesn’t seem to be SFP in the other operational areas. In the
absence of locally available means to support malnourished pregnant and lactating women, Save the
Children should advocate strongly to UNICEF for provision of commodities to extend to this group in
addition to children under 5 years old.
The Save the Children CTC programme is also impressive, especially given the absence of
programmes to treat malnutrition pre Nargis. Save the Children has demonstrated good outcomes
for the children enrolled in the programme despite restrictions on systematic treatment (detailed
under technical merit) which could have compromised the effectiveness of the intervention. As well
as systematic treatment restrictions, the Save the Children CTC programme does not have
stabilisation centres. Children with complicated malnutrition or children under 6 months with
complications are currently referred to MOH hospitals but capacity and quality of care delivered is
not measurable by Save the Children in the context. Initial plans to support to the MoH for one
stabilisation centre were not realised due to lack of interest/willingness on the side of the MoH. The
impact of the lack of stabilisation centres is difficult to assess as few cases of complicated
malnutrition were identified.
51
Community mobilisation, empowerment, risk reduction etc.
Black et al, 2008, Lancet: Maternal and Child Under nutrition, January 2008.
53
Bhutta et al, 2008, Maternal and Child under nutrition 3, Lancet, January 2008.
52
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 39
A nutrition survey was conducted in November which recommended a targeted scaled-up approach
to the treatment and prevention of malnutrition in children 6-59 months. This was based on findings
that Infants aged 6-18 months are the most vulnerable, with most cases of severe acute malnutrition
occurring in this age group. Recommendations included increased CTC programme coverage and
further integration between the CTC and medical teams, both Save the Children and DoH, to address
high morbidity in the sampled population54.
Coordination has been noted as problematic, within nutrition, the IFE and CTC have been perceived
as separate components that do not always overlap though CTC staffs are trained in some IFE
components and IFE staffs have some understanding of malnutrition through MUAC screening.
Furthermore tensions between nutrition and health sectors are well documented and clearly led to
silo approaches which may have impacted the effectiveness in the initial response. However it is
worth noting contextual historical difficulties in joined up programming due to restrictions and
therefore the previous lack of experience in integration. The lack of appreciation of the dependency
of CTC on the health intervention, particularly in the absence of stabilisation centres was evident at
the highest levels during the initial intervention and was demonstrated in the shifting management
of emergency health under the development team in August/September. This will have
compromised the quality of care. There has been development and the integration of CTC, IFE and
health to CHANT – community health and nutrition team’s was a positive step. Where CTC is
present in a village without health referral to INGO initiatives or MOH facilities takes place.
Assumptions on joint programming between health and CTC teams for the provision of routine
medications for OTP were problematic in the initial phase. This was addressed through coordination
with the health team highlighting the importance of integration.
In addition to problems with health integration, the extent of integration between nutrition, WASH
and food and livelihoods programmes is poor. Adequate food security responses are a prerequisite
to selective feeding programmes such as CTC. Likewise nutrition input into food and livelihood
interventions is essential. An example of this can be seen in polished rice based food aid and
absence of micronutrient supplementation which has reportedly (though anecdotally, no written
information available) increased the incidence of vitamin B1 deficiency. Positive reports of
integration with child protection were received. This may be due to the overlap in coverage. Child
protection teams have been reportedly effective in referral of children and caretakers to the
nutrition team.
Health: Save the Children’s emergency health interventions started from 18th May with the team of
doctors, nurses, registers and public health promoters using the floating mobile clinic. This approach
was later converted to a community-based one based in clinics and using small boats for
transportation of medical teams and drugs. The programme was successful in implementing its core
objectives. During the life of the programme it was able to treat half of the targeted population,
though the baseline data to estimate the target figure cannot be considered reliable given the
country context.
Community members requiring immediate health care including children and pregnant women were
targeted by providing treatment of common diseases such as acute respiratory infection, diarrhea,
malaria and dengue fever using standard protocols and essential drugs. The programme followed
the WHO morbidity and mortality reporting formats and disaggregated data is limited to listed
diseases patterns for under-5 children and above-5 children and adults. The programme made
referral arrangement for serious cases to township levels. The programme was able to provide
referral service to 12% of targeted referrals (177/1440) at the point of evaluation.
54
Nutrition survey, Save the Children, November 2008
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 40
While the work of other actors in the health sphere (including government) make it impossible to
attribute health improvements solely to Save the Children, given the near total absence of the preexisting health services in the affected areas immediately after the cyclone, there is no doubt that
the services provided essential support to affected communities from the outset and while the reestablishment of government facilities might have led to some duplication in recent months, this
should not diminish the important contribution of the programme.
WASH: In seeking to find appropriate solutions to providing potable water and sanitation to cyclone
affected communities spread across a huge geographic area, to a standard which met the
requirements of SPHERE and to the satisfaction of many of the communities benefitting from the
services, the WASH response can be considered to have had significant impact. It is in the area of
timeliness that the effectiveness of the response is challenged and in the early days and weeks after
the cyclone, much of the credit for the survival of communities must be put down to their own
resilience and the phenomenal local support that was received by many of the survivors. That the
dry season WASH response was late was a disappointment, particularly given that Save the Children
had been instrumental in sectoral planning.
Despite also suffering from a slow start, the sanitation response should be considered a great
success, providing latrines that were welcomed by community members and underpinned by public
health promotion which may have precipitated significant changes in attitudes and practices55. With
an increase in staff numbers, and a decrease in activities planned for the transition programme, Save
the Children should be well placed to address weaknesses in programme monitoring which will
provide the robust data necessary to unequivocally demonstrate the effectiveness of the
programme.
Education: The education intervention is considered to have been effective in meeting the priority
needs of the communities. In most villages the educational infrastructure, teaching materials and
resources were totally destroyed by the cyclone.
In working through local committees, establishing temporary and semi-permanent school structures,
basic furniture (in some cases) and latrines, Save the Children provided the basic infrastructure and
equipment necessary to allow schools to resume activities, and as the programme progressed, a
growing number of schools have benefitted from being built back better.
The interventions were effective in re-establishing routines. Most children received student kits and
most schools received school/teacher kits. As there were not enough game boxes available for all
schools, schools from poorer areas were prioritised. In many of the communities Save the Children
support did not restart educational activities solely and in many cases communities through their
own efforts, played a vital part in restoring their children’s education. Save the Children did (or
during the transition programme, will) enhance the quality of the built space and capacity of
teachers, making the schools better learning environments.
Training was generally provided to teachers who were from Save the Children-supported schools but
in some areas other teachers attended also, based on requests from township education officers.
Concerning the capacity of teachers, it is unclear what impact Save the Children had on the teaching
and learning processes overall - in schools where teachers were already good practitioners, it
appeared that new skills were being applied, but this certainly wasn’t the case in all schools visited.
55
The lack of a baseline means that this cannot be verified although interviews suggest that there is certainly a
greater knowledge of hygiene issues and uptake has improved.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 41
Though there is an emergency education strategy, it appears to be more of a document for Yangon
purposes than the field and it’s difficult to use this as a tool to measure effectiveness as the targets
are out of date and the timeframe covers 12-months though the overall strategy is for 24 months.
The link between the projects and the strategy is unclear. In reality, the education work appears to
be the sum of various projects without an overall plan. For the evaluation, only 1 of the current 5
project proposals was shared with the evaluation team. When reading the project documents, it is
unclear what has been done, what is planned when, and what changes have been made. The gap in
planning and implementation requires urgent attention.
Picture 6: Save the Children-supported school, Middle Island
Child Protection: Although many of the CP team lacked training and experience as a result of the
rapid scale-up, their commitment and willingness to learn has contributed to the success of the CP
programme and they have given a tremendous amount in terms of their own time and energy. The
project has changed and improved the situation of the communities in affected areas through the
implementation of CFS’ as children were able to overcome some of the trauma they suffered in a
safe and friendly environment. This doubtless made a significant contribution to the psychological
recovery of parents and communities more broadly given that children play such an important role
in Myanmar culture.
In the first months of the response many of the initial programme objectives have been reached: CFS
were established and provided support to parents and children; in most villages children attended
CFS; CFS Committees were formed in each of the villages; separated and unaccompanied children
were registered ; despite the rainy season and extremely difficult working conditions, there was
large geographic coverage and many villages were served. The main gap was the inability of the CFS
to access some of the more vulnerable caseload such as working children.
After the initial emergency period, the programme would have greatly benefited from a strategic
stock take – a look back to review the activities, to analyse lessons learnt, strengthen programme
and participation and adjust the programme to the changed context to ensure it is fit for purpose for
the future. This would have provided a strong foundation for the transition phase.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 42
It is the view of the evaluator that key areas for focus in the future are as follows; to strengthen CFS
committees to better protect children; to seek greater community involvement in the programme;
to develop child participation methods which seek to capture children’s feedback; to provide greater
support to the process of handover of CFS to communities; to deliver CP awareness messages to
communities and analyse the impact these have; to strengthen staff capacity building through
targeted training; to ensure a focus on the most vulnerable children; to look for options for the best
interest of the child rather than a more basic solution; to structure and develop referral with other
sectors and limit the risk of institutionalization.
8.2 Recommendations
Given that livelihoods work to date has been conducted by teams from other sectors, if the
decision is made for the livelihoods programme to be progressed into longer term recovery,
Save the Children will need to carefully analyse and plan strategically the resources needed,
including skill-sets of staff;
Ongoing development of staff capacity will be vital in the coming months as the livelihoods
programme moves away from distributions to a more ‘social development’/ community
mobilisation role. Given that teams are relatively inexperienced, ongoing training programmes/
mentoring of staff in the field should be a priority and be needs-led. Rotation of staff between
field offices to share knowledge and learning and support from technical specialists is
recommended;
Integration of (i) CMAM and IYCF, and (ii) integration of health and nutrition to form a
comprehensive package of care that addresses primary health care, treatment of acute
malnutrition and infant and young child feeding needs which can be linked with the work of
other sectors in the overall country plan is recommended;
Design and implementation of an advocacy campaign, focusing on policies that affect
nutritional status. This should include promotion of optimal infant and young child feeding
practices and incorporation of CTC protocols into IMCI protocols. This campaign should also
highlight the need for country wide nutrition surveillance to guide nutrition and other sector
interventions;
Save the Children should consider assessment of (if not already covered in survey) and
expansion of CTC programming to non CTC villages with high levels of GAM in light of the
hunger season;
Save the Children should consider further assessment and mapping of malnutrition and infant
and young child feeing practices in Save the Children catchment areas to guide appropriate
programmes in addressing both the consequences and underlying causes of malnutrition;
Save the Children should advocate for and if possible gap fill for the treatment of malnutrition
in pregnant and lactating women;
As the emergency phase comes to an end, it is recommended that Save the Children develop
phased ‘exit’ plans across all sectors for relevant villages and inform villages well in advance
accordingly.
9. Critical Decision-making
This section gathers together findings from across the evaluation and speaks to one of the core
questions posed in the evaluation terms of reference. Rather than seek to review the array of
decisions that have been made over the last 10 months, many of them under intense pressure,
priority has been given to decisions that have either had significant historical impact on the
programme, or those which have been consistently raised during interviews and that have
implications for the effective running of the programme in the future.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 43
Summary: The context of decision-making, particularly in the first 3-months, was one of a dynamic
environment, where NGOs were working with very limited information in a catastrophe of
unprecedented proportions. Such an environment necessarily required risks to be taken and for
decisions to be made based on imperfect knowledge. 9-months on, there is far greater stability, and
while significant needs remain, Save the Children now have an opportunity to reflect on and make
changes to systems and structures in order to ensure they are fit for purpose for the future of the
programme. In particular, it is recommended that management structures are reviewed and
strenuous efforts are made to ensure internal cross-sectoral co-ordination.
9.1 Main Findings
Preparedness: Cyclone Nargis was a 100-year event, and the humanitarian access that was
subsequently negotiated with the Myanmar authority’s post-Nargis gave humanitarian agencies an
unprecedented reach into the affected area. While the Save the Children Alliance Operational
Review documents the lack of adequate preparedness planning by the country team, it could be
argued that no agency could have been prepared for the scale of response required. That Save the
Children did launch a proportionate response should be considered a significant achievement.
Picture 7: Save the Children interventions in Labutta
township
56
Programme Scale: One of the challenges of
being the first on the scene was the issue of
scale. Motivated by the humanitarian
imperative, the basic message given by
senior Save the Children management to the
teams was to expand the programme
geographically until they encountered other
agencies – the limited capacity in the sector
tied to the timely deployment of the Save
the Children teams ultimately saw them
spread out across 14 of the 15 most affected
townships. This decision to go to scale was
justified on several levels – humanitarian
need, limited government capacity, lack of
humanitarian preparedness. On day one,
huge swathes of the delta had been
flattened by Nargis – however, over time,
communities in different parts of the delta
have recovered at different speeds – the
field trip from Middle Island through West
and then East Delta showed a stark contrast
in the recovery of different parts of the
Ayeyarwady – with a strong justification for
deepening the assistance in the west where
many communities remain under tarpaulin
compared with those in the east where
recovery has clearly progressed at a faster
pace56.
In Pyapon township interviews with community members revealed that they considered that they had
recovered 90- 100% of their livelihoods while in Labutta and Middle Island, the figure was between 10–30%.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 44
The second dilemma thrown up by scale is the capacity of the team to deliver the programme.
Although Save the Children should be very much congratulated on its massive scale up to meet the
needs of very vulnerable communities, there was an inevitable trade-off between programme scale
and Save the Children’s capacity to deliver high quality programmes in the days, weeks and months
following Nargis. The initial response benefitted from a highly motivated national staff team
redeployed from the development programme but as the humanitarian response expanded and
grew in complexity, hundreds of new staff were rapidly recruited to deliver the programme. While
this is clearly a success of national capacity, there were risks inherent as a result of the large number
of staff that had very little knowledge of Save the Children or the NGO sector. While basic training
was provided to many of the new staff and Yangon-based sectoral advisors and Regional Managers
have made field trips, the decision to not base experienced international staff full time in the delta57
where they could have provided significantly more support and training was a curious one. Senior
managers in Save the Children expressed concern that a decentralised model would have required
additional staff, which would likely have been the case (and with that comes issues of recruitment
and retention), but it is the view of the evaluation team that the benefits would have outweighed
the costs in terms of additional support provided.
Management structure: Despite some early experimentation with different structures, the
programme settled on a management model which placed the sectoral programme managers as the
primary management line. This decision was justified by the limited capacity of field staff and the
constraints on international staff access to the delta and also by the benefits to be gained from the
advantages which include advantages of having national managers which include knowledge of
context, language skills etc. While this has been tempered by the strengthening of the geographic
line (with area and regional managers) with operational budgetary responsibility, the decision to
retain programme budget responsibility with the Yangon-based sectoral managers is considered by
the evaluation team to be a blockage to the effective functioning of line management as it has
served to mitigate against decentralised planning and ownership of programme decision-making by
field-based managers. While recent efforts have been made to elicit the views of field teams through
the establishment of regional meetings, these have achieved only limited success in allowing senior
field staff to have a reach into strategic decision-making. Area and regional managers reported an
understandable frustration with the status quo where they have significant responsibility yet have
very limited ability to influence decisions.
A secondary impact of this decision has been the relative absence of inter-sectoral co-ordination
which the evaluation team consider to be a significant constraint to joined-up programming. It is
understandable that programme planning and design was initially centralised, undertaken by sector
leads and advisors in Yangon58 however, the impact of maintaining the status quo 9-months later is
that considerable sectoral silo’s have been built up where linkages between sectors that would
greatly benefit programme quality are not being made.
It is important to note that a review is planned by the Save the Children team to define a new
structure for the programme by July 2009.
Strategic Planning: A pragmatic approach which prioritises humanitarian action above strategic
planning is understandable at the outset of a large emergency response. At this stage, the approach
taken by Save the Children of prioritising response was the correct course of action to take.
However, it is important that at an appropriate time there is a clear strategy in place to guide the
programme and that a clear goal and objectives that nest beneath this are defined both at an over57
The HR team confirmed that from the perspective of travel permission, international staff could have been
field-based from August 2008.
58
Senior management justified this on the basis of having new, relatively inexperienced staff in the field.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 45
arching level and for each of the sectors. Ideally this should be cascaded down to area offices to
guide the work of the field teams there. An emergency strategy was developed four-months into the
response in September 2008, but is now out of date and of limited value in providing direction to a
programme of the size and complexity of the Nargis response. While the broad ambition remains
valid, in order for it to be relevant to the transition programme it will be necessary to significantly
strengthen and update it.
Disaster Risk Reduction: While the lack of preparedness of the humanitarian community in
responding to Nargis can be justified in part given the 100-year severity of the cyclone and the
history of travel restrictions placed on NGO staff, there will be no such justification in the future.
While aspects of the humanitarian programme have made a significant contribution to the capacity
of communities to cope with future hazards, the Save the Children DRR strategy was finalised on the
last day of the evaluation, and hence it was not possible to review these as a coherent whole. What
can be said, however, is that it will be essential for Save the Children to ensure that plans for DRR
are sufficiently robust and can be implemented across the programme in order to identify potential
hazards, build community resilience, and reduce vulnerability.
9.2 Recommendations
59
While there was a draft emergency preparedness plan for Myanmar prior to the cyclone, this
was considered of limited practical value in guiding the humanitarian response. A document
that provides appropriate guidance to operational emergency response and which builds into it
the changed scope and nature of the Save Alliance programme and contextual changes as a
result of improved humanitarian access should be developed and signed-off59.
While documentation about the transition programme suggests there has been some
adjustment in the focus of the programme to areas that are most in need, the relative fast pace
of recovery in the east compared to that of the west provides a strong justification to make
more significant changes to the geography of the Save the Children response to ensure that
unmet needs in Middle Island and West Delta are prioritised. This would also allow an
opportunity for rationalising what is a geographically diverse programme and by providing
greater focus for the programme, could facilitate stronger sectoral co-ordination.
It is recommended that a skills audit is undertaken and that this coupled with a clearly
articulated transition programme strategy forms the basis of decision-making about staff
numbers and about the location for sectoral managers and advisory staff. The decision in
December to create a small number of field-based livelihoods advisors was a good one and it is
recommended that this is replicated for other sectors if the skills audit justifies this course of
action.
The transition programme provides an opportunity to rationalise the management structure
and it is the view of the evaluation team that a structure that places line management, human
resource management and budget management in the hands of the area/regional managers
would provide a much stronger basis for effective programming;
One of the reasons given for maintaining the status quo was limited staff experience and
capacity to manage multi-million pound budgets. If capacity is considered to be a constraint,
then this should be dealt with through ensuring that sufficiently experienced
international/national staffs are recruited into field-based management posts;
If the status quo is maintained, it is strongly recommended that greater decision-making
authority is delegated down to field teams and that an explicit shift in the balance of power is
made with area/regional managers empowered to drive strategy. In order to facilitate this, it is
recommended that decision-making, communication, M&E and reporting mechanisms should
ELT Operational Review of the Save the Children Alliance Response to Cyclone Nargis, Andy Featherstone,
October 2008.
PART 1: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 46
be refined so that programme planning, design and revision is routinely informed by the field
and systems are established that ensure two-way flow of information between Yangon and the
area offices;
It will be important in the transition period and in future long-term interventions to develop
integrated programming. Management systems and mechanisms should be established to
ensure cross-sectoral analysis and planning (e.g. joint assessments, joint proposal writing,
reporting formats across sectors to encourage staff to work between and across sectors etc).
Area/regional managers should be given greater responsibility to facilitate/enforce crosssectoral programming in the Areas under their responsibility;
Save the Children should seek to develop a strong evidence-base for its future programme by
carrying out cross-sectoral assessments to inform future programming. It may wish to consider
using a vulnerability capacity assessment to frame the multi-sectoral assessments around DRR
(hazard mapping, vulnerabilities and capacities of communities, social organisation) as well as a
stakeholder analyses to identify key actors in need, of influence, and as potential support (civil
society groups, local authorities, other NGOs). Seasonal calendars could be developed for target
areas to ensure activities are implemented at the most appropriate time;
In moving forward it will be important to update the existing emergency planning document
and develop a coherent strategy that has a cross-sectoral vision and which plots a path through
transition to longer-term development (where this is appropriate). It will be important to
ensure this plan is coordinated in particular with the development of Save the Children’s DRR
strategy and plan;
In progressing the proposed DRR programme, it will be important to ensure that lessons are
learnt from the Nargis response and that the basic concepts of DRR are understood by the Save
the Children staff and that the proposed programme cuts across sectors and builds on the
strong community links that have been established.
10. Concluding Comments
Having enjoyed unhindered access to the Save the Children programme in the delta, the evaluation
team is keenly aware of the huge task that remains for agencies seeking to support the recovery of
the communities living in the Ayeyarwady. However, the strong foundations which Save the Children
has laid at village, township and national level through its humanitarian programme provide it with a
significant opportunity to continue to deliver appropriate and effective assistance that recognises
and adapts to the changing context.
That there are lessons to be learnt from the response is not surprising; the programme has arguably
positioned Save the Children as the lead humanitarian provider in Myanmar and future success will
be predicated on the ability of the team to balance the conflicting day-to-day demands of the
programme while creating space to maintain a strategic overview of the humanitarian and recovery
needs of communities and successfully position Save the Children to best meet these.
PART 2: Evaluation of the Save the Children Response to Cyclone Nargis
Page | 47
PART 2: Evaluation of the Save the Children Response to Cyclone Nargis
ANNEX 1: Review Participants1
Save the Children Office, Yangon
Management (country, emergency, operations, regional)
Logistics
Funding
Finance
HR
WASH sector
CP sector
Health sector
Nutrition sector
Food & Livelihoods sector
M&E, Accountability
Save the Children Area/Sub-Area Offices
Deedugone Office, Middle Island
Thingangone Office, Middle Island
Laputta Office, West Delta
Maw’gyun Office, East Delta
Pyapon Office, East Delta
Village-level Community Discussions
Kan Seik village, Middle Island
Ale Thaung village, Middle Island
Kone Gyi village, Middle Island
Htaw Pone Nar village, Middle Island
Thone Gwa village, Middle Island
Ka Nyin village, Middle Island
Gyuan Gyi village, Middle Island
Mya Yar Gone village, West Delta
Bo Gone village, West Delta
Sin Chay Yar village, West Delta
Zee Phyu village, West Delta
Myit Pauk village, West Delta
Daunt Chaung village, West Delta
Yae Dwin Kone village, West Delta
Pyin Htaung Twin village, West Delta
Tha Byu Gone village, West Delta
Gant Eik village, West Delta
Tha Yet Chaung, East Delta
Mazeli Oo Tow, East Delta
Sin Ma Wei Chaung, East Delta
Shwe Bo village, East Delta
1
In the interests of efficiency, review participants are grouped by sector (in the case of the Yangon office), or
location (for Area Offices and community discussions). In each of the Area Offices, interviews were conducted
with management and sectoral staff, in each village, discussions were held with a range of stakeholders.
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Kyone Tut Tanyi village, East Delta
Kyaung Su village, East Delta
Too Myaung village, East Delta
Ta Man Gyi village, East Delta
Ka Ni village, East Delta
Kun Dyne village, East Delta
Page | 48
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Page | 49
ANNEX 2: Terms of Reference for the Evaluation
Evaluation of the Cyclone Nargis Emergency Response, Myanmar 2008
9th February – 30th March 2009
1.
BACKGROUND
Cyclone Nargis struck Myanmar on 2 and 3 May 2008, making landfall in the Ayeyarwady Division
and passing into Yangon Division before hitting the former capital, Yangon. With a wind speed of up
to 200 km/h the damage was the most severe in the Delta region, where the effects of the extreme
winds were compounded by a sizable storm surge. Some 2.4 million people are believed to have
been affected by the cyclone, of a total 4.7 million people living in the affected Townships. Official
figures put the number dead or missing at more than 130,000.2 Cyclone Nargis was the worst
natural disaster in the history of Myanmar, and possibly the most devastating cyclone to hit Asia
since the cyclone that struck Bangladesh in 1991.3
Local and international relief efforts began just after the storm hit, although the international
response was delayed while humanitarian access arrangements were agreed. Save the Children has
been operational in Myanmar (SCiM) for the last 13 years and prior to the cyclone had a sizeable
programme (but not in the Delta area) employing around 500 staff with a planned budget of
GBP4.25m for the financial year 08/09. (NB this represented a sizeable increase of 35% on actual
expenditure from the previous financial year)
In January 2009, eight months after the cyclone, SCiM employs over 1400 staff, working in 14 out of
the 15 most affected townships in the Delta with an emergency response budget of approximately
GBP26m spanning a seven sector response including shelter, food security, livelihoods, WASH,
nutrition, child protection and education 4. The scale up to the response has been massive and the
size, scale and programmatic scope of this emergency has not been seen in Save the Children since
the response to the Boxing Day Indian Ocean tsunami of 2005.
It is essential that the agency take stock of this response and conduct an evaluation of the response
to date both from a programmatic standpoint and in order to satisfy the principles of Alliance
engagement in emergencies which state the requirement for an evaluation on any ELT declared
emergency.
This evaluation will benefit from the Save the Children (SC) Emergency Liaison Team operational
review of the Myanmar response (ELT OR) and the Farringdon based Emergencies Section lessons
learned exercises already conducted, and will also benefit from a number of sector specific
evaluations already carried out in country. However this is a stand alone and independently led
2
OCHA Situation Report No. 33, 19 June 2008
3
Myanmar Revised Appeal: Cyclone Nargis Response Plan 2008 Consolidated Appeal, 10 July 2008. p. 1.
4
th
For more information refer in the first instance to the SC Cyclone Nargis emergency response strategy of 08
September 2008
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Page | 50
evaluation and as such brings a whole level of added value for the organization in terms of ensuring
accountability to our stakeholders through an impartial and robust process.
2.
OVERALL EVALUATION OBJECTIVES
This is an external evaluation with a threefold purpose of;
(a) Assessing the quality and effectiveness and technical strength of Save the Children in Myanmar’s
Emergency Response to cyclone Nargis between May 2008 and January 2009
(b) Measuring the extent to which the response has been accountable to the local needs (of children
and their families).
(c) Recommending improvements for the longer-term strategies of the response programmes
focusing on programme and management quality and accountability.
In doing this, the evaluation should respond to the following seven key questions:
1) What were the (positive and negative) outcomes of the emergency response of SCiM for
children and their families affected by the cyclone?
2) How accountable has the response been in terms of addressing the needs of children and
their families?
a. Did we identify the most relevant needs for the affected children, their families and
communities?
b. Were the needs we identified the needs prioritized by the affected population?
c. Did we do what we said we would do?
d. Was what we did enough to address the needs that beneficiaries had, given the
context and the environment (regarding funding, capacity), etc?
e. In what ways did SCiM internalize (better or worse) feedback from beneficiaries in
its regular implementation of the emergency response?
f. What were the elements of the response that benefited from extended participation
of beneficiaries and how, and which are those that need improvement?
3) How technically strong are sector-specific interventions in education, child protection,
livelihoods, health / nutrition, food security and WASH?
4) How did the way the response was managed in country affect (positively or negatively) the
quality of the programmes and operations? What were the critical management decisions
made and why?
5) Was (internal) inter-sectoral coordination effective and if not, why not?
6) In terms of preparedness (NOTE: It was agreed this would be deleted from the ToR)
a. The cyclone was heading for Burma for 3 or 4 days before landfall. What was the
impact of any contingency planning activities conducted by SCiM. This question
should prioritise internal (to SCiM) preparedness, but if time permits, contact could
be made with other agencies to discuss the issues.
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Page | 51
b. What were the most important emergency preparedness measures in the initial
mobilisation and later response period? Following areas should be considered (i)
national staff capacity including training and domestic response teams, (ii)
emergency preparedness planning, (iii) international staff capacity, (iv) specific
technical interventions in development programmes (i.e. data gathering to create
baselines, livelihoods work) (v) response systems at a national level, (vi) work with
other agencies in Burma, (vii) other
7) In terms of connectedness
a. What improvements could be made to ensure high quality implementation of the
emergencies response into longer-term strategies?
What would be the
recommendations?
b. Have key linkages between the relief and recovery phase been established?
c. To what extent has local capacity been supported and developed?
The evaluation will also apply selective criteria from the overall framework of the OECD-DAC5 criteria
in identifying the worth and merit of the interventions: The OECD-DAC framework includes
relevance and appropriateness of the intervention/s, timeliness, connectedness, coherence,
coverage, efficiency, effectiveness and impact.
For the purposes of this evaluation, which is looking at quality, effectiveness and technical strength,
the OECD-DAC criteria to be used are; relevance and appropriateness, effectiveness, timeliness and
coverage. The criteria of connectedness will be covered under objective c of the overall evaluation
objectives and will also be covered under question 7 of the seven key questions above.
The criteria of impact, efficiency and coherence will not be studied for this evaluation. It has been
agreed that an impact study, while interesting to conduct, will not be possible in the time frame
available. Similarly this evaluation will not look at the criteria of connectedness. It has been agreed
that an analysis of the broader operating environment, policy and external actors is not necessary
for this response. The criteria of coherence is rarely applied to the evaluation of single project, single
country, single agency responses. Lastly, the criteria of efficiency will also not be evaluated as it
would require a review of alternative models of aid provision, an activity that is not possible in the
time available. It has been agreed that this criteria is best looked at separately by a team consisting
of logistics, finance and administration staff. The Global Logistics rollout in April 2009, and
accompanying logistics review might be an ideal opportunity to complete this.
The team will be expected to apply SCiM’s emergency strategy as reference when making
recommendations6. It is expected that relevance and appropriateness is scrutinized in the light of
SCiM’s commitment to accountability to beneficiaries (HAP-I, SPHERE, ECB) to help SCiM understand
the best practices in being accountable to beneficiaries (children) and what improvements SC
(globally) may realistically put in place to better the quality and accountability aspect of the
emergency response.
5
LINK HERE: http://www.odi.org.uk/alnap/publications/eha_dac/pdfs/eha_2006.pdf
6
Three pillars
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Page | 52
Each sector evaluator / leader will have a specific terms of reference, with detailed leading questions
for their sector.
3.
EVALUATION TEAM
The evaluation team will be composed of 10 members (1 x team leader and 7 x technical evaluators,
1 x child participation specialist, and 1 colleague from Myanmar civil society) SCiM will be
responsible to engage technical advisers from peer organizations to join the evaluation team to this
end. Evaluators from the Save the Children Alliance are also to be approached to make this a
learning process across the alliance and to also represent the continued involvement and
importance of the alliance to the Myanmar response. One evaluator will be sourced from the
London Emergencies Team in order to ensure that learning is also shared throughout the
department. This evaluator will take on the WASH sector.
The SCiM regular programme will identify and train children in the sampled geographical areas
where evaluation will take place, to prepare them for the evaluation in order to provide children’s
perspective/s of the emergency response.
4.
METHODOLOGY
At the beginning of the response, a set of multi-sector or sector-specific initial rapid assessments
were held, to inform the further design of the Nargis response. While those were not consolidated
into a formal baseline yet, it is expected that the evaluation will compare the currently achieved
results against the situation in May, 08 (as per rapid assessments). It will need to focus on general
and sector specific comparisons in analyzing strengths and areas for improvements using not only
SCiM but also data collected by other agencies (to regain a full picture of the situation in May).
It is important to note that the findings and recommendations need to be contextualized,
understood and explained in the given time and period when they were relevant, given the Nargis
Response was managed in a very dynamic and changing environment.
The evaluation team must be prepared to work with children to ensure their voices are heard and
represented in the evaluation. Effort should be made to use terminology and languages (apply
translation as needed) to integrate children in the discussions of highly sophisticated topics and
debates. A Child Participation Working Group (CPWG) has been formulated in country, consisting of
SCiM staff with expertise in this area. The CPWG will provide overview and a steer for the
methodology and will link with the Child Participation Specialist as a primary client, and will provide
guidance to the evaluation team through this channel. Lastly an Evaluation Steering Group (ESG) has
also been formulated in country and is composed of senior managers. This group will provide
management support to the evaluation, the team and will also ensure that the logistical
arrangements related to the evaluation are taken care of.
5.
RESPONSIBILITIES
The team leader will be responsible for:
Being thoroughly familiar with the response (objectives, outputs, previous phases including
previous activities, problems, budget time schedule, context, etc.).
Page | 53
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Agreeing with project management and funding agencies on the methodology of the
documentation review and methodology of evaluation, in terms of i) level of participation vis-àvis management of the process, data collection, data analysis, drawing conclusions/supplying
recommendations and giving reactions to draft conclusions and ii) the methodology to be
followed.
Making a brief summary of the results of the initial rapid assessment/s for the evaluation team
(adult and children members) to be used in the assessment of impact. (Note: this was deleted)
Leading on the evaluation process and outputs.
Drafting and leading data collection tools, data analysis.
Ensuring that the evaluation report responds to the needs of the client and is actionable.
Ensure equal participation of the evaluation team members in the process, particularly the voice
of children.
Drafting the expected outputs and circulating for feedback and comments.
Finalizing the expected outputs and submitting to SCiM.
The evaluation team members will be responsible for:
Being thoroughly familiar with the response (objectives, outputs, previous phases including
previous activities, problems, budget time schedule, context, etc.).
Supporting the consultant in designing data collection tools.
Collecting data and taking record of the data.
Participating at data analysis and providing technical input into data collection and analysis.
Drafting and finalising their own sector report.
Providing technical input into the draft versions of the final overall report.
Save the Children will be responsible for:
6.
Finalising the Terms of Reference (both generic and sectoral)
Providing all necessary documentation to the evaluation team members
Recruiting evaluation team members
Coordinating with fellow agencies to collect available initial rapid assessment data for the
evaluation team
Providing administrative support to the evaluation team.
Providing feedback to the initial draft of the evaluation report.
ACTIVITIES AND TIMEFRAME
The evaluation time period will be between 9th February 2009 and March 30th 2009. The field work
will take place between 15th February and 01st March 2009. It is expected that a final report will be
available by 30th March.
ACTIVITY
LOCATION/MEANS
WHO
Desk review
UK/Yangon
Lead Evaluator
START AND
END DATE
09 -13 Feb
Training of
children
Labutta, Pyapon
SC Staff,
Participation
16 – 22 Feb Lbt
24 –28 Feb Pyp
EXPECTED
OUTPUT
Famliarisation
with programme,
key staff and
external contacts
Children’s views
inform evaluation
Page | 54
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Consultant
Evaluation Team,
Lead Evaluator
Evaluation team
preparation work
Yangonn – reading
and one team
meeting
Data collection
Thingangone,
Labutaa,
Mawlameigngyun,
Pyapon
Evaluation Team
Data analysis
Yangon
Lead Evaluator
and Evaluation
Team
Draft of internal
report
Draft of external
report
Final internal
report
Final external
report
13 Feb (reading
and documents
distributed
prior to
meeting)
16 Feb – 01st
March
27 Feb (all
team)
findings
Familiarisation
with programme
strategies and
achievements
Judgement on
quality of SC
emergency
response as per
TOR
Evaluation Report
1-5 March Lead
Evaluator
9 March
(tentative)
9 March
(tentative)
16 March
(tentative)
16 March
(tentative)
BUDGET
The total budget for the evaluation is currently set at GBP 41,000. Please see attached DRAFT budget
spreadsheet.
7.
EXPECTED OUTPUTS
External report
The external report should convey the broader findings and recommendations of the Nargis
emergency response focusing on the top line findings to meet the interests of external audiences,
that of general public and affected populations, donors, media and partners. It should discuss
strengths and areas for improvement that concern management of the operation, technical strength
of the programmes, coordination and the extent to which SC managed (or didn’t) to impact the lives
of children, their families and communities. It should also reflect on the extent to which the Nargis
response was accountable to the needs of the affected populations in terms of quality and quantity
of offered services, as well as the systems to capture and operationalize feedback from beneficiaries.
The report should not exceed 30 pages of length (with annexes) and should be comprehensible for a
group of children.
Internal report
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Page | 55
The main audience of the internal report is SCiM and SC Alliance. The report should focus on details
that will not find a place in the external report, bringing more specific components to help decision
making and action planning going forward. It should closely tie to SC’s specific documents such as
Programming Framework, internal management issues and specific processes and systems of SCiM
that would not be relevant for an external audience. The internal report will inform SCiM and
regional/global team on the effectiveness of various preparedness interventions in the past and will
offer guidance as to what kind of organisational preparedness should be maintained after the
current emergency operation has run its course. It will also inform the DRR programming which is
currently being developed across the Delta.
(Note: it was agreed that a single final report would be produced, which would form the basis for a
children’s report to be written by the child participation consultant)
Recommendations
It is expected that the recommendations of both internal and external reports follow a template that
will be given by SCiM. In brief, the recommendations have to be specific, have to be tied to specific
management levels of the Response, have to be time-bound and should clarify the responses
needed to implement those. They should directly link to the long-term strategies, particularly
focusing to support better emergency preparedness and Disaster Risk Reduction work for SCiM.
Recommendations should avoid recommending changes for programmes that are due to end in
January or at any closer date. Guidance on those programmes will be shared with the consultant and
the team in January 2009.
(Note: It was agreed that the report would contain detailed recommendations but that the
timeframe and plan of implementation would be determined by SCiM)
Page | 56
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
ANNEX 3: Methodology Matrix
Evaluation of Save the Children’s Response to Cyclone Nargis
Andy Featherstone, METHODOLOGY MATRIX
Evaluation Criteria
Relevance
7
Effectiveness
Timeliness
Coverage (inc.co-ord)
Technical Merit
Connectedness
Description
The extent to which the
activites achieved their
purpose or whether this
can be expected to
happen on the basis of
the outputs - linked
closely to other criteria
The extent to which
the programme
was delivered
according to its
timetable and the
extent to which this
appropriate to the
needs of
beneficiaries
The overall size of the programme
relative to needs taking into
account other agency programmes
and gaps in coverage
The extent to which the
programme activities are in
line with local needs and
priorities (as well as donor
policy)
Strengths and
weaknesses of
response across all
technical areas
(education, child
protection,
livelihoods,
health/nutirition, food
security, WASH)
Standards,
Benchmarks
&
Information
Sources
CoC: We shall respect
culture and custom;
Ways shall be found to
involve programme
beneficiaries in the
management of relief
aid; Relief aid must
strive to reduce future
vulnerabilities to
disaster as well as
meeting basic needs;
CoC: The
humanitarian
imperative comes
first; In our
information,
publicity and
advertising
activities, we shall
recognise disaster
victims as dignified
humans, not
hopeless objects
CoC: The humanitarian imperative
comes first. ER Cluster Strategy:
The delivery and coordination
capacity of assistance providers and
implementing agencies is key to the
success of this strategy. Agencies
must be able to demonstrate their
presence in the affected areas, and
their capacity to deliver against
agreed objectives. Early recovery
activities should be closely
coordinated within and between
agencies and clusters, to maximize
their effectiveness, avoid gaps,
minimize overlap, and reduce any
burden that agency presence may
place on communities. Ideally, early
recovery initiatives should work to
foster cooperation among local
authorities and civil society groups
CoC: The humanitarian
imperative comes first; Aid is
given regardless of the race,
creed or nationality of the
recipients and without
adverse distinction of any
kind. Aid priorities are
calculated on the basis of
need alone; Aid will not be
used to further a particular
political or religious
standpoint; We shall
endeavour not to act as
instruments of government
foreign policy;We shall
respect culture and custom;
We shall attempt to build
disaster response on local
capacities; Ways shall be
found to involve programme
beneficiaries in the
management of relief aid
Sphere standards,
INEE standards,
internal policy on
child participation,
HAP Accountability
standard
The extent to which
the activities of the
emergency
programme are
carried out in a
context that takes
longer-term and
issues of programme
integration into
account
CoC: Relief aid must
strive to reduce
future vulnerabilities
to disaster as well as
meeting basic needs.
ER Cluster Strategy:
Projects and
programmes will seek
to address the
vulnerabilities of
cyclone-affected
communities,
ensuring that risks are
not rebuilt into the
recovery effort and
the capacities of
communities are
strengthened to
reduce further risk.
7
This matrix is based on a concept by Sarah Routley, methodology consultant
Accountability (inc. Participation)
Upwards & downwards accountability:
provision of information, communication,
participation, reporting, decision-making
CoC: We shall respect culture and custom; We
shall attempt to build disaster response on
local capacities; Ways shall be found to involve
programme; beneficiaries in the management
of relief aid; We hold ourselves accountable to
both those we seek to assist and those from
whom we accept resources. ER Cluster
Strategy: ER efforts will be designed and
implemented in a transparent manner,
through effective information sharing,
inclusive local level consultations and dispute
resolutions to ensure full accountability to
beneficiaries and donors. HAP Accountability
Principles: (i) commitment to humanitarian
standards and rights (ii) setting standards and
building capacity (iii) communication (iv)
partication in programme (v) monitoring
compliance (vi) addressing complaints (vii)
implementing partners; SC Practice standards
in child participation: See standards document
Page | 57
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Questions
Effectiveness: Are the
outcomes clearly
articulated in project
documents and
measurable? Did the
programme/project
outputs achieve their
intended outcomes?
Why was this so (or
not)? Who benefitted
(disaggregated data)?
Were the changes a
direct result of the
project or did factors
outside of the project
contribute to these
changes? At Community
Level: What affect has
the cylcone had? What
assistance was provided
by SC? What were the
objectives and to what
extent were they met?
What number of
beneficiaries were
targeted? How did the
assistance change the
situation of the project
participants? What
project activites
contributed to the
change? What is
different as a result of
the assistance? (do the
impact of the cylcone
and the contribution of
the project match?)
Timeliness: Timeline
for SC activties
compared against
log frames. At
Community Level:
When did you need
assistance and what
assistance did you
need? When did the
team arrive and the
project start? When
would you have
most liked to have
received
assistance?
Coverage: What were the main
reasons that the intervention
provided or failed to provide major
population groups with assistance
and protection, proportionate to
their need? What approach did SC
take towards coverage (globally
and locally)? Why was this decision
taken? How was targeting
conducted? At Community Level:
Did some groups within the
community have special needs?
Were these needs met by the
assistance provided? How were
specific needs linked to agencies
interventions? Did any specific
groups not receive the assistance
they needed? Why was this? How
were beneficiaries selected? Could
this have been improved? Did those
most at need get the most? Coordination: How did SC seek to coordinate its activities with other
agencies (were interagency coordination mechanisms adequate
and did CS participate in these?)
Were there different agencies
carrying out similar work? Was
there duplication, gaps, delays? Did
agencies work together to ensure
they worked in a similar way? Were
there differences in approaches
used? Did SC co-ordinate
appropriately with local
authorities?
Relevance: Did SC have
relevant skills to deliver the
programme? If not, how
successful were they in
obtaining these? What
approach was taken by SC to
seek to ensure projects were
relevant to the situation
(culturally appropriate,
needs-based, appropriate for
different sections of the
community)? Is this borne out
by project data such as
assessment reports? Did the
type of assistance match
community requirements?
Was it of good quality (how
did SC decide what was
adequate? Were they guided
by quality standards such as
Sphere?)? Could any
improvements have been
made? Would more or less
have been better?
Interventions appropriate to
needs of indivduals within the
community (HIV, children,
orphans, women, elderly etc.)
Technical Merit:
These have been
developed by SCiM
and circulated to
evaluation team
members. It will be
important that these
are checked for
consistency and that
ALL CRITERIA are
applied across all
technical sectors.
Connectedness: Were
appropriate linkages
made between relief,
early recovery and
recovery in SCs
programmes? Have
partnerships been
developed (with local
NGOs, Int orgs,
comms) that support
the transition of the
programme? To what
extent has the
programme built local
capacity that will
strengthen the
process of recovery?
What practical steps
should SC take to
ensure a smooth
transition from relief
to recovery?
Integration: To what
extent did the relief
response build on the
strengths of the longterm programme? To
what extent was
internal co-ordination
evident and to what
extent did sectoral
activities build on
each other? If not,
why was this?
The level of involvement of beneficiaries in key
aspects of the project cycle: What approach
did SC take towards beneficiary involvement
throughout the project cycle? At Community
Level: What is the perception of community
members about their participation? How were
they involved (decision-making,
implementation, responsibility, provision of
ideas, planning, assist in understanding,
consent on decisions, informed only)? What
do beneficiaries feel about their involvement?
How did it benefit the project activities? How
else would they have liked to have been
involved? Had a say in what agencies did,
priorities, process, provided feedback,
consulted on effectiveness: What was the
approach of SC to providing information,
consulting and incorporating feebdack (find
examples of this from project records, minutes
etc)? Did communities know in advance what
SCs plans were? How did they find out? Did SC
discuss any changes to these plans? Who was
consulted (chief, men, women, children, all)?
Was there any written agreement or
documentation about the entitlements? Were
communities asked what they wanted and
what their priorites were? If there had been a
problem, how would it have been dealt with ?
Did communities have to report any
problems? How was this dealt with? Do SC
have examples of complaint handling?
Page | 58
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Child
Participation:
Good
Practice and
Methods
Timelines – Explores key milestones and challenges over time as well as issues of timeliness, accountability, participation and is a good entry point into sectoral programmes;
‘H’ Assessment – uses the shape of an ‘H’ to explore the strengths, weaknesses and ideas to improve humanitarian/development initiatives;
Before/After Body Map – where children draw, write or describe differences to them (what they see, hear, say, what they think, where they go, what they know, how they feel) and any changes about how people behave
with them or listen to them as a result of the programme
Data
Collection &
Information
Sources
project proposals, logical
frameworks, sitreps, trip
reports, interviews with
SC staff in country,
beneficiary interviews,
household surveys,
observation
Assess risks. Before each consultation, assess the risks participants may face by taking part in the consultation and take steps to minimise these risks
Do no harm: Efforts should always be made to ensure that children’s participation does not harm children in any way. This includes taking measures to protect children from abuse by those working with them.
Gain consent and provide information. Families should be consulted well in advance regarding the involvement of children in the consultation and their consent should be sought wherever possible. In addition, seek the
support of community leaders or others who play an important role in children’s lives. Children (and adults) should always have a choice about whether or not they take part in the consultation. To make this choice they
need to understand the purpose of the consultation and how the findings will be used and by whom. They must be aware that they can withdraw from the consultation at any time.
Avoid raising expectations about the direct benefits of the consultation. Be honest and clear about how the outcomes of the research will affect those who are taking part.
Ensure representation of a wide range of children in terms of age, gender, ability, circumstances, ethnicity and religion etc. Make sure that your programme does not reinforce existing inequalities in the society by acting
upon the opinions of a few more advantaged children and excluding others. Applying a rights based approach brings with it an obligation to ensure that the voices and concerns of boys and girls whose rights are most
violated are raised with priority. Separate focus group discussions using participatory activities should be arranged with girls and boys of different age groups (for example, under 7 years, 8-12 years, 13-18 years) and from
different backgrounds.
Consider the timing and location. Consultations must take place in appropriate settings that give children the confidence, privacy and space to share their true feelings (but preferably in their own environment). It should
not interfere with children’s schooling or rest.
Develop child friendly communication skills. For example, staff should be patient and non-judgmental, use language appropriate to a child’s age/ culture, and encourage children through listening attentively and respecting
children’s views.
Prepare resources: For example, trained staff (trained in child protection, work in pairs to enable monitoring of colleagues behaviour), food, transport, materials.
Plan follow-up and seek advice on how to respond to any allegations or disclosures (for example, of abuse), make referrals, and provide further support to children and families.
project proposals,
logical frameworks,
sitreps, trip reports,
interviews with SC
staff in country,
beneficiary
interviews,
household surveys,
observation
project strategy, project reports,
gap analysis, interviews with SC
staff; interviews with co-ordinating
agencies (OCHA) and NGOs,
observation
project proposals, donor
agreements, beneficiary
interviews at individual/
household, community level,
observation, technical
reports, trip reports, shelter
survey, household survey
project proposals, trip
reports, interviews
with technical staff,
beneficiary
interviews,
observation
project proposals,
logical frameworks,
sitreps, trip reports,
interviews with SC
staff in country,
beneficiary
interviews, household
surveys, observation
structured commuity survey work; beneficiary
interviews at individual, households,
community level; internal documents on
participation; internal interviews on project
implementation; observation
PART 2: Evaluation of Save the Children’s Response to Cyclone Nargis
Page | 59
ANNEX 4: Children’s Suggestions
Below are suggestions from children compiled from the two children feedbacks workshops.
Programme areas
Child protection
Suggestions
•
•
•
•
•
•
•
•
Education
•
•
•
•
•
•
Food
Livelihood
•
•
•
•
•
Nutrition
•
•
•
•
•
•
•
Help us not to get scolded by elders.
We want many drawing items at CFS.
We want you to rebuild/reopen the CFS.
(If the CFS reopen) We want a better building for CFS.
We want more toys at CFS.
We need more knowledge.
Some children can’t go to the CFS. They need to learn to read and
write.
Some teachers shout at children. We don’t like it.
We want you to provide books for children in order to improve
knowledge.
We want jogging machine and slides at ECCD (to strengthen
muscles and bones).
ECCD teachers should take care of children more.
We would like you to provide us TV (DVD) so that we can watch
educational movies.
We would like you to provide us all types of game.
We want teachers from ECCD to teach the children in order to
improve educational knowledge.
More distribution of livelihood items.
We would like to get re-distribution of rice, pulse, cooking oil and
salt. (We have to spend all our wages daily)
We would like you to do more distribution of pigs, ducks, and
boats. We need distribution of business materials.
We would like you to conduct discussions on knowledge.
We want boats. We can use it in going to school/travel in rainy
season. We can use it in fishing.
We want you to encourage people not to feed babies with
disqualified milk powder.
We want you to provide more nutritional education to mothers on
proper breast-feeding for babies from 6 months to 2 years old.
We want you to provide people with nutritional knowledge on
supplementary feeding to the babies who are over 6 months old.
Some pregnant women need awareness raising so they will want
to breastfeed their children.
Pregnant women need more nutrition food.
We want you to find lactating woman for motherless babies.
We want you to cure children who are sick.
PART 2: Review of the Save the Children’s Response to Cyclone Nargis
Health
WASH
•
•
•
•
•
Page | 60
We want to put fence on more ponds.
We would like you to provide a latrine for each household.
We would like you to make a brick road heading to the pond (to get
cleaner water).
We prefer latrines with bamboo walls.
We would like you to build a rain-collection tank at CFS.
PART 2: Review of the Save the Children’s Response to Cyclone Nargis
Page | 61
ANNEX 5: Timeline of Significant Events that Shaped the SCiM Response
DATE
MAY
May 2/ 3
May 05
May 06
May 07
May 08
May 09
May 10
May 11
May 12
May 13
May 14
May 15
May 16
May 18
May 20
ACTIVITY
Cyclone Nargis makes landfall in Myanmar in the late evening / early hours of the
morning. Programme team decision to launch a response
SC assessment teams reach some locations in the delta and Yangon Division. SC
immediate response reaches over 50,000 people with immediate relief including
cash, food and NFI’s. First SC sitrep issued to the alliance. ELT decision on emergency
response and lead member made. First official SC press release issued although lines
and quotes were going out from 04 May.
IASC Country team meeting and development of flash appeal. SC Alliance appeal
for GBP5m opened. First SC Funding tracker spreadsheet developed. SC emergency
snapshot issued by lead member
First SC (private and public) advocacy points circulated and updated three times over
the course of the day. Media efforts carried out with CNN international, Bangkok
Post, International Herald Tribune, CBS, BBC, Ashai TV, TV News 2 Denmark, Spanish
Radio, Radio New Zealand, Swedish Radio. SCUK media manager arrives in Bangkok
to coordinate media (Kathryn Rawe). Sarah Tyler, Alliance Communications Manager
already in Bangkok to communicate alliance communications. DEC appeal launched
in the UK. First SC HR Spreadsheet developed
David Wightwick (SCUK Emergencies Advisor) and Sonia Khush (SCUS Emergency
Programme Manager) arrive in Bangkok, awaiting visas, joining other regional
support including Dan Collison (Regional Programme Manager). Eight (8) Alliance
members pledge funds to the response totalling approx GBP100, 000. GBP5m
alliance appeal opened. Media interest massive (UK, Danish, Aljazeera, US all
interested)
UN flash appeal launched.
Letter sent to John Holmes on behalf of IWG pressing for UN pressure on
international aid agency access to the Delta (SC is a signatory). SC circulates
advocacy / power analysis to the alliance on key actors / governments in the
response. SC updates previous (public and private) advocacy points with updates, to
the alliance.
Lead member advocacy team develops scenario planning for the response. Rapid SC
assessment conducted in East Delta
Child Protection Advisor and logistics Advisor arrive in Yangon following successful
visa application
First aid flights arrive in Yangon including two for SC. Second logistics advisor arrives
in Myanmar for briefing. SC given official letter from Govt of Myanmar stating
official partnership in the relief response
Advocacy points updated and reissued to alliance by lead member
Lead member Advocacy team update advocacy points.
SC relief operation reaches 115,000 people with cash, NFI and tarpaulin. SCUK
Advocacy team issue briefing for IASC telephone hook up, supporting calls for
increased aid and humanitarian access.
6 SC (international) staff members arrive in Yangon including the Emergency
Response Coordinator (David Wightwick) from Bangkok. Lead member releases
update emergency snapshot (10 days on). Lead member updates advocacy points
Jasmine Whitbread (SCUK CEO) arrives in Yangon.
Strategic Finance Manager arrives in Bangkok.
PART 2: Review of the Save the Children’s Response to Cyclone Nargis
May 25
May 26
JUNE
June 01
June 02
June 03
June 06
June 10
June 12
June 18
June 24
June 29
June 30
Page | 62
Strategic Finance Manager arrives in Yangon.
International staff (for SC) granted access visas to the delta for the first time.
Rapid assessment of west delta report released.
Lead member issues one month on press release.
Lead member Advocacy team update advocacy points.
Lead member advocacy team update advocacy points. SC team in Yangon begin cash
transfer programme, distributing 58m Kyat in this first five days
SC team in Yangon begin distributing a further 72m Kyat in the cash transfer
programme over 2 days
SC cash transfer programme suspended, following political concerns
Lead member advocacy team update advocacy points
SC draft emergency strategy issued
A final 41m Kyat is distributed over 2 days by the SC team in Yangon as the cash
transfer programme is temporarily resumed. Gareth Owen (Emergency Director)
makes first visit to Yangon
The SC cash transfer programme is terminated but not before 171m Kyat
(approximately USD171, 000) is distributed in total in the nine days that the
programme runs for.
JULY
July 24
SC emergency strategy updated and finalised
AUGUST
August 02
August 18
Lead member issue 3 months-on press release and media / comms work
Lead member advocacy team develop ideas for 3 months on communication work